l10n repackaging - part 3 (it feels so goooood to have a solution)

I started this past week to work on the code (which is highly inspired on Axel's work for the l10n build processes) that will allow us to distribute l10n repackages between slaves to do repackages of all the locales we have.

In my previous blog post I was worried on how to get the master to have the latest information of all the locales we have without a) having to restart a buildbot master to grab the latest list of locales and b) not doing extremely hacky things with buildbot.

I wanted to try a couple of things by making the slaves do some work as you can see in this quote from last post:

How can we change this?

• An initial slave could do some "thinking" and notify "someone" (an object) in the master which locales to be repackaged
• An initial slave checks out a file, uploads it to the master and somehow notify the master to reconfigure itself

BUT I realized that I should go to the moment that all the build requests are generated and just before that get the latest all-locales file from the repository.

The "good-enough" solution

  def getLocales(self):
"""It checks out the all-locales file"""
Thanks to bsmedberg in this one
tuple =  subprocess.Popen(
['cvs', '-d:pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot',
'co', '-p', 'mozilla/browser/locales/all-locales'],
stdout=subprocess.PIPE).communicate()
list = tuple[0].split('\n')
if list[-1]=='':
list = list[0:-1] #get rid of the last one
return list

def doPeriodicBuild(self):
if self.locales:
locales = self.locales
else:
locales = self.getLocales() <-- We get a list with the latest list of locales

#look line
for locale in locales: <-- We create a non-merge-able
                     build request per locale in the list
obj = PeriodicL10n.BuildDesc(locale)
#print obj.locale
self.queue.insert(0, obj)
bs = buildset.BuildSet(self.builderNames,
                    #SourceStamp(branch=self.branch),
                    PeriodicL10n.NoMergeStamp(branch=self.branch),
                    self.reason)
      self.submit(bs)

What do we have solved so far

• By using NoMergeStamp, we have build requests that do not get merged by buildbot
• The function getLocales() will always generate the full list of all-locales without doing anything hacky with buildbot and always generate the right amount of buildrequests
• In the function PeriodicL10n.BuildDesc(locale) we generate objects that contain a property "locale" which in a later process gets passed to a Build object and therefore we can use WithProperties("l10n/%(locale)s") which is a class that will generate a string with values extracted from the current build object of a step. For example:
  l10n_depBuildFactory.addStep(ShellCommand(
  command=["cvs","-q","-z3","-d",cvsl10nroot,"co",WithProperties("l10n/%(locale)s")])
• We have a queue with Builds that are taken every time there is a slave available, therefore, the more slaves we have ---> the shorter it takes to do the whole process
  

Great relief

I was really frustrated just before I reached the previous solution because I did not want to spend what it was going to be a lot of "trial and fail" with the different options which could have led to very complicated solutions or dead end roads.
I am glad that I did got rid of what it was for me the biggest bug of my project and now I can dedicate my self to put all the pieces of my research into a bunch of steps in buildbots which should be able

What is to come ...

I still haven't received any feedback but that is fine because I still have to continue working on what the l10n repackage of a single locale involves, which I will describe in a later post, but for now let me list what is in my mind of things left to be done:

1. Write and test the set of steps that generate a single locale (I am half way through)
2. Research what the push and announce steps do (this needs further explanation)
3. There are common steps to all locales that a slave has to do. This means that a slave might be repeating the same task from one locale to another one. It might be interesting to keep some of the work done for a previous locale for the current one. 1) checkout of firefox's code, 2) make configure step, 3) download of latest en-US and 4) partial compilation of some objects
4. It is really important to check out the same timestamp of firefox's code for all locales. I have an eye on "Trigering Schedulers" and the last comment on it: "This is useful to ensure that all of the builds use exactly the same SourceStamp, even if other Changes have occurred while the build was running."

NOTE: You can see an image on the right side that shows which step I am at right now. "make installers" of a single locale. Soon we should see it green as well