In my previous blog post I was worried on how to get the master to have the latest information of all the locales we have without a) having to restart a buildbot master to grab the latest list of locales and b) not doing extremely hacky things with buildbot.
I wanted to try a couple of things by making the slaves do some work as you can see in this quote from last post:
How can we change this?BUT I realized that I should go to the moment that all the build requests are generated and just before that get the latest all-locales file from the repository.
- An initial slave could do some "thinking" and notify "someone" (an object) in the master which locales to be repackaged
- An initial slave checks out a file, uploads it to the master and somehow notify the master to reconfigure itself
The "good-enough" solution
"""It checks out the all-locales file"""
Thanks to bsmedberg in this one
tuple = subprocess.Popen(
'co', '-p', 'mozilla/browser/locales/all-locales'],
list = tuple.split('\n')
list = list[0:-1] #get rid of the last one
locales = self.locales
locales = self.getLocales() <-- We get a list with the latest list of locales
for locale in locales: <-- We create a non-merge-able
build request per locale in the list
obj = PeriodicL10n.BuildDesc(locale)
bs = buildset.BuildSet(self.builderNames,
What do we have solved so far
- By using NoMergeStamp, we have build requests that do not get merged by buildbot
- The function getLocales() will always generate the full list of all-locales without doing anything hacky with buildbot and always generate the right amount of buildrequests
- In the function PeriodicL10n.BuildDesc(locale) we generate objects that contain a property "locale" which in a later process gets passed to a Build object and therefore we can use WithProperties("l10n/%(locale)s") which is a class that will generate a string with values extracted from the current build object of a step. For example:
- We have a queue with Builds that are taken every time there is a slave available, therefore, the more slaves we have ---> the shorter it takes to do the whole process
Great reliefI was really frustrated just before I reached the previous solution because I did not want to spend what it was going to be a lot of "trial and fail" with the different options which could have led to very complicated solutions or dead end roads.
I am glad that I did got rid of what it was for me the biggest bug of my project and now I can dedicate my self to put all the pieces of my research into a bunch of steps in buildbots which should be able
What is to come ...
I still haven't received any feedback but that is fine because I still have to continue working on what the l10n repackage of a single locale involves, which I will describe in a later post, but for now let me list what is in my mind of things left to be done:
- Write and test the set of steps that generate a single locale (I am half way through)
- Research what the push and announce steps do (this needs further explanation)
- There are common steps to all locales that a slave has to do. This means that a slave might be repeating the same task from one locale to another one. It might be interesting to keep some of the work done for a previous locale for the current one. 1) checkout of firefox's code, 2) make configure step, 3) download of latest en-US and 4) partial compilation of some objects
- It is really important to check out the same timestamp of firefox's code for all locales. I have an eye on "Trigering Schedulers" and the last comment on it: "This is useful to ensure that all of the builds use exactly the same SourceStamp, even if other Changes have occurred while the build was running."
NOTE: You can see an image on the right side that shows which step I am at right now. "make installers" of a single locale. Soon we should see it green as well