Monday, June 23, 2008

l10n repackaging - part 5 - introduction

This is the first time you are reading one of my blogs posts and you will be able to hear on this Monday's meeting talking briefly about the project I am working on related to change the way we do l10n repackages.

The way localizations are generated are by: 1) downloading an en-US binary, 2) unpack it and 3) repack it by replacing certain files from the specific locale.
The way we do repackages are in dedicated machines and we are trying to move it over the buildbot infrastructure to make it scalable and do them on our generic buildslaves

Key concepts for the next weeks
I have done several blogs posts over the past weeks (1, 2, 3, 4) and I will just talk what my attention is at right now:

Nightly repackages
In this scenario we have an en-US build being generated at certain hour of the night, checking out a specific source stamp and after it finishes we can continue by checking out the same source stamp that we are going to use to do l10n repackages. For now, I want this code into the staging configuration we have since it involves 2 schedulers (Nighlty and DependentL10n) and it can be done with one machine until we find the proper way to pass the source stamp from one scheduler to the next one.

Dependent repackages
We generate dependent en-US builds continually and it can take less than 6-8 minutes to generate one of them, therefore, we do not really have enough time to keep up with that rhythm with l10n repackages (in my local machine it takes around 1-2 mins to get ready and 30 secs to repackage each locale and with 50 locales it takes a total of 25 mins) unless we had 4-6 slaves for l10n repackages and that is just thinking of linux or mac machines, the windows slaves take way longer than the previous ones.
I am thinking that we would want to do l10n repackages every hour and the source stamp would most likely have to be different from the dependent en-US that we would download (unless we have the en-US source stamp somewhere in the application.ini), but for now it should do

Release repackages
In our release automation system we already do l10n repackages in the generic slaves but it still does it by using the code used in tinderbox client which is not scalable.
Changes in here will require to check out the same tag as the en-US build does

Bootstrap configuration
There is a lot of configuration that is read in our release automation code from "bootstrap configuration" files which contain information as which tag to checkout, where to upload the builds, which tree to report on tinderbox, etc... and this information will be needed in the process of doing l10n repackages on buildbot. The best solution is to have a factory (or a step) to parse this information and add it as a property to the build object that gets passed from step to step on buildbot and read them to be able to use them at the proper time.

Major problems

  1. Make sure that multiple slaves run preparatory steps. Currently, the only way I have found is to have a builder per slave under a Dependent scheduler to make sure that all slaves run these steps but this has its problems, which is that if a slave disconnects (does not fail) it will not trigger the next DependentL10n scheduler
  2. Pass build properties from Scheduler to Scheduler which I thought we would not need but it seems like it. I might be able to "cheat" by passing the checkout time in the application.ini file or by passing text files to the slaves with the desired information


  1. As a note, SeaMonkey is currently repackaging all it's builds from the "Nightly" machines. It's of course still tinderbox (and I'd be happy to switch over to buildbot one time), but we're doing building and repackaging all in one go, which eases the checkout stuff of course, but probably only works with the smaller number of locales we have.

  2. Kairo, could you explain me or point me out where it is explained how the repackages SeaMonkey works? Is it different from how the firefox repackages work currently?
    Does it download the en-US binaries from
    Oh! I can see that you have l10n repackages along with the en-US, that's nice...

    What is the code that gets run for SeaMonkey repackages? The code same from Tinderbox client?

    The way I prepare a slave to do repackages is to 1) check out with l10n-checkout, 2) make configure, 3) make -c config and 4) a wget of the latest en-US binary. Once ready, I basically check out the code for that specific locale and I "make installers-FOO" inside of browser/locales

    This is a screenshot that shows a little about what I just talked about
    and this is the bug that captures the things I have been trying

  3. I still think that we shouldn't have per-slave setup steps, and that we don't really need them. We should catch up on that on the phone or something.

  4. Pike, as you say I do not like it either and I wish we had this feature in buildbot.
    I will continue the thread on buildbot's mailing list and let's try to arrange another meeting next week

  5. armenzg, SeaMonkey is using plain tinderbox client code currently, the config is up at mozilla/tools/tinderbox-configs/seamonkey/

    Of course, we'll need different solutions when we likely switch over to 1.9.1, where tinderbox clients probably won't work that way.