Wednesday, August 13, 2008

L10n parallelization results

We want to do L10n repackages in parallel to improve the time it takes to generate them since we are growing every month the number of locales.
This blog posts shows some early results on staging machines and it is just for future references

List of machines used for setup A and B:
  • 3 linux, 1 mac and 3 win32
Setup A (buildbot repackages in paralell)
In this setup I made the slaves to "remove everything (mozilla and l10n) before you start with the next locale"

Start time:
  • 15:44 - Tuesday Aug. 12th
End times:
  • Linux -> 16:25 (41 mins)
  • Mac -> 17:04 (1hr 20mins)
  • Win32 -> 18:09 (2h 25mins)
Setup B (buildbot repackages in parallel)
In this setup I did not remove anything, just checked out again

Start time:
  • 10:20 Wednesday Aug. 13th
End times:
  • Linux -> 10:37 (17 mins)
  • Mac -> 11:04 (44 mins)
  • Win32 -> 11:44 (1h 24 mins)
NOTE: See the results in this tinderbox page (look at columns *_l10n_nightly that are in the right side)

Old Tinderbox setup (everything taken care by one machine)
Start date:
  • Aug. 13th (each platform starts as its own time - we just know that it can happen after 9am)
Start and end times:
  • Linux -> 9:11->9:39 (28 mins)
  • Mac -> 9:04 -> 9:26 (22 mins)
  • Win32 -> 10:48 -> 14:26 (3h 37 mins)
Conclusions
We can say that the platform that gets most benefited is windows (from over 3hr to 1hr and a half), while the mac looses performance (from 22mins to 44mins)

The drop in performance with the mac is expected because of the new way of doing things and can be improved

Wednesday, July 30, 2008

L10n build presentation slides

Thanks everyone who attended the "L10n build process" presentation at the Summit 08.

This the presentation that was used for the session, please contact me if something does not make sense:



These are some of the notes/feedback that I got from the discussion with the rest of the localizers:
- No back check-in to locale's repository
- Generate installer with missing string so the installer does not break
- Modify the branding if the locale is missing entities
- Add Source Stamp in "About" for better litmus testing

The last two notes/suggestions are not at hand right now and they will be worked on in due time

Monday, July 07, 2008

Make our nightly builds identical for each platform - part 1

The way we do nightly builds every night for Firefox 2.0.0.x and Firefox 3.0.x have few problems that will be good to have them fixed.
  1. The build for each platform happens AFTER a determined hour BUT whenever the build slave can start it. Therefore, the build start time is different from platform to platform. This will be even more obvious when the buildID will capture minutes and seconds. This is not a big problem but it is not a correct behavior and it would be good to have the same buildID for all 3 platforms.
  2. The build for each platform is NOT build from the same source code. You knew it or not, this has been happening for many years. This is a big problem specially when a commit to the source code happens around the time that the nightly builds are to happen and this change might be captured by one platform and not by the others. This problem affects when you want to see if a bug introduced in one of the platforms is part of the other two platforms
Things we are doing to improve this
  1. Separating the generation of nightly builds from generating dependent builds
    For years, we have generated dependent and nightly builds from the same slave running the exact same code all day. We generate nightly builds after the last dependent build that gets finished after 3AM PDT. Each platform finishes at different time and each platform checks out different source code. To be more precise a nightly build is started if 1) it is passed the $build_hour and 2) twenty four hours have passed since the last nightly build. To stop the dependt builds process to generate nightly builds we need to set $OfficialBuildMachinery to zero in the tinder-config.pl
  2. Use Nightly scheduler to trigger nightly builds
    Using a nightly scheduler will allow us to add more steps after an en-US nightly build like trigerring L10n repackages and/or trigger other schedulers as needed
  3. Setting the checkout time before the build starts
    If we set up the same MOZ_CO_DATE for the 3 different platforms before the build start they will check out the same source stamp no matter when the slave will start the build
  4. Remember the checkout time
    Saving the SourceStamp inside of the application.ini file will allow us to know which SourceStamp was used for that build even if the log for that build gets removed from the tinderbox logs
How would things look like after these changes?





















As the image shows:
  1. Our nightly builds will not depend on previous dependent builds
  2. The nightly builds will be triggered when we say
  3. The nightly builds will check out the same source code
  4. From a binary we will know which source stamp was used to generate the build