Win64 builds status update

Two things:

• we now have all Win64 build machines cloned
• we now have all branches with Win64 builds (Try coming soon)

The builds are currently not compiling but khuey and Makoto Kato are tackling each issue until we get back to compiling builds.

Meanwhile I will be setting up a subset of the pool to take care of doing Try builds.

I will give another status update when something new happens.
If you want to be up-to-date on all details please follow bug 558448 [1].
You can read my previous post for background information and

[1] Bug 558448 - (support-win64) [Tracking bug] officially support Windows 64-bit builds

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Go Faster: improved download times for test jobs and merged few talos suites together

As part of the Go Faster initiative I have been involved with a couple of bugs that have now recently gone live.

The first one is establishing a p2p link between two of our colos which improved the download times of builds, symbols and test packages to the test slaves. [1]

We can see in the diagram how the average setup time (which download times are part of) has stabilized after IT enabled the p2p link was enabled last Tuesday Aug. 9th in between our collocations.  This means that we have faster download times per run and proportionally more time is spent running tests. This makes our jobs to take less time and increases our pool's capacity. Unfortunately, this also is causing some builds to fail to upload artifacts to ftp (bug 677348). IT has a plan to fix this.

On the other hand, I have merged three talos jobs into only one (this went live yesterday Aug. 16th) [2]. This means that we have removed 2 setup times plus 2 reboot times per push. This is minimal but it gets us started to do more of the same.
Our goal is that every talos suites that takes less than 10 minutes should be joined with other jobs as long as they don't take over 30 minutes. In other words, keep every talos job within 10 and 30 minutes (initial goal).
You probably won't be able to tell that this changed as you will still be able to see the email regressions showing up on the dev.tree-management list. The only difference is that there will be two less "T" jobs per push on tbpl and when you click on a "chrome" talos job you will see the a11y and tscoll suites show up in the summary like this:

• tsspider: 11.68 (details)
• tdhtml: 538.94 (details)
• a11y: 273.5 (details)
• ts: 411.26 (details)
• ts_shutdown: 328.32 (details)
• tscroll: 11582.2 (details)

Now I will gather new data and determine which other suites are to be merged together.
I previously gathered some data and analyzed but I guess I did not write a blog post but just made a comment in a bug.

Stay tuned for more information!

[1] Bug 661656 - Determine if we can improve the download times between sjc1 and scl1
[2] Bug 659328 - Merge talos suites that finish in less than 10 minutes to improve wait times

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Firefox and Windows 64-bit builds (testing version; not a release version)

We now have a small set of Windows 2008 64-bit slaves ready to be put in our production systems that can generate the 64-bit version of Mozilla Firefox.

NOTE: This is not a released version but a testing version.




I will leave out any talking about when this could be released to our users and just focus on explaining where the project has been and where we are now from a Release Engineering point of view. Releasing depends on evaluating what the problems are on the product side before we would release it.
If you have done any comparison of the pros/cons of the 32-bit and 64-bit version of Firefox running on a 64-bit machine please let me know as I am interested to know.

Try it out
We have been producing Firefox 64-bit nightly builds since last week but we now have a small pool of machines and we are upstreaming the process to production levels.

You can give it a try by downloading the installer.

Help needed
Right now I have several bugs that I need help from developers to get them fixed.

• bug 671000 - make buildsymbols takes 45 mins rather than 5 mins
• bug 669384 - make buildsymbols fails for leak test builds
• bug 670915 - make package fails for leak test builds
• bug 670697 - sporadic make check failures 

There are many more bugs but the ones I mentioned are the ones that affect releng infrastructure.
We are using a tracking bug for product problems and another one for releng problems.

Background
I started working on this project last year on Q2 and by May 2010 I had some proof of concept going on. To my surprise the media picked up on this and brought a lot of attention to it.
By Q3 we started having problems with OPSI (a system to deploy changes to our machines) and all efforts started little by little shifting towards supporting developers to ship Firefox 4. On Q2 of this year all focus was to adapt to the new fast release cadence.

Nevertheless, by the end of March I had set up a machine to be cloned unto other machines.
Unfortunately the tools that we had in IT were not being able to clone the machine.
IT at that point started to look for a solution and in June we hired digipengi who has Windows experience (for real!).
We worked together and we had to recreate the Windows 64-bit machine from scratch with only one partition rather than three.
We are now at the point that we have 5 production slaves, another 4 to be added soon and we will be cloning the remaining ones in the near future.

If you want to know more CC yourself to the tracking bug.

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.