Gaia code changes and how the trickle-down into Mozilla's RelEng CI

Too long; did not read: In our pushes' monthly report we more or less count all Gaia commits through the B2G-Inbound repository.

For the last few months, I've been creating reports about the pushes to the tbpl trees and I had to add some disclaimers about the code pushes to the Gaia repositories. I've decided to write the disclaimer in here and simply put a hyperlink to this post.

Contributions to the Gaia repositories are done through GitHub and are run through the Travis CI (rather than through the Release Engineering infrastructure). However, independently from the Travis CI, we bring the Gaia merges into the Release Engineering systems this way:

• github.com
• Changes are merged to master (mozilla-b2g/gaia)
• These changes trigger the Travis CI
• git.mozilla.org 

• We mirror the github changes into our git setup (gaia.git)
• These changes trigger the Travis CI

• hg.mozilla.org
• We convert our internal git repo to our hg repos (e.g. gaia-central)
• There is a B2G Bumper bot that will change device manifests on b2g-inbound with gonk/gaia git changesets for emulator/device builds
• There is a Gaia Bumper bot that will change device manifests on b2g-inbound with gaia hg changesets for b2g desktop builds
• Those manifest changes indicate which gaia changesets to checkout
• This will trigger tbpl changes and run on the RelEng infrastructure

Here's an example:

• https://github.com/mozilla-b2g/gaia/commit/341bb57b196f94cdb4cb91f9bce1352fe50565b1
• https://git.mozilla.org/?p=releases/gaia.git;a=commit;h=341bb57b196f94cdb4cb91f9bce1352fe50565b1
• http://hg.mozilla.org/integration/gaia-central/rev/b0252a5e2c49
• https://hg.mozilla.org/integration/b2g-inbound/rev/f0c288ecee4a
• https://tbpl.mozilla.org/?tree=B2g-Inbound&rev=c02b023766e9

Long-story-short: Even though we don't have a Gaia tree on tbpl.mozilla.org, we test the Gaia changes through the B2G-Inbound tree, hence, we take Gaia pushes into account for the monthly pushes report.

For more information, the B2G bumper bot was designed in this bug.

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Mozilla's pushes - March 2014

Here's March's monthly analysis of the pushes to our Mozilla development trees (read about Gaia merges at the end of the blog post).
You can load the data as an HTML page or as a json file.

TRENDS

March (as February did) has the highest number of pushes EVER.
We will soon have 8,000 pushes/month as our norm.
The only noticeable change in the distribution of pushes is that non-integration trees had a higher share of the cake (17.80% on Mar. vs 14.60% on Feb.).

HIGHLIGHTS

• 7,939 pushes
• NEW RECORD
• 284 pushes/day (average)
• NEW RECORD
• Highest number of pushes/day: 435 pushes on March, 4th
• NEW RECORD
• 16.07 pushes/hour (average)

GENERAL REMARKS

Try keeps on having around 50% of all the pushes.
The three integration repositories (fx-team, mozilla-inbound and b2g-inbound) account around 30% of all the pushes.

RECORDS

• March 2014 was the month with most pushes (7,939 pushes)
• March 2014 has the highest pushes/day average with 284 pushes/day
• February 2014 has the highest average of "pushes-per-hour" is 16.57 pushes/hour
• March 4th, 2014 had the highest number of pushes in one day with 435 pushes

DISCLAIMERS

• The data collected prior to 2014 could be slightly off since different data collection methods were used
• Gaia pushes are more or less counted. I will write a blog post about in the near term.

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Kiss our old Mac Mini test pool goodbye

Today we have stopped running test jobs on our old Revision 3 Mac Mini test pool (see previous announcement).

There's a very very long list of people that have been involved in this project (see bug 864866).
I want to thank ahal, fgomes, jgriffin, jmaher, jrmuizel and rail for their help on the last mile.

We're very happy to have finally decommissioned this non-datacenter-friendly infrastructure.

A bit of history

These minis were purchased back in early 2010 and we bought more than 300 of them.
At first, we run on them Fedora 12, Fedora 12 x64, Windows Xp, Windows 7 and Mac 10.5. Later on we also added 10.6 to the mix (if my memory doesn't fail me).

Somewhere in 2012, we moved the Mac 10.6 testings to the revision 4 new mac server minis and deprecated the 10.5 rev3 testing pool. We then re-purposed those machines to increase the Windows and the Fedora pools.

By May of 2013, we stopped running Windows on them.
During 2013, we moved a lot of the Fedora testing to EC2.
Now, we managed to move the B2G reftests and Firefox debug mochitest-browser-chrome to EC2.

NOTE: I hope my memory does not fail me

Delivery of the Mac minis (photo credit to joduinn)

Racked at the datacenter (photo credit to joduinn)

CHANGES: fixed small typo. Remove labels to prevent re-posting into planet feeds.

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Mozilla's recent CI improvements saves roughly 60-70% on our AWS bill

bhearsum, catlee, glandium, taras and rail have been working hard for the last few months at cutting our AWS bills by improving Mozilla RelEng's CI.

From looking at it, I can say that with the changes they have made we're roughly saving the 60-70% on our AWS bill.

If you see them, give them a big pat on the back, this is huge for Mozilla.

Here’s some of the projects that helped with this:

• Jacuzzis
• Experiments with smaller pools of build
• Firefox builds are way cheaper now!
• EC2 spot instances experiments
• AWS, networks and burning trees
• More and faster CI for less on AWS
• Analyzed shared cache on try
• Cost-efficient continuous integration
• Linux and Android try builds, now up to twice as fast

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.