Re-thinking our Mac OS X continuous integration testing infrastructure

tl;dr: On RelEng for OS X, we currently build on 10.7, and run tests on 10.6, 10.7 and 10.8. Now that Apple has shipped 10.9 as a free upgrade to 10.6.8, 10.7, 10.8 users, we’re planning to recycle our 10.7 and 10.8 test machines to grow our 10.6 pool, to better handle 10.6 load, and quickly start testing 10.9 in production.

Update: I've updated the caption for figure 1 reducing the comment to just 10.8 users rather than all versions.

=====

On Oct. 22nd, 2013, when Apple announced the release of Mac OS X 10.9, they did something new: OS X 10.9 would be available as a free upgrade to all 10.6.8, 10.7 and 10.8 users. It's less than a month since the launch, and you can clearly see the trend already.

In the graph below we can see Active Daily Installations of Firefox, and we can clearly see the increase of 10.9 users and the decline of 10.8, 10.7 and 10.6 users (the last two are expected to decline at a slower rate than 10.8) after Oct. 22nd:

Figure 1: ADI

In short: We see less installations from 10.8 users and a high spike from users on 10.9.

This graph shows the number of active daily users for Firefox on Mac OS X (rather than installations as above). We can also see how after Oct. 22nd, we got a huge 10.8 users reduction. This graph does not show 10.9 but on the previous graph we could see the huge spike for it.

Figure 2 - ADU

Another important piece of information to consider is that:

• Mac OS X 10.6 users have not received a security update since July 2011 [1]
• That's more than 2 years
• Mac OS X 10.7 users have not received a security release since October 2012 [2]
• That's more than 1 year

On another note, AUS data is showing that our Mac OS X users are broken down like this (data from the 15th):

• 10.6 - 34%
• 10.7 - 23%
• 10.8 - 21%
• 10.9 - 21%

NOTE: This is to make up for the ADU graph that does not show the 10.9 info.

Given that, we intend to do the following:

1. Re-purpose our 10.7 test machines as 10.6 test machines
1. These two versions run on the Revision 4 Minis which we can’t buy anymore
2. This will increase our 10.6 testing pool by 106% (80->165), so improving 10.6 wait times.
2. Re-purpose our 10.8 test machines as 10.9 test machines
1. This will let us quickly turn on 10.9 testing

This is not new. We did something similar when we decided to drop Vista as most of our users were on Windows XP and Windows 7.

We’re still working on the logistics, bugs will soon be filled. Please let us know if we have missing anything.

Note for press: This blog post does not mean that Firefox will stop running on Mac OS X 10.7 or 10.8. This post is about the automation we run for our developers.

For a list of the minimum system requirements for Firefox please read this page:

https://www.mozilla.org/en-US/firefox/25.0.1/system-requirements

NOTE: I removed the Y axis values from the graph screenshots just to make sure that I was not publishing something I should have not. At Mozilla we're encouraged to be open, however, I would not want to have to find who can give sign off about which data is or is not supposed to be public. I'm almost sure it is fine but I don't want to make the mistake. 

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

How to install Mozilla Releng's Buildbot on my Ubuntu machine

I've found myself many times in the last few years having to setup buildbot on a clean machine.
Setting buildbot up is easy, however, setting it up to match Mozilla's Release Engineering requires more work.

Install it inside of a virtual environment

Unfortunately, I never documented it. This is one of the many ways to be setup and I hope I haven't missed any steps:

mkdir repos && cd repos
hg clone http://hg.mozilla.org/build/buildbotcustom
hg clone http://hg.mozilla.org/build/tools
# add this to your .bashrc
# export PYTHONPATH=~/repos/tools/lib/python:~/repos/buildbotcustom:$PYTHONPATH
sudo apt-get install python-virtualenv libpython-devel libssl-dev
virtualenv ~/venvs
source ~/venvs/bin/activate
cd ~/venvs
pip install pyOpenSSL
hg clone http://hg.mozilla.org/build/buildbot && cd buildbot/master
hg up -r production-0.8
python setup.py install
# The previous step will install the latest Twisted which won't work
pip install Twisted=="10.1.0"

From here on you will need to use "source ~/venvs/bin/activate" before you can use buildbot.
You can also avoid having to use virtualenvs and install it system wide.

How can you know if it is working?

Check that you have the right versions:

$ buildbot --version
Buildbot version: 0.8.2-hg-5dda15da8f5d-production-0.8-hg-a3f25f0d508d+-default
Twisted version: 10.1.0

Also check that checkconfig returns success:

cd ~/repos/buildbot-configs
./setup-master.py master bm67-tests1-linux
cd master
buildbot checkconfig .

How to install through your distro's packaging system?

If you want to install the normal available buildbot through Ubuntu's package manager you just have to do this:
sudo apt-get install buildbot

You probably need libpython-devel as well as libssl-dev.

Update: added title "How to install through your distro's packaging system?"
Update: improved section "How can you know it is working?"

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.