Re-thinking our Mac OS X continuous integration testing infrastructure

tl;dr: On RelEng for OS X, we currently build on 10.7, and run tests on 10.6, 10.7 and 10.8. Now that Apple has shipped 10.9 as a free upgrade to 10.6.8, 10.7, 10.8 users, we’re planning to recycle our 10.7 and 10.8 test machines to grow our 10.6 pool, to better handle 10.6 load, and quickly start testing 10.9 in production.

Update: I've updated the caption for figure 1 reducing the comment to just 10.8 users rather than all versions.

=====

On Oct. 22nd, 2013, when Apple announced the release of Mac OS X 10.9, they did something new: OS X 10.9 would be available as a free upgrade to all 10.6.8, 10.7 and 10.8 users. It's less than a month since the launch, and you can clearly see the trend already.

In the graph below we can see Active Daily Installations of Firefox, and we can clearly see the increase of 10.9 users and the decline of 10.8, 10.7 and 10.6 users (the last two are expected to decline at a slower rate than 10.8) after Oct. 22nd:

Figure 1: ADI

In short: We see less installations from 10.8 users and a high spike from users on 10.9.

This graph shows the number of active daily users for Firefox on Mac OS X (rather than installations as above). We can also see how after Oct. 22nd, we got a huge 10.8 users reduction. This graph does not show 10.9 but on the previous graph we could see the huge spike for it.

Figure 2 - ADU

Another important piece of information to consider is that:

• Mac OS X 10.6 users have not received a security update since July 2011 [1]
• That's more than 2 years
• Mac OS X 10.7 users have not received a security release since October 2012 [2]
• That's more than 1 year

On another note, AUS data is showing that our Mac OS X users are broken down like this (data from the 15th):

• 10.6 - 34%
• 10.7 - 23%
• 10.8 - 21%
• 10.9 - 21%

NOTE: This is to make up for the ADU graph that does not show the 10.9 info.

Given that, we intend to do the following:

1. Re-purpose our 10.7 test machines as 10.6 test machines
1. These two versions run on the Revision 4 Minis which we can’t buy anymore
2. This will increase our 10.6 testing pool by 106% (80->165), so improving 10.6 wait times.
2. Re-purpose our 10.8 test machines as 10.9 test machines
1. This will let us quickly turn on 10.9 testing

This is not new. We did something similar when we decided to drop Vista as most of our users were on Windows XP and Windows 7.

We’re still working on the logistics, bugs will soon be filled. Please let us know if we have missing anything.

Note for press: This blog post does not mean that Firefox will stop running on Mac OS X 10.7 or 10.8. This post is about the automation we run for our developers.

For a list of the minimum system requirements for Firefox please read this page:

https://www.mozilla.org/en-US/firefox/25.0.1/system-requirements

NOTE: I removed the Y axis values from the graph screenshots just to make sure that I was not publishing something I should have not. At Mozilla we're encouraged to be open, however, I would not want to have to find who can give sign off about which data is or is not supposed to be public. I'm almost sure it is fine but I don't want to make the mistake. 

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.