Armen Zambrano's battlefield: Running *hidden* Firefox for Android 4.2 x86 tests

Tuesday, September 17, 2013

Running hidden Firefox for Android 4.2 x86 tests

As of last night, we're running *hidden* the first two sets of Android 4.2 x86 test jobs on every Gecko 27 tree on tbpl:
https://tbpl.mozilla.org/?jobname=Android 4.2 x86&rev=2520866d5874&showall=1
More sets will be added as we green the test suites on Cedar.

Image from http://en.wikipedia.org/wiki/File:Android-x86.png

Side topic, if you have an Android x86 or Armv6 device and want to test Firefox for it please give it a shot in https://play.google.com/store/apps/details?id=org.mozilla.firefox_beta&hl=en

Back to the topic, there are various aspects of the Android x86 testing which are different from the rest of our mobile automated testing, here's a brief list:

We don't run them on a device like a Tegra board or a Panda board

We run them on Linux 64-bit in-house machines (we can't use AWS)
No foopies involved

We run them under the Android emulator for the x86 architecture
We test on Android 4.2 rather than on Android 2.2 or 4.0
We run them in parallel: four emulator instances at most
An Android x86 job on tbpl is called a "set"

All Android x86 jobs will be grouped under "Sets" rather than per suite category grouping (e.g. reftest or mochitest)
A set can run test suites from any category (mix mochitests with reftests)

Here's how it looks on tbpl:

From a developer's perspective here's what you need to know:

The jobs are currently not running visibly on tbpl

see bug 917361 if you want to help

Android x86 is on its way to become a tier-1 platform
You can borrow a releng machine to run the tests on them

You will be able to run it on your machine if this bug gets fixed

We're greening more suites on Cedar and we will add them to other tbpl development trees
We could potentially be adding another 7 to 11 sets in the next few weeks
The suites on each set are currently in-flux as we move suites from orange sets to green sets
Android x86 testing will ride the Gecko 27 trains

In a following blog post I will go into the technical details on how these tests are run by mozharness.

All of this work happened under bug 895186.

If you want to contribute on how to make them easily runnable outside of a releng machine please visit this bug.

After several months of development, big thanks (sorted alphabetically) to aki, Callek, dminor, gbrown, rail, RyanVM and ted.

We still have some work to go but it is good to capture this small step towards victory.