Tuesday, September 17, 2013

Running *hidden* Firefox for Android 4.2 x86 tests

As of last night, we're running *hidden* the first two sets of Android 4.2 x86 test jobs on every Gecko 27 tree on tbpl:
https://tbpl.mozilla.org/?jobname=Android 4.2 x86&rev=2520866d5874&showall=1
More sets will be added as we green the test suites on Cedar.
Image from http://en.wikipedia.org/wiki/File:Android-x86.png
Side topic, if you have an Android x86 or Armv6 device and want to test Firefox for it please give it a shot in https://play.google.com/store/apps/details?id=org.mozilla.firefox_beta&hl=en

Back to the topic, there are various aspects of the Android x86 testing which are different from the rest of our mobile automated testing, here's a brief list:

  • We don't run them on a device like a Tegra board or a Panda board
    • We run them on Linux 64-bit in-house machines (we can't use AWS)
    • No foopies involved
  • We run them under the Android emulator for the x86 architecture
  • We test on Android 4.2 rather than on Android 2.2 or 4.0
  • We run them in parallel: four emulator instances at most
  • An Android x86 job on tbpl is called a "set"
    • All Android x86 jobs will be grouped under "Sets" rather than per suite category grouping (e.g. reftest or mochitest)
    • A set can run test suites from any category (mix mochitests with reftests)
Here's how it looks on tbpl:

From a developer's perspective here's what you need to know:
  • The jobs are currently not running visibly on tbpl
  • Android x86 is on its way to become a tier-1 platform
  • You can borrow a releng machine to run the tests on them
    • You will be able to run it on your machine if this bug gets fixed
  • We're greening more suites on Cedar and we will add them to other tbpl development trees
  • We could potentially be adding another 7 to 11 sets in the next few weeks
  • The suites on each set are currently in-flux as we move suites from orange sets to green sets
  • Android x86 testing will ride the Gecko 27 trains

    In a following blog post I will go into the technical details on how these tests are run by mozharness.

    All of this work happened under bug 895186.

    If you want to contribute on how to make them easily runnable outside of a releng machine please visit this bug.

    After several months of development, big thanks (sorted alphabetically) to aki, Callek, dminor, gbrown, rail, RyanVM and ted.

    We still have some work to go but it is good to capture this small step towards victory.


    Creative Commons License
    This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.