Wednesday, January 30, 2008

armenzg: Sess9 - Test integration build steps

Building with Buildbot requires to set up a file master.cfg with instructions to be sent to each buildslave to build or do whatever you want.

These steps are constructed by using the method addStep() of a BuildFactory and it goes like this:
myFactory = facory.BuildFactory()
myFactory.addStep( ........) //You do it multiple times until you have all steps and then...

// You set up a Builder for that build slave
firefox_trunk_centos5_builder = {
'name': "Linux qm-centos5-01 dep unit test",
'slavenames': ['linux'],
'builddir': "trunk_centos5",
'factory': myFactory
'category': "Firefox"}
What I am going to show you now is the steps required to do a build with running different test suites, but instead of the code I will just mention the classes passed to the addStep() method and where that step is executed (there are more attributes and parameters, if you want to see the full steps click on this link):

NOTE: If not specified the working directory is: "mozilla"
  • MozillaCheckoutClientMk workdir="."
  • FileDownload //To get the .mozconfig file
  • ShellCommand -> "cat .mozconfig"
  • Compile -> "make -f checkout"
  • MozillaClobber //I am not sure why Clobber is for
  • Compile -> "make -f build"

Up to here nothing special, at this point, after the build, we will have to run all of the tests:
  • 1) MozillaCheck -> workdir="mozilla/objdir"
  • 2) MozillaUnixReftest -> workdir="mozilla/layout/reftests"
  • 3) MozillaUnixCrashtest -> "mozilla/testing/crashtest"
  • ShellCommand -> "rm places.sqlite" workdir="/home/buildbot/.mozilla/firefox/vrttezm8.default" //I do not know why this step, maybe this master.cfg was specified to unit test with PLACES enabled
  • 4) MozillaMochitest -> workdir="mozilla/objdir/_tests/testing/mochitest"
  • 5) MozillaMochichrome -> workdir="mozilla/objdir/_tests/testing/mochitest"
  • 6) MozillaBrowserChromeTest -> workdir="mozilla/objdir/_tests/testing/mochitest"
  • And that's it!
You also pass to each test step the parameter env=MozillaEnvironment['centos'] that contains all environment variables for this specific builder and its specific build steps.

Note about the classes passed to addStep()
All of the classes passed to the addStep() method inherit from the class ShellCommand, I will show you:
  • In we have this line:
    from buildbot.process.step import ShellCommand
  • Then when we define one of the classes:
    class MozillaReftest(ShellCommand): //I believe this indicates inheritance in Python
    class MozillaUnixReftest(MozillaReftest): //This is a subclass of MozillaReftest, which is used in one of our steps
  • The test classes define these two methods, which are to evaluate the command and to create a summary:
    def createSummary(self, log): //This method calls addCompleteLog(), which I believe sends a summary to the status of the build
    def evaluateCommand(self, cmd):


After this blog post, I am closer to understand every step required to run all automated tests but I still have to understand how this will work together with a "try server" set up, since what I have is an automated build with dependencies and the "try server" works upon check in and I believe it has to be without dependencies.

Tuesday, January 22, 2008

armenzg: Sess7 - Build Cairo

Another project that is really similar to the one I am doing with Mozilla is to set up a Buildbot Farm to build the latest Cairo code (Mozilla relies on Cairo), to run unit testing tests and to run performance tests.
So far what I have done is to try to build manually Cairo in Linux Ubuntu 7.1

I can say that I did not find instructions on how to build Cairo over the internet and neither on Cairo's website (the closest thing is and something about building cairomm but I can't find the URL anymore). Maybe there is some info in here,

NOTE: Candy bar at the bottom of this post

Anyways, these are the steps I manage to do by the help of behdad, ssp (from freenode #cairo) and cesar (from #seneca)
NOTE = I don't have most of the concepts behind building things on Linux, which might be obviuos to a lot of Linux developers

Get the source code:
  • sudo apt-get install git-core //To get the GIT Version Control
  • git clone git:// //To get the latest source code of Cairo - Checkout 2m43s
  • git clone git:// //Cairo cannot be built without this project installed - Checkout 3sec

Some instructions that will apply for Builbot configuration (I think)
  • makedir ~/buildslave/base TO BE EXPLAINED
  • PKG_CONFIG_PATH=~/buildslave/base/
NOTE: I read this in an email I got but I am not sure why or if I did it right

Let's build pixman before cairo
  • cd to where I have pixman
  • I tried to run ./configure as it said in pixman's INSTALL file but nothing
  • I tired to run ./ but it said that could not find autoreconfig or something like that
  • The next three steps were needed for me to be able to run ./
  • sudo apt-get install autoconf //as suggested in IRC
  • sudo apt-get install automake //from errors in screen
  • sudo apt-get install libtool //as suggested in IRC but could be induced from errors
  • ./ //FINALLY
  • ./configure
  • make
  • sudo make install
  • To see if you have the pixman library do this:
    whereis libpixman-1 //I think that it should work even before we run sudo ldconfig

Build Cairo
  • Then CD to where the Cairo source code is
  • ./
  • make
  • sudo make install

Let's run some tests and others
  • make test //Allmost all tests out of 147 have failed
  • One of the failed tests error: "/home/armen/sandbox/cairo/cairo/test/.libs/lt-xlib-surface: error while loading shared libraries: cannot open shared object file: No such file or directory"
  • cd test && make html //"Created index.html" - Just to see what will I get
  • sudo ldconfig // Suggested by Cesar - link to explanation
  • QUESTION: What is the difference if I do "make test" inside of test or one level above?
  • make test // A lot of graphics going on and 32 of 147 tests failed
  • make html //"Created index.html" - Just to see what will I get

The Candy bar
I have packed the index.html file with the png and the log files into a tar.gz file so you can see the results of what the tests of Cairo look like (Open index.html to see it)
I have packed the index.html file with the png and the log files into a tar.gz file so you can see the results of what the tests of Cairo look like.
NOTE: I have removed 300 objects from it to reduce the file size from 50 something MB to 9 MB.
NOTE2: You still have another 5000+ objects that I have not removed

armenzg: Sess8 - Mochitest, Hera server and project page

Writing posts can become really time consuming but it is a good way to don't loose some information you might have researched about.
For instance, this last Friday I had to choose between writing the steps to run the Mochitests and creating a project page for this semester's project. I chose to create the project's page and now that is Tuesday I just can't remember all the details of what I did with Mochitest but I know I run them!

Run Mochitests
As I said, I can't remember the exact steps but I believe that were as said in previous instructions:
  • cd ($OBJ_DIR)/_tests/testing/mochitest (Note that the working directory matters for!)
  • perl --autorun
I also remember that I saw a lot of pop up pages opening, doing something and then closing themselves (and each one of them is a mochitest)

AHA! As I was writing this post I realized that one of my questions got answered; In Mochitest#Running_Mochitest, says to run "perl" and that's what I did because I remember that after that a page opened and I had to click on a link saying "Run Tests!" and I thought "this does not look automated" and I have just found Mochitest#How_do_I_get_started.3F which shows that --autorun argument that I believe makes things automated.

List of things to note about Mochitest:

The Hera Try Server
As mentioned before we are setting up a Buildbot Try Server farm, therefore Dave moved IT people to let us have a Buildbot farm and we are going to configure to work as a Try Server with EC2 integration (Adam) and automated testing (me). This is the page in which is explained the Hera Try Server into detail ( ) and will be polish as soon as things get rolling

The project page
This last Friday, I created the project page for automated testing (which is called "Add to Try Server Automated Testing Support") instead of explaining the mochitest and I believe it was better this way to make consistent the project

Sunday, January 13, 2008

My initial plan for the project

After three days I can only have these initial goals and I have ordered in the order I have understood it should happen:
  1. Initially, I want to run manually the mochitests (one of the interactive types of tests) and the xpcshell based test units (aka "make check") - (This last one already accomplished)
  2. I have chosen tunit ("make check") to be the first one to be working with try-server capability
  3. Learn how to run the rest of different tests (reftest and crashtest)
  4. Integrate one by one the rest of the tests to be have try-server capabilities
  5. Add regression search Buildbot improvement (read note below)

NOTE: I believe that Tunit tests is just an abbreviation of Test Unit, I think is not something like Junit or Nunit

NOTE2: About == Regression Search Buildbot improvement ==

I [ setup a
buildbot] using [ db48x's
regression-search script] to make it easy to write a mochitest or
reftest and automatically find out when something regressed. Currently
it's still not terribly easy to use, and it's also hosted on my home
machine, so I often forget to restart it. This could be improved by
making it more reliable, and making it easier to use.

Sess5 - xpcshell-based unit tests - Tunits (aka "make check")

First of all, I have build a non-static build of firefox without --disable-tests

Running xpcshell-based unit tests (reading from Writing_xpcshell-based_unit_tests )
  • To run one test example, go to OBJDIR and run:
    make -C tools/test-harness/xpcshell-simple/example check
  • If you just want to run ONE MODULE, cd to the module or component under the OBJDIR folder like ($OBJDIR)/browser and run "make check" to run all tests of that module
  • To run all xpcshell based unit tests do:
    make check //which took around 4 mins in my laptop
    which runs all the tests in ($OBJDIR)/_tests/, inside there we have the "xpcshell-simple" folder with all the xpcshell-based unit tests
NOTE: If you want to the results of a "make check" have a look at this file (makeCheckResults.txt)

The xpcshell-simple folder, who is there and how does it look like
  • In that folder we have more folders that most of them have the prefix "test" (test_autocomplete, test_browser_feeds, test_intl_uconv and others...)
  • Inside each of these folders you will find out a "unit" folder or a "test" folder (which also contains one level down a "unit" folder)
  • Each "unit" folder contains javascript files that are the actual tests.
  • You can also realize that the tests can have this format, test_.js, which is normally (if not mistaken) the number of the associated bug being tested
  • Another thing is that you get the result of the test in a log file in the same place where a xpcshell test is with a file name like this: test_.js.log. These logs look like this:
    *** test pending
    *** test pending
    *** test finished
    *** running event loop
    *** test finished
    *** exiting
    *** PASS ***

    which are also the messages that you get when doing "make check"

  • To read more about how to write a test read this (Writing_xpcshell-based_unit_tests)

More info on how things get generated
When you do a "make -f build", the make arrives to the "" and read something like this (modules/libpref/test/ ):
  46 MODULE          = test_libpref
Therefore it creates a folder "test_libref" in the "xpcshell-simple" folder (under (OBJDIR)/_tests) and inside the "test_libref" folder it creates a folder "unit" with the exact content on the source (source/modules/libpref/test/ ), which is a bunch of tests written on javascript.

When we run "make check" from the ($OBJDIR), the make goes through each folder looking (I think) for all the files that have XPCSHELL_TESTS defined and runs the tests that are in ($OBJDIR)/_tests/xpcshell-simple/MODULE/XPCSHELL_TESTS

It is not always easy to map back to the original file by following the folder structure in the "_tests" folder, i. e. embed/tests/unit which doesn't map to source/embed/tests/unit but it maps back to source/embedding/tests/unit/

The best way is to look for one of the Javascript tests inside of the "unit" folder, i.e. the file _tests/xpcshell-simple/embed/tests/unit/test_wwauthpromptfactory.js by doing a search for file "test_wwauthpromptfactory.js" we find it in MXR by searching for the filename

"try-server" and automated testing

This semester I am switching to a new project that will allows to learn a lot aobut automated testing on BuildBot.

My problem right now is that I have been reading too much and will be difficult to structure this blog post and I didn't want to split it into several posts. Please bar with me

Context of the project
First things first, to put you into context:
Links to read:
People involved:
  • Rob Campbell, Robert Helmer,Ben Hearsum,Robert Sayre,Mike Shaver, Adam Delyea, Dave Humphrey and Armen Zambrano G. (me ;)
Related Blog posts:
Concepts involved:
  • Buildbot
  • try-server
  • These are the tests they run in Mozilla: 1)reftest, 2)mochitest, 3)tunit, crashtest (from which tunit aka "make check" seems to be the easiest)
Unit Testing in Mozilla
NOTE from Writing_xpcshell-based_unit_tests:
"If you want to execute the test, you must create a non-static build of the browser without --disable-tests"

How do I run Mochitests?
  • cd ($OBJ_DIR)/_tests/testing/mochitest (Note that the working directory matters for!)
  • perl --autorun

How do I run Tunits ("make check")?

On I read on the left side this:
unit test Builds source then runs test suites. Build only after checkin, buildbot, VM, QA administered (robcee).
and from what I see in the page, they dedicate some servers to build and test while in the other they just build and others they do performance test

Thursday, January 10, 2008

This is the beginning of the DPS911 course and I am getting ready for the project I will be working on.
NOTE: Not everything on this document will be specifically related to my project
NOTE2: Starting from a clean Ubuntu 7.1 (gutsy) installation

The public link to this document is

Sess1 - Getting Xchat build

I build from source code since it allows to download for Windows and Feedora (maybe it could work for Ubuntu but I want to be ready to build other things)
  • Extract the source code to /tmp
  • Inside x-chat folder run:
  • But got this error: configure: error: C compiler cannot create executables

  • Since I will build firefox and/or other things in this CLEAN Ubuntu Installation I do this:
    sudo apt-get build-dep firefox
    ./configure (the same problem)
  • Tried this (read from here ):
    sudo apt-get install libc6-dev
    ./configure (now the problem is "GLIB - version >= 2.0.3... no")
  • After a little research finding thins in the "synaptic Package Manager" (which is cheating for others). I saw few libglib style packages so I tried this:
    sudo apt-get install libglib2.0-dev
    ./configure (it ends by saying " 'make' and pray)
  • The praying did't work so I tried to go back to since I know Anthony Hughes was running Ubuntu and Xchat, therefore, it shouldn't be so complicated
  • I don't download the Fedora build but I try to go to their site, I download their 2.8.2, extract it to /tmp and try to run by doing this:
    and it replies with: "The program 'xchat' is currently not installed. You can install it by typing: sudo apt-get install xchat" which I try:
  • sudo apt-get install xchat
  • Now we can see packages like: tcl8.4 xchat xchat-common
  • I type "whereis xchat" which shows me the different place it now exists

  • Now add an icon to the panel above
  • Add a "Mozilla" network, edit the server to be "", select Use SSL and accept invalid certificates. Now for the #seneca channel you need to set your nick with NickServ

Running and working!

Sess2 -