armenzg: Hera Try Server Unit Tests status

The Mozilla@Seneca Hera Try Server has been able to run Unit Tests after uploading a patch during this week!

This is basically what it does:

• Upload a patch through thos form sendchange_test.cgi (You need credentials)
• The processchanges_test.pl in the build master checks the folder on the try server where the patch are and see if there is any one new. Right not it checks every 1 or 2mins but we will change this to incron (Upon new file creation - This means immediate response)
• The perl script copies all new patches locally and sends the buildbot a message "builbot sendchange"
• This triggers the buildbot to make the "tests" branch to get started
• It applies the patch to the latest source code, it builds, it uploads the build back to the try server through SCP and SSH (this allows the developer to download it) and it runs all 6 unit test suites against that build

Left to be done:

• Add incron for the processchanges_test.pl to run
• Put the waterfall view behind some security measures (We don't want random people to just "force builds" and loose CPU while we want identified users to be able to)
• Add ssh key pairs for the mac and the windows machine
• Create the slave on the windows machine
• Once done, clone all linux machines and the windows machines
• Modify the sendchange_test.cgi file to identify the user who uploaded the patch (right now, every patch says to be uploaded by "armenzg")
• Unify the sendchange_test.cgi form to allow users to upload patches and just get builds without "tests"
• It will be nice to notify the user when the builds are done OR when the unit tests are run (I do not know how)

Another idea - Regionalized builds

I believe that it would be a good project to allow the Hera server to spit back a regionalised build, using Rueen auto-localization tool. This obviously requires more research, but I believe it is viable project to try.

This post could have been much more technical and longer but I decided to KISS since I won't have time to spend time documenting until Tuesday or Wednesday.

armenzg: Where do I display the unit tests?

Unfortunately for me, the major system that I am implementing this semester has drained too many hours (~150hours accumulated) and that for being a SINGLE course is way too much. You can tell of this because the quantity of work I have generate for each release of my Mozilla project (integrate unit tests to Seneca's try server) is a one day worth of research and work. I expect that after this coming Wednesday (final run) I could rest of that project and dedicate myself to my Mozilla project and my upcoming exams.

For this release I will mention what I have tried and found out. Two major things that I have been working on and have to get done/fixed:

• The unit tests need a DISPLAY in which to show up. I tried to see what is going in the slaves and I have to do one of two things "get someone with root access to either set it to boot to runlevel 5, or do init 5 after it boots and that'll start up the desktop and you'll be able to log in via vmware console" OR "look up an Xvnc tutorial and figure it out (does not require root access :) )" (thanks a lot to rhelmer and I wish you the best!)
• Through "Seneca Buildbot Try Server - Unit Tests enabled (soon)"(for security reasons it needs credentials - Look at the screenshot) you will be able to upload a patch and will run the unit tests but for now I will have to see if the perl script is properly set up. So far I know that I can submit patches to the folder "patches" in the try server. This will have to be retrieved by the buildslave to apply it to the latest source code.
  

NOTE: If your force a build of the unit test column, you will see that I have removed a lot of the steps, just to be able to run the test as soon as possible

armenzg: Sess16 - Automated testing on Hera

This blog post announces that I have got my hands dirty adding the steps to do automated builds on buildbot with testings on the hera server.
I have got access this last past Tuesday and the last 4 weeks I was working on the buildbot configuration for CAIRO, and this is the summary of what I have tried to do:

• Access the buildmaster in hera.senecac.on.ca
• Modify the master.cfg to add the build steps for the linux machine
• Add mozbuild.py file and the mozconfig-linux-tests

My goal was to force a complete build with tests without ruining Adam's configuration (he set up the try server)

NOTE: This buildbot setup will allow to force builds with tests and builds without tests









NOTE: You can see the second yellow column that indicates that is building - check it yourself OR force a build (and go for a coffe)

Problems

As always, I keep a section for problems and solutions for future reference:

• I used "buildbot restart ." instead of "buildbot reconfig ." and therefore I did put the try server down :( - BTW, do not use "restart" if you want to just re-read master.cfg so your changes are applied, because instead of using the last working copy of master.cfg (as it says) it just dies and never starts again
• I had to put some of the environment variables (MozillaEnvironments['centos']) inside buildcustom/env.py file since it was defined in there and also in my code but did not catch it this way
  
• At some point my "mozconfig-linux" file contained CRLF and I would get "line 17 errors" when I had only 9 lines; I used file mozconfig-linux-tests to confirm and dos2unix mozconfig-linux-tests to fix it (thanks rhelmer) - I remember the "line" errors but I can't find the log that said it - When I do cat .mozconfig, you can actually see that there are extra new lines in the output)
• There are has been a lot of small problems that I have run into but not relevant

Conclusion

I am in the Hera server, I can make changes and I have got a build with automated testing.

What is left?

• Make the other machines to build too (I do not know if they are up and running)
  
• Learn how to make it build following the "tryserver" way (give a patch and run the build with tests)
• Find out why the compiling takes forever
  

armenzg: Sess12: Automated tests running and getting results

As you know from last post I have set up buildbot on my laptop to build Firefox and run automated tests every 50 mins.
I started to build properly around midnight when BUILD #5 started which was completed successfully.

• I have uploadded the log files for the BUILD #6 which are text files that for the tests we have "stdio" output and "summary" output (which is pretty poor) - buildbot-files/build6_results.tar.gz
• I have uploaded the buildbot's activity (the links point to the local log files as mentioned before) - buildbot-files/buildbot-waterfall.tar.gz
  
• These are the master.cfg and other files for the buildmaster - buildbot-config/BuildbotMasterArmen0.5.tar.gz
  

Let me talk about the results of BUILD #6(which has already build once on BUILD#5), #7 & #8:

	BUILD #6	BUILD #7	BUILD #9
checkout and compile	2-3 mins	2-3 mins	2-3 mins
MAKE CHECK	1hr 8mins	1hr 20mins	42 mins
REFTEST	10mins	20mins	20mins
CRASHTEST	4mins	1min	1min
MOCHITEST	22mins	31mins	30mins
MOCHICHROME	6mins	11mins	12mins
BROWSER CHROME	2mins	2mins	2mins
TOTAL	~2hr05mins	~2hr28mins	~1hr50mins

I have marked in red those values that are over 20mins
We can easily notice that testing takes A LOT OF TIME:

• Estimated time for all 6 tests: 2 hours
• DEP BUILD takes 2-3mins
  
• a CLEAN CHECKOUT & CLEAN BUILD takes 41mins (value comes from BUILD# 5)

COMMENTS

• The buildbot's waterfall view is really interesting and allows you to view how the build is going, but the summaries of the tests are really poor and were meant to output on Tinderbox, therefore the "
  " that are shown when looking the summaries of the tests
  

PROBLEMS

• I had FF2 open to see the results of the build and when it reached the mochitests it required to open FF3, therefore, there was an alert saying that Firefox was open bla bla bla.
• At that point, I thought that it waits for me to press "OK" and then run the test but if so, how did I got the results during of builds 6, 7 & 8 if I was sleeping?
• I remember that during the night time I woke up and I had to select "OK" and it seems that only after that it run
• I have tried to modify the "perl runtest.pl" step that runs the mochitest by adding "--setenv=MOZ_NO_REMOTE=1" or "--appname='../../../bin/dist/firefox --no-remote" but this did not make any change (BTW I restarted to be able to do apply the changes) and I have also tried to run it manually and I would still get that alert that would wait for a user "OK"

QUESTIONS

• shall I try to set up a "try server" style set up on my computer? Adam suggested not too, because he will take care of doing it for the Hera cluster
• would it be interesting to run tests in parallel? Let see what the timings are in the Hera cluster, I bet is better than in my laptop. I believe it could be done if a slave would pass to other computers the BUILD OBJDIR over the net and give them the steps to run the MAKE CHECK (which takes the longest out of all tests) and get the results back
• which others things shall I add to my steps? Bonsai-poller? Allow clobbering?
• we should set up a way for the developer to get in their inbox the results of the build/test
  

In this week's iteration I do the same but with Cairo and see what Adam does with the Hera cluster and try to put my stuff there

armenzg: Sess11: How to set up Buildbot to build Firefox and run automated tests

I have modified the master.cfg file to be able to schedule build with unit tests every 50 minutes on my laptop.
I have reused this file http://lxr.mozilla.org/mozilla/source/tools/buildbot-configs/testing/unittest/master.cfg
and the modified files are posted here:

NOTE: I already have Buildbot set up from last blog post

• I checked out the buildbot-config for unit test firefox with places (which I am not going to use)
  cvs -d :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot co mozilla/tools/buildbot-configs/testing/unittest/
• I moved the checked out files to my ~/BuildbotMaster folder
• Made all needed changes to set up only one buildslave - These are the main configuration
  

  c['buildbotURL'] = "http://localhost:8010/"
  c['slavePortnum'] = 9989
  c['status'].append(html.Waterfall(http_port=8010, css="/home/armen/BuildbotMaster/waterfall.css"))
  c['bots'] = [("slave1", "slavepassword1")]
  c['schedulers'].append(Periodic(name="50 minutes build scheduler",
  builderNames=["Linux Ubuntu7.1 dep unit test"],
  periodicBuildTimer=50*60))
  firefox_trunk_ubuntu_builder = {
  'name': "Linux Ubuntu7.1 dep unit test",
  'slavenames': ['slave1'],
  'builddir': "trunk_ubuntu",
  'factory': ubuntuFactory,
  'category': "Firefox"}
  

• I have also set up in the buildbot master folder a mozconfig-armen file that create a non-static build and enables tests. This mozconfig-armen gets downloaded by the slave
  
• buildbot start ~/BuildbotMaster
• buildbot start ~/Buildslave
• I open http://localhost:8010 and I can see the results on it

THE PROBLEM
The problem was that it would never pass the step "client.mk update stdio", because it required human intervention asking to say "yes" or "no" about accepting certain key identifying the server (or something like that)
If I did this step manually would be:

• cvs -d :ext:unittest@cvs.mozilla.org:/cvsroot co mozilla/client.mk
  which got me this message: "Permission denied (publickey,gssapi-with-mic)"

Therefore, I decided to use this command instead:

• cvs -d :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot co
  

And I updated the master.cfg accordingly by changing it to:

• CVSROOT = ":pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot"

I restarted then the master by doing this:

• builbot restart ~/BuildbotMaster

Now all that is left is to leave the computer work on it and give me results tomorrow when I wake up.


Image 1. Before fixing CVSROOT
Image 2. After fixing CVSROOT

armenzg: Sess10 - Buildbot initial set up

NOTE: I am setting up the Buildmaster and the buildslave in the same machine
NOTE2: I am going to use the default master.cfg

• sudo apt-get install buildbot
• buildbot create-master ~/BuildbotMaster
• These are the values that I have modified in the master.cfg file and/or are relevant to create a buildslave:
  

  c['bots'] = [("slave1", "slavepassword1")
  c['slavePortnum'] = 9989
  

• To create the slave we have to follow this format:
  buildbot create-slave BASEDIR MASTERHOST:PORT SLAVENAME PASSWORD
• buildbot create-slave ~/Buildslaves localhost:9989 slave1 slavepassword1
  

  Creating info/admin, you need to edit it appropriately
  Creating info/host, you need to edit it appropriately
  Please edit the files in /home/armen/Buildslaves/info appropriately.
  //You just have to update your name and email address
  

• buildbot start ~/BuildbotMaster
  The buildmaster appears to have (re)started correctly.
• buildbot start ~/Buildslaves/
  The buildslave appears to have (re)started correctly.

After this I can see the buildbot status on http://localhost:8010 and I can try to force a build and ping my buildslave but unfortunately I can not make anything happen but I get "pending builds". It could be that I did not set up a branch name since I left it as NONE

Later on, I will try to have a more complexed master.cfg, maybe a "try server" set up.
Meanwhile, have a look at the screen shot of my attempt

armenzg: Sess9 - Test integration build steps

Building with Buildbot requires to set up a file master.cfg with instructions to be sent to each buildslave to build or do whatever you want.

These steps are constructed by using the method addStep() of a BuildFactory and it goes like this:

myFactory = facory.BuildFactory()
myFactory.addStep( ........) //You do it multiple times until you have all steps and then...

// You set up a Builder for that build slave
firefox_trunk_centos5_builder = {
'name': "Linux qm-centos5-01 dep unit test",
'slavenames': ['linux'],
'builddir': "trunk_centos5",
'factory': myFactory
'category': "Firefox"}

What I am going to show you now is the steps required to do a build with running different test suites, but instead of the code I will just mention the classes passed to the addStep() method and where that step is executed (there are more attributes and parameters, if you want to see the full steps click on this link):

NOTE: If not specified the working directory is: "mozilla"

• MozillaCheckoutClientMk workdir="."
• FileDownload //To get the .mozconfig file
  
• ShellCommand -> "cat .mozconfig"
• Compile -> "make -f client.mk checkout"
• MozillaClobber //I am not sure why Clobber is for
• Compile -> "make -f client.mk build"

Up to here nothing special, at this point, after the build, we will have to run all of the tests:

• 1) MozillaCheck -> workdir="mozilla/objdir"
• 2) MozillaUnixReftest -> workdir="mozilla/layout/reftests"
• 3) MozillaUnixCrashtest -> "mozilla/testing/crashtest"
• ShellCommand -> "rm places.sqlite" workdir="/home/buildbot/.mozilla/firefox/vrttezm8.default" //I do not know why this step, maybe this master.cfg was specified to unit test with PLACES enabled
• 4) MozillaMochitest -> workdir="mozilla/objdir/_tests/testing/mochitest"
  
• 5) MozillaMochichrome -> workdir="mozilla/objdir/_tests/testing/mochitest"
  
• 6) MozillaBrowserChromeTest -> workdir="mozilla/objdir/_tests/testing/mochitest"
• And that's it!
  

You also pass to each test step the parameter env=MozillaEnvironment['centos'] that contains all environment variables for this specific builder and its specific build steps.

Note about the classes passed to addStep()
All of the classes passed to the addStep() method inherit from the class ShellCommand, I will show you:

• In mozbuild.py we have this line:
  from buildbot.process.step import ShellCommand
• Then when we define one of the classes:
  class MozillaReftest(ShellCommand): //I believe this indicates inheritance in Python
  class MozillaUnixReftest(MozillaReftest): //This is a subclass of MozillaReftest, which is used in one of our steps
• The test classes define these two methods, which are to evaluate the command and to create a summary:
  def createSummary(self, log): //This method calls addCompleteLog(), which I believe sends a summary to the status of the build
  def evaluateCommand(self, cmd):

Conclusion

After this blog post, I am closer to understand every step required to run all automated tests but I still have to understand how this will work together with a "try server" set up, since what I have is an automated build with dependencies and the "try server" works upon check in and I believe it has to be without dependencies.

armenzg: Sess8 - Mochitest, Hera server and project page

Writing posts can become really time consuming but it is a good way to don't loose some information you might have researched about.
For instance, this last Friday I had to choose between writing the steps to run the Mochitests and creating a project page for this semester's project. I chose to create the project's page and now that is Tuesday I just can't remember all the details of what I did with Mochitest but I know I run them!

Run Mochitests
As I said, I can't remember the exact steps but I believe that were as said in previous instructions:

• cd ($OBJ_DIR)/_tests/testing/mochitest (Note that the working directory matters for runtests.pl!)
• perl runtests.pl --autorun

I also remember that I saw a lot of pop up pages opening, doing something and then closing themselves (and each one of them is a mochitest)

AHA! As I was writing this post I realized that one of my questions got answered; In Mochitest#Running_Mochitest, says to run "perl runtests.pl" and that's what I did because I remember that after that a page opened and I had to click on a link saying "Run Tests!" and I thought "this does not look automated" and I have just found Mochitest#How_do_I_get_started.3F which shows that --autorun argument that I believe makes things automated.

List of things to note about Mochitest:

• Mochitests has nothing to be feared of!!
• When you check out the trunk, you also check out testing/mochitest by default
• Try it you self by this self explanatory and easy to follow README file!!
• When you create a new test with the template you can associate a BUG NUMBER - Mochitest#Writing_new_Mochitest_tests
  

The Hera Try Server
As mentioned before we are setting up a Buildbot Try Server farm, therefore Dave moved IT people to let us have a Buildbot farm and we are going to configure to work as a Try Server with EC2 integration (Adam) and automated testing (me). This is the page in which is explained the Hera Try Server into detail (http://zenit.senecac.on.ca/wiki/index.php/Hera_Try_Server_Setup ) and will be polish as soon as things get rolling


The project page
This last Friday, I created the project page for automated testing (which is called "Add to Try Server Automated Testing Support") instead of explaining the mochitest and I believe it was better this way to make consistent the project

My initial plan for the project

After three days I can only have these initial goals and I have ordered in the order I have understood it should happen:

1. Initially, I want to run manually the mochitests (one of the interactive types of tests) and the xpcshell based test units (aka "make check") - (This last one already accomplished)
   
2. I have chosen tunit ("make check") to be the first one to be working with try-server capability
   
3. Learn how to run the rest of different tests (reftest and crashtest)
4. Integrate one by one the rest of the tests to be have try-server capabilities
   
5. Add regression search Buildbot improvement (read note below)
   

NOTE: I believe that Tunit tests is just an abbreviation of Test Unit, I think is not something like Junit or Nunit

NOTE2: About == Regression Search Buildbot improvement ==

I [http://ted.mielczarek.org/code/mozilla/regression-search.html setup a
buildbot] using [http://db48x.net/regression-search/ db48x's
regression-search script] to make it easy to write a mochitest or
reftest and automatically find out when something regressed. Currently
it's still not terribly easy to use, and it's also hosted on my home
machine, so I often forget to restart it. This could be improved by
making it more reliable, and making it easier to use.

Sess5 - xpcshell-based unit tests - Tunits (aka "make check")

First of all, I have build a non-static build of firefox without --disable-tests

Running xpcshell-based unit tests (reading from Writing_xpcshell-based_unit_tests )

• To run one test example, go to OBJDIR and run:
  make -C tools/test-harness/xpcshell-simple/example check
• If you just want to run ONE MODULE, cd to the module or component under the OBJDIR folder like ($OBJDIR)/browser and run "make check" to run all tests of that module
  
• To run all xpcshell based unit tests do:
  make check //which took around 4 mins in my laptop
  which runs all the tests in ($OBJDIR)/_tests/, inside there we have the "xpcshell-simple" folder with all the xpcshell-based unit tests

NOTE: If you want to the results of a "make check" have a look at this file (makeCheckResults.txt)

The xpcshell-simple folder, who is there and how does it look like

• In that folder we have more folders that most of them have the prefix "test" (test_autocomplete, test_browser_feeds, test_intl_uconv and others...)
• Inside each of these folders you will find out a "unit" folder or a "test" folder (which also contains one level down a "unit" folder)
• Each "unit" folder contains javascript files that are the actual tests.
• You can also realize that the tests can have this format, test_.js, which is normally (if not mistaken) the number of the associated bug being tested
• Another thing is that you get the result of the test in a log file in the same place where a xpcshell test is with a file name like this: test_.js.log. These logs look like this:
  

  *** test pending
  *** test pending
  *** test finished
  *** running event loop
  *** test finished
  *** exiting
  *** PASS ***
  

  which are also the messages that you get when doing "make check"

• To read more about how to write a test read this (Writing_xpcshell-based_unit_tests)
  

More info on how things get generated
When you do a "make -f client.mk build", the make arrives to the "Makefiles.in" and read something like this (modules/libpref/test/Makefile.in ):

   MODULE          = test_libpref
   
   XPCSHELL_TESTS = unit

Therefore it creates a folder "test_libref" in the "xpcshell-simple" folder (under (OBJDIR)/_tests) and inside the "test_libref" folder it creates a folder "unit" with the exact content on the source (source/modules/libpref/test/ ), which is a bunch of tests written on javascript.

When we run "make check" from the ($OBJDIR), the make goes through each folder looking (I think) for all the Makefile.in files that have XPCSHELL_TESTS defined and runs the tests that are in ($OBJDIR)/_tests/xpcshell-simple/MODULE/XPCSHELL_TESTS

It is not always easy to map back to the original file by following the folder structure in the "_tests" folder, i. e. embed/tests/unit which doesn't map to source/embed/tests/unit but it maps back to source/embedding/tests/unit/

The best way is to look for one of the Javascript tests inside of the "unit" folder, i.e. the file _tests/xpcshell-simple/embed/tests/unit/test_wwauthpromptfactory.js by doing a search for file "test_wwauthpromptfactory.js" we find it in MXR by searching for the filename

"try-server" and automated testing

This semester I am switching to a new project that will allows to learn a lot aobut automated testing on BuildBot.

My problem right now is that I have been reading too much and will be difficult to structure this blog post and I didn't want to split it into several posts. Please bar with me

Context of the project
First things first, to put you into context:
Links to read:

• http://wiki.mozilla.org/Build:TryServer (Most important I believe)
• http://lxr.mozilla.org/mozilla/source/tools/buildbotcustom/ - Buildbot customization
  
• http://lxr.mozilla.org/mozilla/source/tools/buildbot-configs/testing/unittest/ - Unit testing source code
  
• http://wiki.mozilla.org/BuildbotTestfarm -
  
• http://developer.mozilla.org/en/docs/Build_Documentation - Build documentation
  
• http://buildbot.sourceforge.net/manual-0.7.5.html - Buildbot Manual
  
• http://wiki.mozilla.org/Buildbot - Mozilla Buildbot
• http://ted.mielczarek.org/code/mozilla/regression-search.html -
• http://db48x.net/regression-search/ -
  

People involved:

• Rob Campbell, Robert Helmer,Ben Hearsum,Robert Sayre,Mike Shaver, Adam Delyea, Dave Humphrey and Armen Zambrano G. (me ;)
  

Related Blog posts:

• buildbot “try” support by rhelmer

Concepts involved:

• Buildbot
• try-server
• These are the tests they run in Mozilla: 1)reftest, 2)mochitest, 3)tunit, crashtest (from which tunit aka "make check" seems to be the easiest)

Unit Testing in Mozilla

• http://developer.mozilla.org/en/docs/Mochitest -
• http://mochikit.com/ - Mochitest is based on Mochikit
• http://developer.mozilla.org/en/docs/Mozilla_automated_testing - Mozilla automated test
• http://developer.mozilla.org/en/docs/Writing_MochiTest-based_unit_tests - Writting mochitest
• http://developer.mozilla.org/devnews/index.php/2006/11/06/toolkit-unit-testing/ - Unit testing for the Mozilla toolkit
http://mxr.mozilla.org/mozilla/source/layout/tools/reftest/README.txt
• - Reftest's README
• http://developer.mozilla.org/en/docs/How_to_add_a_build-time_test - How to add build time test
• http://wiki.mozilla.org/MozillaQualityAssurance:Test_Suite_and_Tool_Inventory - QA: Test Suite and Tool Inventory

NOTE from Writing_xpcshell-based_unit_tests:
"If you want to execute the test, you must create a non-static build of the browser without --disable-tests"

How do I run Mochitests?

• cd ($OBJ_DIR)/_tests/testing/mochitest (Note that the working directory matters for runtests.pl!)
• perl runtests.pl --autorun

How do I run Tunits ("make check")?

• cd ($OBJ_DIR)
• make check (see the check target in rules.mk - http://mxr.mozilla.org/seamonkey/source/config/rules.mk#1877 )
  

On http://tinderbox.mozilla.org/showbuilds.cgi?tree=Firefox I read on the left side this:
unit test Builds source then runs test suites. Build only after checkin, buildbot, VM, QA administered (robcee).
and from what I see in the page, they dedicate some servers to build and test while in the other they just build and others they do performance test