"try-server" and automated testing

This semester I am switching to a new project that will allows to learn a lot aobut automated testing on BuildBot.

My problem right now is that I have been reading too much and will be difficult to structure this blog post and I didn't want to split it into several posts. Please bar with me

Context of the project
First things first, to put you into context:
Links to read:

• http://wiki.mozilla.org/Build:TryServer (Most important I believe)
• http://lxr.mozilla.org/mozilla/source/tools/buildbotcustom/ - Buildbot customization
  
• http://lxr.mozilla.org/mozilla/source/tools/buildbot-configs/testing/unittest/ - Unit testing source code
  
• http://wiki.mozilla.org/BuildbotTestfarm -
  
• http://developer.mozilla.org/en/docs/Build_Documentation - Build documentation
  
• http://buildbot.sourceforge.net/manual-0.7.5.html - Buildbot Manual
  
• http://wiki.mozilla.org/Buildbot - Mozilla Buildbot
• http://ted.mielczarek.org/code/mozilla/regression-search.html -
• http://db48x.net/regression-search/ -
  

People involved:

• Rob Campbell, Robert Helmer,Ben Hearsum,Robert Sayre,Mike Shaver, Adam Delyea, Dave Humphrey and Armen Zambrano G. (me ;)
  

Related Blog posts:

• buildbot “try” support by rhelmer

Concepts involved:

• Buildbot
• try-server
• These are the tests they run in Mozilla: 1)reftest, 2)mochitest, 3)tunit, crashtest (from which tunit aka "make check" seems to be the easiest)

Unit Testing in Mozilla

• http://developer.mozilla.org/en/docs/Mochitest -
• http://mochikit.com/ - Mochitest is based on Mochikit
• http://developer.mozilla.org/en/docs/Mozilla_automated_testing - Mozilla automated test
• http://developer.mozilla.org/en/docs/Writing_MochiTest-based_unit_tests - Writting mochitest
• http://developer.mozilla.org/devnews/index.php/2006/11/06/toolkit-unit-testing/ - Unit testing for the Mozilla toolkit
http://mxr.mozilla.org/mozilla/source/layout/tools/reftest/README.txt
• - Reftest's README
• http://developer.mozilla.org/en/docs/How_to_add_a_build-time_test - How to add build time test
• http://wiki.mozilla.org/MozillaQualityAssurance:Test_Suite_and_Tool_Inventory - QA: Test Suite and Tool Inventory

NOTE from Writing_xpcshell-based_unit_tests:
"If you want to execute the test, you must create a non-static build of the browser without --disable-tests"

How do I run Mochitests?

• cd ($OBJ_DIR)/_tests/testing/mochitest (Note that the working directory matters for runtests.pl!)
• perl runtests.pl --autorun

How do I run Tunits ("make check")?

• cd ($OBJ_DIR)
• make check (see the check target in rules.mk - http://mxr.mozilla.org/seamonkey/source/config/rules.mk#1877 )
  

On http://tinderbox.mozilla.org/showbuilds.cgi?tree=Firefox I read on the left side this:
unit test Builds source then runs test suites. Build only after checkin, buildbot, VM, QA administered (robcee).
and from what I see in the page, they dedicate some servers to build and test while in the other they just build and others they do performance test