armenzg: Sess14: Building Cairo on Mac; No problems?? No way!!

It has been quite a long time and time constraints are always flying on top of me (like most of you reading this)
Anyways, I am going to work on writing the steps to build Cairo on Mac, let's see if there are differences

NOTE: This has been my longest session when I expected to spend just an hour or two - I have found so many obstacles that I cannot recall all of them

NOTE2: No performance tests and the "make test" takes a pretty LONG TIME - More than 30 minutes (In Linux I believe it took shorter time, not sure)

Requirements

• sudo port install git-core
• sudo port install buildbot

svn is not in ports' repository (I think I just saw svn extensions for python and perl) so I can get if with Fink or get an installer (Subversion .dmg); I decide to go the command line way:

• sudo port install autoconf
• sudo port install automake
• sudo port install libtool //I think I did: fink install libtool instead

Set-up

• mkdir sandbox && cd sandbox
  
• buildbot create-master cairomaster
• cd cairomaster
• I tried to checkout buildbot-configs/cairo-one-ubuntu-slave/master.cfg from my repository but svn does not checkout single files; But I placed the master.cfg to the ~/cairomaster folder
• buildbot create-slave ./cairoslave localhost:9876 macslave slavepassword
  

Running

• buildbot start cairomaster/
• buildbot start cairoslave/
• firefox http://localhost:8020
  

Conclusion

One of my longest sessions and with more problems ever!!!! Read below all my obstacles and I hope that all that I have done will help the cairo developers, in fact, cworth from the cairo channel was really helpful and he made some changes to the INSTALL file to reveal some of the things I have discovered. I love the new section called "Extremely detailed build instructions" that he has added :D

Obstacles

The autoconf and automake that come with MacOS X are not good for what I am trying to do (http://osdir.com/ml/finance.quantlib.devel/2005-09/msg00001.html ); I don't know if both or just one of them, but read below:

• sudo port install automake // because when I try to do .autogen.sh I get the following error (even though I have libtool installed):
  NOTE2: later when trying to ./autogen.sh in cairo's folder I had a similar problem and I got libtool from Fink since MacPort was skipping since I already had libtool; After that it worked
  

  Can't exec "libtoolize": No such file or directory at /usr/share/autoconf/Autom4te/FileUtils.pm line 288, line 3.
  autoreconf: failed to run libtoolize: No such file or directory

• I don't know what I did, I might have got autoconf from Fink or MacPorts // when I try to run ./autogen.sh with cairo I get this error:
  

  ./autogen.sh: line 177: libtoolize: command not found
  

• I did not know how to configure PKG_CONFIG_PATH but now I do:
  PKG_CONFIG_PATH=/Users/armenzg/Sandbox/libs/lib/pkgconfig //since I configured pixman to build there and it created a pkgconfig folder with a .pc file
  export PKG_CONFIG_PATH // to export to the environment variables
  
• After setting up the PKG_CONFIG_PATH properly and tried again to ./autogen.sh:
  

  configure: error: Cairo requires at least one font backend.
  Please install freetype and fontconfig, then try again:
  http://freetype.org/ http://fontconfig.org/

• I ended up doing "sudo fink install fontconfig" because neither through Fink or MacPorts I got it working
  

• Another problem, after doing a "make":
  

  libtool: ltconfig version `' does not match ltmain.sh version `1.3.5'
  Fatal configuration error. See the libtool docs for more information.
  make[2]: *** [libcairo_la-cairo.lo] Error 1

• The libtool that comes with MacOS X is not the one we want (thanks cworth from #cairo channel on freenode):
  LIBTOOLIZE=glibtoolize ./autogen.sh --prefix=/Users/armenzg/Sandbox/libs
• This last problem was found when I wrote the buildbot steps and I was trying to set environment variable PKG_CONFIG_PATH but that is not Shell Command and I found through bhearshum that I can do this:
  

  macFactory.addStep(step.ShellCommand, name="autogen for cairo",
  descriptionDone="autogen for cairo",
  command=["./autogen.sh","--prefix="+mac_build_prefix+"/lib"],
  workdir="cairo",
  env={"PKG_CONFIG_PATH": '%s/lib/lib/pkgconfig' % mac_build_prefix,
  "LIBTOOLIZE": 'glibtoolize'})
  

armenzg: Sess13: Buildbot with Cairo

In this post you will see that I have set up a master.cfg file to build cairo and run the tests BUT not the performance tests

My results:

• master.cfg
• buildbot-status-cairo.jpg - After build #8 (no tests at this point)
• Buildbot:Cairo-results.tar.gz - After build #11 (tests were run)
  

What the buildbot steps do is basically this:

• Remove folders "pixman" and "cairo"
• Checkout from GIT repository "pixman" and "cairo"
• Build pixman
• ./autogen.sh
• ./configure --prefix=/home/armen/lib //This is to avoid to do "sudo make install"
  
• make
• make install
• Build Cairo
• ./autogen.sh
• ./configure --prefix=/home/armen/lib
• make
• make install
• Run Cairo's Tests
• make test
• cd test && make html //This generates html file that displays images, all together is around 35MB
  
• Run Cairo's performance tests
• TO BE DONE
  

QUESTIONS:

• How do I pack the results and provide it? Upon request? After each build?
• PKG_CONFIG_PATH tells pkg-config where to look for information about packages and we use pkg-config to locate things like glib and cairo.
  My question is when am I supposed to run export PKG_CONFIG_PATH=/home/armen/lib?? //I don't have it in my master.cfg
• I still have to run the performance tests and see what happens
  

TO REMEMBER:

• "Better support for git" ticket http://buildbot.net/trac/ticket/130

EXTRA, BUT DO NOT READ:

FROM THIS POINT ON, these steps are my first attempt and you should not waste your time reading it, but it is good for learning purposes


This is a FRESH manual build of Cairo, the steps to set up the environment are ommited.
I have a problem/question relating that I have to raise privileges to do "make install", will this be OK with buildbot? Is there a way around?

• mkdir cairo & cd cairo
• git clone git://git.cairographics.org/git/pixman
• git clone git://git.cairographics.org/git/cairo
• cd pixman && ./autogen.sh
• ./configure
• make
• make install //I have a problem can I set up buildbot to use SUDO??
  

  make[2]: Entering directory `/home/armen/sandbox/cairo2/pixman/pixman'
  test -z "/usr/local/lib" || /bin/mkdir -p "/usr/local/lib"
  /bin/bash ../libtool --mode=install /usr/bin/install -c 'libpixman-1.la' '/usr/local/lib/libpixman-1.la'
  /usr/bin/install -c .libs/libpixman-1.so.0.9.6 /usr/local/lib/libpixman-1.so.0.9.6
  /usr/bin/install: cannot remove `/usr/local/lib/libpixman-1.so.0.9.6': Permission denied
  make[2]: *** [install-libLTLIBRARIES] Error 1
  
  

• sudo make install //I had to use SUDO
  
• whereis libpixman-1
• cd ../cairo && ./autogen.sh
• ./configure
  
• make
• make install //the same problem with SUDO
  

  make[2]: Entering directory `/home/armen/sandbox/cairo2/cairo/src'
  test -z "/usr/local/lib" || /bin/mkdir -p "/usr/local/lib"
  /bin/bash ../libtool --mode=install /usr/bin/install -c 'libcairo.la' '/usr/local/lib/libcairo.la'
  /usr/bin/install -c .libs/libcairo.so.2.14.0 /usr/local/lib/libcairo.so.2.14.0
  /usr/bin/install: cannot create regular file `/usr/local/lib/libcairo.so.2.14.0': Permission denied
  

• sudo make install

• make test //takes long time
  
• cd test && make html

armenzg: Sess12: Automated tests running and getting results

As you know from last post I have set up buildbot on my laptop to build Firefox and run automated tests every 50 mins.
I started to build properly around midnight when BUILD #5 started which was completed successfully.

• I have uploadded the log files for the BUILD #6 which are text files that for the tests we have "stdio" output and "summary" output (which is pretty poor) - buildbot-files/build6_results.tar.gz
• I have uploaded the buildbot's activity (the links point to the local log files as mentioned before) - buildbot-files/buildbot-waterfall.tar.gz
  
• These are the master.cfg and other files for the buildmaster - buildbot-config/BuildbotMasterArmen0.5.tar.gz
  

Let me talk about the results of BUILD #6(which has already build once on BUILD#5), #7 & #8:

	BUILD #6	BUILD #7	BUILD #9
checkout and compile	2-3 mins	2-3 mins	2-3 mins
MAKE CHECK	1hr 8mins	1hr 20mins	42 mins
REFTEST	10mins	20mins	20mins
CRASHTEST	4mins	1min	1min
MOCHITEST	22mins	31mins	30mins
MOCHICHROME	6mins	11mins	12mins
BROWSER CHROME	2mins	2mins	2mins
TOTAL	~2hr05mins	~2hr28mins	~1hr50mins

I have marked in red those values that are over 20mins
We can easily notice that testing takes A LOT OF TIME:

• Estimated time for all 6 tests: 2 hours
• DEP BUILD takes 2-3mins
  
• a CLEAN CHECKOUT & CLEAN BUILD takes 41mins (value comes from BUILD# 5)

COMMENTS

• The buildbot's waterfall view is really interesting and allows you to view how the build is going, but the summaries of the tests are really poor and were meant to output on Tinderbox, therefore the "
  " that are shown when looking the summaries of the tests
  

PROBLEMS

• I had FF2 open to see the results of the build and when it reached the mochitests it required to open FF3, therefore, there was an alert saying that Firefox was open bla bla bla.
• At that point, I thought that it waits for me to press "OK" and then run the test but if so, how did I got the results during of builds 6, 7 & 8 if I was sleeping?
• I remember that during the night time I woke up and I had to select "OK" and it seems that only after that it run
• I have tried to modify the "perl runtest.pl" step that runs the mochitest by adding "--setenv=MOZ_NO_REMOTE=1" or "--appname='../../../bin/dist/firefox --no-remote" but this did not make any change (BTW I restarted to be able to do apply the changes) and I have also tried to run it manually and I would still get that alert that would wait for a user "OK"

QUESTIONS

• shall I try to set up a "try server" style set up on my computer? Adam suggested not too, because he will take care of doing it for the Hera cluster
• would it be interesting to run tests in parallel? Let see what the timings are in the Hera cluster, I bet is better than in my laptop. I believe it could be done if a slave would pass to other computers the BUILD OBJDIR over the net and give them the steps to run the MAKE CHECK (which takes the longest out of all tests) and get the results back
• which others things shall I add to my steps? Bonsai-poller? Allow clobbering?
• we should set up a way for the developer to get in their inbox the results of the build/test
  

In this week's iteration I do the same but with Cairo and see what Adam does with the Hera cluster and try to put my stuff there

armenzg: Sess11: How to set up Buildbot to build Firefox and run automated tests

I have modified the master.cfg file to be able to schedule build with unit tests every 50 minutes on my laptop.
I have reused this file http://lxr.mozilla.org/mozilla/source/tools/buildbot-configs/testing/unittest/master.cfg
and the modified files are posted here:

NOTE: I already have Buildbot set up from last blog post

• I checked out the buildbot-config for unit test firefox with places (which I am not going to use)
  cvs -d :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot co mozilla/tools/buildbot-configs/testing/unittest/
• I moved the checked out files to my ~/BuildbotMaster folder
• Made all needed changes to set up only one buildslave - These are the main configuration
  

  c['buildbotURL'] = "http://localhost:8010/"
  c['slavePortnum'] = 9989
  c['status'].append(html.Waterfall(http_port=8010, css="/home/armen/BuildbotMaster/waterfall.css"))
  c['bots'] = [("slave1", "slavepassword1")]
  c['schedulers'].append(Periodic(name="50 minutes build scheduler",
  builderNames=["Linux Ubuntu7.1 dep unit test"],
  periodicBuildTimer=50*60))
  firefox_trunk_ubuntu_builder = {
  'name': "Linux Ubuntu7.1 dep unit test",
  'slavenames': ['slave1'],
  'builddir': "trunk_ubuntu",
  'factory': ubuntuFactory,
  'category': "Firefox"}
  

• I have also set up in the buildbot master folder a mozconfig-armen file that create a non-static build and enables tests. This mozconfig-armen gets downloaded by the slave
  
• buildbot start ~/BuildbotMaster
• buildbot start ~/Buildslave
• I open http://localhost:8010 and I can see the results on it

THE PROBLEM
The problem was that it would never pass the step "client.mk update stdio", because it required human intervention asking to say "yes" or "no" about accepting certain key identifying the server (or something like that)
If I did this step manually would be:

• cvs -d :ext:unittest@cvs.mozilla.org:/cvsroot co mozilla/client.mk
  which got me this message: "Permission denied (publickey,gssapi-with-mic)"

Therefore, I decided to use this command instead:

• cvs -d :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot co
  

And I updated the master.cfg accordingly by changing it to:

• CVSROOT = ":pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot"

I restarted then the master by doing this:

• builbot restart ~/BuildbotMaster

Now all that is left is to leave the computer work on it and give me results tomorrow when I wake up.


Image 1. Before fixing CVSROOT
Image 2. After fixing CVSROOT

armenzg: Sess10 - Buildbot initial set up

NOTE: I am setting up the Buildmaster and the buildslave in the same machine
NOTE2: I am going to use the default master.cfg

• sudo apt-get install buildbot
• buildbot create-master ~/BuildbotMaster
• These are the values that I have modified in the master.cfg file and/or are relevant to create a buildslave:
  

  c['bots'] = [("slave1", "slavepassword1")
  c['slavePortnum'] = 9989
  

• To create the slave we have to follow this format:
  buildbot create-slave BASEDIR MASTERHOST:PORT SLAVENAME PASSWORD
• buildbot create-slave ~/Buildslaves localhost:9989 slave1 slavepassword1
  

  Creating info/admin, you need to edit it appropriately
  Creating info/host, you need to edit it appropriately
  Please edit the files in /home/armen/Buildslaves/info appropriately.
  //You just have to update your name and email address
  

• buildbot start ~/BuildbotMaster
  The buildmaster appears to have (re)started correctly.
• buildbot start ~/Buildslaves/
  The buildslave appears to have (re)started correctly.

After this I can see the buildbot status on http://localhost:8010 and I can try to force a build and ping my buildslave but unfortunately I can not make anything happen but I get "pending builds". It could be that I did not set up a branch name since I left it as NONE

Later on, I will try to have a more complexed master.cfg, maybe a "try server" set up.
Meanwhile, have a look at the screen shot of my attempt