Releng contributors from Seneca: Mozharness and buildapi

When I wrote my previous post trying to get students to contribute to Release Engineering I did not expect to have up to 8 students getting involved with us. 4 of them will be involved with the Fedora-Mozilla project.

This quarter we are going to have students contributing to the following projects:

• Add Firefox nightly/beta updates for Fedora - bug 600317
• Contribute to mozharness
• Contribute to buildapi
• Maybe some old and simple bugs

It is going to be very interesting to work with so many students as I have to be careful not be dragged down but at the same time I believe it is very valuable to both the students and our team.

You can ignore the rest of the post if you are not one of the students. Unless you are very curious to see what their options/projects are going to be.

I am going to extend a little bit on what the last 3 points involve. This should get adrianp, mustafaj, asingh, jing yan started. Please congratulate them when you see them on an IRC channel for taking up on the challenge.

To understand how our systems work checkout this new hire brownbag.

Mozharness
Mozharness is a new idea which is still being baked by aki. The goal is to pull much of the build logic that is currently intertwined on the buildbot factories (more than 40 different factories and we have passed 8 thousand lines in process/factory.py). The benefits of doing this is that it would become a standalone project which would be very easy to contribute to and run locally without having to setup buildbot.
Some problems would be that much of the API still on the works but at the same time it has a lot of value for students to face real life development and not school isolated development.

adrianp and mustafaj will be planning their development in here:
* http://zenit.senecac.on.ca/wiki/index.php/Mozharness
and here is the repo to work against:
* http://hg.mozilla.org/users/asasaki_mozilla.com/mozharness

Here are some thoughts on where this project could get some love:

• logging options
• documentation
• add new scripts
• do a locale repackage (easy)
• setup the right environment
• download en-US nightly
• repackage with the right locale and strings
• run packaged unit tests
• download binaries and packaged tests and run them
• generating MAR files (updates)
• signing
• give love to mozharness/scripts/partner-repacks.py
• porting hgtools.py to mozharness as a supporting library
• some POCs (proof-of-concepts)

Tip on writing new scripts:

• Use the two mozharness existing scripts to understand what format to follow
• To find out what steps are used for a certain type of job you have two places to check:
• Look for logs on tinderbox. There is one for L10n as well. Each step starts with
  "======== BuildStep started ========"
• TinderboxPrint steps are metadata to show on the tinderbox boxes to be scrapped by other tools (which slave the job run on, changeset used, etc)
• Look at process/factory.py and ask a releng which script you are trying to write and to which factory it corresponds. For instance, L10n repacks corresponds to BaseRepackFactory and NightlyRepackFactory. NOTE that there is inheritance involved.

Thoughts on logging:

The most standalone item I could find in mozharness is fleshing out the logging options/todo items:

• http://hg.mozilla.org/users/asasaki_mozilla.com/mozharness/file/default/lib/log.py 
• http://hg.mozilla.org/users/asasaki_mozilla.com/mozharness/file/default/test/test_log.py

Namely:
 - network logging
 - ability to change log settings mid stream?  Meaning, can we have a logger that's set to INFO level; can we set it to DEBUG halfway through the script?  If not, that's ok, but I'd like to know.
 - per-module/class log settings: e.g. can we have anything that goes through a Mercurial object to be DEBUG with a timestamp, and everything else be INFO with no timestamp?
 - turn off global logging settings:  when I create two log objects, I think I end up getting duplicate log lines on screen and in logfiles, even if I'm only calling one of them.
 - generic log rotation that is configurable


Hopefully the patches would end up being generic and reusable, but optional.  For instance, if they created the SimpleNetworkLogger and MultiNetworkLogger objects that could be drop-in replacements for SimpleFileLogger or MultiFileLogger, that would be awesome.

Problems faced by aki:

I need to figure out how to keep the "generic harness" (anyone at any company using python scripts could find them useful, like log.py, config.py, errors.py, script.py) separate from the Mozilla-specific code (l10n.py or repack.py, aus2.py, talos.py, etc.) 

Porting hgtools.py

mozharness and hgtool.py will need to communicate somehow.
I was going to just call hgtool.py from commandline in AbstractMercurialScript, but Catlee wants to merge it in.
So I'd lean towards creating a generic mozharness/lib/source/mercurial.py that has generic methods.

(Generic meaning we would hopefully later be able to create a mozharness/lib/source/git.py, or a mozharness/lib/source/svn.py, and be able to use them interchangeably.  SOURCEMODULE.checkout() for example, where SOURCEMODULE is any of the above.)
We can take a lot of that from work already done in Buildbot's Mercurial/Git/etc. steps, but remove any Buildbot dependencies.  And of course Catlee has most of it written already in hgtool.py.

Thoughts on POC:

• Here are some thought of aki but I think POCs will make more sense a little later when we understand where we want to take mozharness to. aki's thoughts are good but involve setting buildbot up which I want students to avoid at first. We'll leave it for now.

I'd love to see that.  Maybe take a small project and try it in mozharness?
How about something like porting buildbot-configs/test-masters.sh + setup_master.py to mozharness?

Like |checkconfigMasters.py --list-masters| or |checkconfigMasters.py -8 --only-builder-masters| or something.

That should be a relatively small project, but potentially useful.  We should end up with something way more configurable, and verify that you know your way around the harness.
Open to other ideas for a first project too... whatever POC script should hopefully be doable in a day or two.

BuildAPI
ssalbiz (our current kick-ass intern) has taken some time to prepare instructions for our new students on how to setup BuildAPI locally.
I still scratch myself and try to understand why it is called BuildAPI because this part of the project is more of a web-based project. It is based on pylons (python web framework) and it also uses the Google Chart Tools API (I think).

asingh and jing yang would probably coordinate this in here:
http://zenit.senecac.on.ca/wiki/index.php/BuildAPI

What I need students to do is one of the two:

• generate graphs, charts, CSVs and CPU totals for infrastructure load blog posts like this
• this is very useful and could move us forward towards having this information being published publicly for consumption
• I highly encourage this one as understanding the mental model behind it is easier
• write a tool that analyzes our statusDB and figure out slaves that have been continually been burning jobs (sometimes it takes us several days to spot them)

They can get started by pulling the repo and these snapshots from August:

• http://hg.mozilla.org/build/buildapi
• https://wiki.mozilla.org/ReleaseEngineering/BuildAPI
• http://people.mozilla.org/~catlee/statusdb.sql.bz2
• http://people.mozilla.org/~catlee/schedulerdb.sql.bz2

Simple releng bugs
Out of the list of bugs that we have tagged as [simple] on the Whiteboard I have pulled the following bugs:

• bug 563941 - Fake signing for staging releases
• bug 437482 - create Mercurial bundles
• bug 510770 - make source-package
• bug 586664 - Normalize builder names
• bug 563939 - Staging release should download previous release to staging-stage
• bug 577696 - Automate sending an email to metrics team once a release is pushed live
• bug 590329 - fail early if the clobberer is broken (at least for releases) 

This work could be coordinate in here:
http://zenit.senecac.on.ca/wiki/index.php/Release_Simple_Bugs#Project_Description

NOTE: All of these bugs would require you to setup buildbot and use Dummy factories to skip certain parts of our automation. Three or four of these bugs could count as a one student project. Less than that there is no value.

[EDIT] I have fixed a couple of URLs

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.