Making mozharness easier to hack on and try support

Yesterday, we presented a series of proposed changes to Mozharness at the bi-weekly meeting.

We're mainly focused on making it easier for developers and allow for further flexibility.
We will initially focus on the testing side of the automation and make ground work for other further improvements down the line.

The set of changes discussed for this quarter are:

1. Move remaining set of configs to the tree - bug 1067535
• This makes it easier to test harness changes on try
2. Read more information from the in-tree configs - bug 1070041
• This increases the number of harness parameters we can control from the tree
3. Use structured output parsing instead of regular where it applies - bug 1068153
• This is part of a larger goal where we make test reporting more reliable, easy to consume and less burdening on infrastructure
• It's to establish a uniform criteria for setting a job status based on a test result that depends on structured log data (json) rather than regex-based output parsing
• "How does a test turn a job red or orange?" 
• We will then have a simple answer that is that same for all test harnesses
4. Mozharness try support - bug 791924
• This will allow us to lock which repo and revision of mozharnes is checked out
• This isolates mozharness changes to a single commit in the tree
• This give us try support for user repos (freedom to experiment with mozharness on try)

Even though we feel the pain of #4, we decided that the value gained for developers through #1 & #2 gave us immediate value while for #4 we know our painful workarounds.

I don't know if we'll complete #4 in this quarter, however, we are committed to the first three.

If you want to contribute to the longer term vision on that proposal please let me know.

In the following weeks we will have more updates with regards to implementation details.

Stay tuned!

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Which builders get added to buildbot?

To add/remove jobs on tbpl.mozilla.org, we have to modify buildbot-configs.

Making changes can be learnt by looking at previous patches, however, there's a bit of an art to it to get it right.

I just landed a script that sets up buildbot for you inside of a virtualenv and you can pass a buildbot-config patch and determine which builders get added/removed.

You can run this by checking out braindump and running something like this:
buildbot-related/list_builder_differences.sh -j path_to_patch.diff

NOTE: This script does not check that the job has all the right parameters once live (e.g. you forgot to specify the mozharness config for it).

Happy hacking!

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Run tbpl jobs locally with Http authentication (developer_config.py) - take 2

Back in July, we deployed the first version of Http authentication for mozharness, however, under some circumstances, the initial version could fail and affect production jobs.

This time around we have:

• Remove the need for _dev.py config files
• Each production config had an associated _dev.py config file
• Prevented it from running in production environment
• The only way to enable the developer mode is by appending --cfg developer_config.py

If you read How to run Mozharness as a developer you should see the new changes.

As quick reminder, it only takes 3 steps:

1. Find the command from the log. Copy/paste it.
2. Append --cfg developer_config.py
3. Append --installer-url/--test-url with the right values

To see a real example visit this. 

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.