Mozhginfo/Pushlog client released

Hi,
If you've ever spent time trying to query metadata from hg with regards to revisions, you can now use a Python library we've released to do so.

In bug 1203621 [1], our community contributor @MikeLing has helped us release the pushlog.py module we had written for Mozilla CI tools.

You can find the pushlog_client package in here [3] and you can find the code in here [4]

Thanks MikeLing!

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1203621
[2] https://github.com/MikeLing
[3] https://pypi.python.org/pypi/pushlog_client
[4] https://hg.mozilla.org/hgcustom/version-control-tools/rev/6021c9031bc3

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Welcome F3real, xenny and MikeLing!

As described by jmaher, we started this week our first week of mozci's quarter of contribution.

I want to personally welcome Stefan, Vaibhav and Mike to mozci. We hope you get to learn and we thank you for helping Mozilla move forward in this corner of our automation systems.

I also want to give thanks to Alice for committing at mentoring. This could not be possible without her help.

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Mozilla CI tools meet up

In order to help the contributors' of mozci's quarter of contribution, we have set up a Mozci meet up this Friday.

If you're interested on learning about Mozilla's CI, how to contribute or how to build your own scheduling with mozci come and join us!

9am ET -> other time zones
Vidyo room: https://v.mozilla.com/flex.html?roomdirect.html&key=GC1ftgyxhW2y

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Buildapi client released - thanks F3real!

When we wrote Mozilla CI tools, we created a module called buildapi.py in order to schedule Buildbot jobs via Self-serve/Buildapi.

We recently ported it as a Python package and released it:
https://pypi.python.org/pypi/buildapi_client

This was all thanks to F3real, who joined us from Mozilla's community and released his first Python package. He has also brought forth the integration tests we wrote for it. Here's the issue and  PR if you're curious.

F3real will now be looking at removing the buildapi module from mozci and making use of the python package instead.

Thanks F3real!

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Fixes for Sheriffs' Treeherder backfilling and scheduling support

Sheriffs have the ability to backfill jobs on Treeherder (to determine the push where are regression starts) and the ability to run missing jobs on a push (due to coalescing/SETA).

Few weeks ago we added alerts to pulse_actions:

• System exit calls where crashing pulse_actions and we did not notice

 and we've fixed a couple of bugs since then:

• Backfilling of a test job of pushes with coalesced builds should not test against the wrong build
• Thanks @KWierso for reporting it!
• Filling in a revision on aurora partially fails

Other changes:

• Better documentation
• Moved under the Mozilla Org on GitHub
• Added auto deploy to Heroku from the 'heroku' branch

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

mozci 0.15.1: Buildbot Bridge support + bug fixes

It's been a while since our last announced released and I would like to highlight some of the significant changes since then.
A major highlight for the latest release is the support to schedule Buildbot jobs through TaskCluster making use of the Buildbot Bridge. For more information about this please read in here.

Contributors

@nikkisquared is our latest contributor who landed more formalized exceptions and error handling. Thanks!
As always, many thanks to @adusca and @vaibhavmagarwal for their endless contributions.

How to update

Run "pip install -U mozci" to update

New Features

• Query Treeherder for hidden jobs
• BuildbotBridge support
• It allows submitting a graph of Buildbot jobs with dependencies
• The graph is submitted to TaskCluster

Minor changes

• Fix backfilling issue
• Password prompt improvements
• More specific exceptions
• Skip Windows 10 jobs for talos

All changes

You can see all changes in here:
0.14.0...0.15.1

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Mozilla CI tools contribution opportunities

A lot of feature development has happened around Mozilla CI tools this year and it has been great to have so many contributors to help us move forward! We would not be here without all the many contributions we've had.

For now, MozCI needs to wait a bit before we can do full progress around TaskCluster support (blocked on big-graph scheduling work )

Some of the work where we could get some help until then is the following:

• Use the full 20 byte / 40 character hashes for revisions
• Make the project flake8_docstrings compatible
• Readme and instructions are not usable for non-python-wizards
• Increase our test coverage
• Remove the need for LDAP credentials to determine valid repositories
• Add the ability in mozci scripts to re-trigger all existing jobs / all failed jobs
• Add TaskCluster GitHub integration to Mozci
• Release the pushlog module as a Python package and depend on it
• Release the buildapi module as a Python package and depend on it
• Create script which generates .yml files under Mozilla's Gecko tree
• Handle KeyboardInterrupt for all various scripts
• Add --existing-only support to triggerbyfilters

There are more issues, however, I believe these are easier to contribute to.

Feel free to ping "armenzg" on IRC at the #ateam channel (Read more on how to contact us in here)

EDIT: I fixed all urls poiting to https://github.com/armenzg/mozilla_ci_tools instead of https://github.com/mozilla/mozilla_ci_tools

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Minimal job scheduling

One of the biggest benefits of moving the scheduling into the tree is that you can adjust the decisions on what to schedule from within the tree.

As chmanchester and I were recently discussing, this is very important as we believe we can do much better on deciding what to schedule on try.

Currently, too many developers push to try with  -p all -u all (which schedules all platforms and all tests). It is understandable, it is the easiest way to reduce the risk of your change being backed out when it lands on one of the integration trees (e.g. mozilla-inbound).

In-tree scheduling analysis

What if your changes would get analysed and we would determine the best educated guess set of platforms and test jobs required to test your changes in order to not be backed out on an integration tree?

For instance, when I push Mozharness changes to mozilla-inbound, I wish I could tell the system that I only need these set of platforms and not those other ones.

If everyone had the minimum amount of jobs added to their pushes, our systems would be able to return results faster (less load) and no one would need to take short cuts.

This would be the best approximation and we would need to fine tune the logic over time to get things as right as possible. We would need to find the right balance of some changes being backed out because we did not get the right scheduling on try and getting results faster for everyone.

Prioritized tests

There is already some code that chmanchester landed where we can tell the infrastructure to run a small set of tests based on files changed. In this case we hijack one of the jobs (e.g. mochitest-1) to run the most relevant tests to your changes which would can normally be tested on different chunks. Once the prioritized tests are run, we can run the remaining tests as we would normally do. Prioritized tests also applies to suites that are not chunked (run a subset of tests instead of all).

There are some UI problems in here that we would need to figure out with Treeherder and Buildbot.

Tiered testing

Soon, we will have all technological pieces to create a multi tiered job scheduling system.

For instance, we could run things in this order (just a suggestion):

• Builds
• Prioritized tests (run in 5 to 15 mins)
• Remaining tests in normal mode

This has the advantage of using prioritized tests as a canary job which would prevent running the remaining tests if we fail the canary (shorter) job.

Post minimal run (automatic) precise scheduling (manual)

This is not specifically to scheduling the right thing automatically but to extending what gets scheduled automatically.

Imagine that you're not satisfied with what gets scheduled automatically and you would like to add more jobs (e.g. missing platforms or missing suites).

You will be able to add those missing jobs later directly from Treeherder by selecting which jobs are missing.

This will be possible once bug 1194830 lands.

NOTE: Mass scheduling (e.g. all mochitests across all platforms) would be a bit of a pain to do through Treeherder. We might want to do a second version of try-extender.

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Mozharness' support for Buildbot Bridge triggered test jobs

I have landed today [1] some code which allows Buildbot *test* jobs triggered through the Buildbot Bridge (BBB) to work properly.

In my previous post I explain a bit on how Mozci works with the Buildbot Bridge.
In this post I will only explain what we fixed on the Mozharness side.

Buildbot Changes

If a Buildbot test job is scheduled through TaskCluster (The Buildbot Bridge supports this), then the generated Buildbot Change associated to a test job does not have the installer and
test urls necessary for Mozharness to download for a test job.

What is a Buildbot Change? It is an object which represents the files touched associated to a code push. For the build jobs, this value gets set as part of the process of polling the Mercurial repositories, however, the test jobs are triggered via  a "buildbot sendchange" step part of the build job.

This sendchange creates the Buildbot Change for the test job which Mozharness can then use.

The BBB does not listen to sendchanges, hence, jobs triggered via the BBB have an empty changes object. Therefore, we can't download the files needed to run a test job and fail to execute.

In order to overcome this limtation, we have to detect when a Buildbot job is triggered normally or through the Buildbot Bridge.

Buildbot Bridge triggered jobs have a 'taskId' property defined (this represents the task associated to this Buildbot job). Through this 'taskId' we can determine the parent task and find a file called properties.json [2], which, it is uploaded by every BBB triggered job.

In such file, we can find both the installer and test urls.

[1] https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=03233057f1e6
[2] https://s3-us-west-2.amazonaws.com/taskcluster-public-artifacts/E7lb9P-IQjmeCOvkNfdIuw/0/public/properties.json

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Mozilla CI tools: Upcoming support for Buildbot Bridge

What is the Buildbot Bridge?

The Buildbot Bridge (BBB) allows scheduling Buildbot jobs through TaskCluster.
In other words, you can have taskcluster tasks which represent Buildbot jobs.
This allows having TaskCluster graphs composed of tasks which will be executed either on Buildbot or TaskCluster, hence, allowing for *all* relationships between tasks to happen in TaskCluster.

Read my recent post on the benefits of scheduling every job via TaskCluster.

The next Mozilla CI tools (mozci) release will have support for BBB.

Brief explanation

You can see in this try push both types of Buildbot jobs [1].
One set of jobs were triggered through Buildbot's analysis of the try syntax in the commit message while two of the jobs should not have been scheduled.

Those two jobs were triggered off-band via Mozci submitting a task graph.
You can see the TaskCluster graph representing them in here [2].

These jobs were triggered using this simple data structure:
{
  'Linux x86-64 try build': [
    'Ubuntu VM 12.04 x64 try opt test mochitest-1'
  ]
}

Mozci turns this simple graph into a TaskCluster graph.
The graph is composed of tasks which follow this structure [3]

Notes about the Buildbot Bridge

bhearsum's original post, recording and slides:
http://hearsum.ca/blog/buildbot-taskcluster-bridge-an-overview.html
https://vreplay.mozilla.com/replay/showRecordDetails.html?recId=1879
http://hearsum.ca/slides/buildbot-bridge-overview/#/

Some notes which Selena took about the topic:
http://www.chesnok.com/daily/2015/06/03/taskcluster-migration-about-the-buildbot-bridge/

The repository is in here:
https://github.com/mozilla/buildbot-bridge

---------
[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=9417c6151f2c
[2] https://tools.taskcluster.net/task-graph-inspector/#mAI0H1GyTJSo-YwpklZqng
[3] https://github.com/armenzg/mozilla_ci_tools/blob/mozci_bbb/mozci/sources/buildbot_bridge.py#L37

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Platform Operations lightning talks (Whistler 2015)

You can read and watch in here about the Platform Operations lighting talks:
https://wiki.mozilla.org/Platform_Operations/Ligthning_talks

Here the landing pages for the various Platform Operations teams:
https://wiki.mozilla.org/Platform_Operations

PS = I don't know what composes platform operations so feel free to add your team if I'm missing it.

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

The benefits of moving per-push Buildbot scheduling into the tree

Some of you may be aware of the Buildbot Bridge (aka BBB) work that bhearsum worked on during Q2 of this year. This system allows scheduling TaskCluster graphs for Buildbot builders. For every Buildbot job, there is a TaskCluster task that represents it.
This is very important as it will help to transition the release process piece by piece to TaskCluster without having to move large pieces of code at once to TaskCluster. You can have graphs of

I recently added to Mozilla CI tools the ability to schedule Buildbot jobs by submitting a TaskCluster graph (the BBB makes this possible).

Even though the initial work for the BBB is intended for Release tasks, I believe there are various benefits if we moved the scheduling into the tree (currently TaskCluster works like this; look for the gecko decision task in Treeherder).

To read another great blog post around try syntax and schedulling please visit ahal's post "Looking beyond Try Syntax".

NOTE: Try scheduling might not have try syntax in the future so I will not talk much about trychooser and try syntax. Read ahal's post to understand a bit more.

Benefits of in-tree scheduling:

• Per-branch scheduling matrix can be done in-tree
• We can define which platforms and jobs run on each tree
• TaskCluster tasks already do this
• Accurate Treeherder job visualization
• Currently, jobs that run through Buildbot do not necessarily show up properly
• Jobs run through TaskCluster show up accurately
• This is due to some issues with how Buildbot jobs are represented in between states and the difficulty to have a way to related them
• It could be fixed but it is not worth the effort if we're transitioning to TaskCluster
• Control when non-green jobs are run
• Currently on try we can't say run all unit tests jobs *but* the ones that should not run by default
• We would save resources (do not run non-green jobs) and confusion for developers (do not have to ask why is this job non-green)
• The try syntax parser can be done in-tree
• This allows for improving and extending the try parser
• Unit tests can be added
• The parser can be tested with a push
• try parser changes become atomic (it won't affecting all trees and can ride the trains)
• SETA analysis can be done in-tree
• SETA changes can become atomic (it won't affecting all trees and can ride the trains)
• We would not need to wait on Buildbot reconfigurations for new changes to be live.
• Per push scheduling analysis can be done in-tree
• We currently only will schedule jobs for a specific change if files for that product are being touched (e.g. Firefox for Android for mobile/* changes)
• PGO scheduling can be done in-tree
• PGO scheduling changes become atomic (it won't affecting all trees and can ride the trains)
• Environment awareness hooks (new)
• If the trees are closed, we can teach the scheduling system to not schedule jobs until further notice
• If we're backlogged, we can teach the scheduling system to not schedule certain platforms or to schedule a reduced set of jobs or to skip certain pushes
• Help the transition to TaskCluster
• Without it we would need to transition builds and associated tests to TaskCluster in one shot (not possible for Talos)
• Deprecate Self-serve/BuildApi
• Making changes to BuildApi is very difficult due to the lack of testing environments and set-up burden
• Moving to the BBB will help us move away from this old system

There are various parts that will need to be in place before we can do this. Here's some that I can think of:

• TaskCluster's big-graph scheduling
• This is important since it will allow for the concept of coalescing to exist in TaskCluster
• Task prioritization
• This is important if we're to have different levels of priority for jobs on TaskCluster
• On Buildbot we have release repositories with the highest priority and the try repo having the lowest
• We also currently have the ability to raise/decrease task priorities through self-serve/buildapi. This is used by developers, specially on Try. to allow their jobs to be picked up sooner.
• Treeherder to support LDAP authentication
• It is a better security model to scheduling changes
• If we want to move away from self-server/buildapi we need this
• Allow test jobs to find installer and test packages
• Currently test jobs scheduled through the BBB cannot find the Firefox installer and the 

Can you think of other benefits? Can you think of problems with this model? Are you aware of other pieces needed before moving forward to this model? Please let me know!

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Enabling automated back-filling on mozilla-inbound for few hours

tl;dr; we're going to enable automated back-filling tomorrow Tuesday for
few hours on mozilla-inbound.

We were aiming for Monday but pushed it to Tuesday to help publicize this more.

If on Wednesday there are no fall-outs we will leave it running for m-i for a week before enabling it on other places.

Posted on various mailing lists including mozilla.dev.tree-management.

> Hello all,
>
> We are planning to turn on a service that automatically backfills
> failed test jobs on m-i. If there are no concerns, we would like to
> turn this on experimentally for a couple of hours on [Tuesday]. We
> hope this will make it easier to identify which revision broke a
> test. Suggestions are welcome.
>
> The backfilling works like this: - It triggers the job that failed
> one extra time - Then it looks for a successful run of the job on the
> previous 5 revisions. If a good run is found, it triggers the job
> only on revisions up to this good run. If not, it triggers the job on
> every one of the previous 5 revisions. Previous jobs will be
> triggered one time.
>
> The tracking bug is:
> https://bugzilla.mozilla.org/show_bug.cgi?id=1180732
>
> Best, Alice

--
Zambrano Gasparnian, Armen
Automation & Tools Engineer
http://armenzg.blogspot.ca

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Few mozci releases + reduced memory usage

While I was away adusca released few releases of mozci.

From the latest release I want to highlight that we're replacing the standard json library for ijson since it solves some memory leak issues we were facing for pulse_actions (bug 1186232).

This was important to fix since our Heroku instance for pulse_actions has an upper limit of 1GB of RAM.

Here are the release notes and the highlights of them:

• 0.9.0 - Re-factored and cleaned-up part of the modules to help external consumers
• 0.10.0:
• --existing-only flag prevents triggering builds that are needed to trigger test jobs
• Support for pulse_actions functionality
• 0.10.1 - Fixed KeyError when querying for the request_id
• 0.11.0 - Added support for using ijson to load information, which decreases our memory usage

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

mozci 0.8.2 - Allow using TreeHerder as a query source

In this release we have added an experimental feature where you can use Treeherder as your source for jobs' information instead of using BuildApi/Buildjson.

My apologies as this should have been a minor release (0.9.0) instead of a security release (0.8.2).

Contributors

Thanks to @adusca @vaibhavmagarwal and @chmanchester for their contributions.
Our latest new contributor is @priyanklodha - thank you!

How to update

Run "pip install -U mozci" to update

Major highlights

• Added --query-source option to get data from Treeherder or Buildapi
• Improved usage of OOP to allow for different data sources seamlessly

Minor improvements

• Better documentation of --times
• Cleaning up old builds-*.js files
• Enforced line character limit

All changes

You can see all changes in here:
0.8.1...0.8.2

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

mozci 0.8.0 - New feature -- Trigger coalesced jobs + improved performance

Beware! This release is full awesome!! However, at the same time new bugs might pop up so please let us know :)
You can trigger now all coalesced jobs on a revision:

mozci-trigger --coalesced --repo-name REPO -r REVISION

Contributions

Thanks to @adusca @glandium @vaibhavmagarwal @chmanchester for their contributions on this release.

How to update

Run "pip install -U mozci" to update

Major highlights

• #259 - New feature - Allow triggering
• #227 - Cache files as gzip files instead of uncompressed
  • Less disk print
• #227 - Allow using multiple builders
• 1e591bf - Make sure that we do not fetch files if they are not newer
  • We were failing to double check that the last modification date of a file was the same as the one in the server
  • Hence, we were downloading files more often than needed
• Caching builds-4hr on memory for improved performance

Minor improvements

• f72135d - Backfilling did not accept dry_run or to trigger more than once
• More tests and documents
• Support for split tests (test packages json file)
• Some OOP refactoring

All changes

You can see all changes in here:
0.7.3...0.8.0

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

mozci 0.7.2 - Support b2g jobs that still run on Buildbot

There are a lot of b2g (aka Firefox OS) jobs that still run on Buildbot .
Interestingly enough we had not tried before to trigger one with mozci.
This release adds support for it.
This should have been a minor release (0.8.0) rather than a security release (0.7.2). My apologies!
All jobs that start with "b2g_" in all_builders.txt are b2g jobs that still run on Buildbot instead of TaskCluster (docs - TC jobs on treeherder).

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

mozci 0.7.1 - regression fix - do not look for files for running jobs

This release mainly fixes a regression we introduced in the release 0.7.0.
The change (#220) we introduced checked completed and running jobs for files that have been uploaded in order to trigger tests.
The problem is that running jobs do not have any metadata until they actually complete.
We fixed this on #234.

Contributions

Thanks to @adusca and @glandium for their contributions on this release.

How to update

Run "pip install -U mozci" to update

Major highlights

• #234 - (bug fix) - Do not try to find files for running jobs
• #228 - For try, only trigger talos jobs on existing build jobs ** rather than triggering builds for platforms that were not requested
• #238 - Read credentials through environment variables

Minor improvements

• #226 - (bug fix) Properly cache downloaded files
• #228 - (refactor) Move SCHEDULING_MANAGER
• #231 - Doc fixes

All changes

You can see all changes in here:
0.7.0...0.7.1

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

mozci 0.7.0 - Less network fetches - great speed improvements!

This release is not large in scope but it has many performance improvements.
The main improvement is to have reduced the number of times that we fetch for information and use a cache where possible. The network cost was very high.
You can read more about in here: http://explique.me/cProfile

Contributions

Thanks to @adusca @parkouss @vaibhavmagarwal for their contributions on this release.

How to update

Run "pip install -U mozci" to update

Major highlights

• Reduce drastically the number of requests by caching where possible
  • #221
• If a failed build has uploaded good files let's use them
  • #220
• Added support for retriggering and cancelling jobs
• Retrigger a job once with a count of N instead of triggering individually N times

Minor improvements

• Documenation updates
• Add waffle.io badge

All changes

You can see all changes in here:
0.6.0...0.7.0

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Welcome adusca!

It is my privilege to announce that adusca (blog) joined Mozilla (since Monday) as an Outreachy intern for the next 4 months.

adusca has an outstanding number of contributions over the last few months including Mozilla CI Tools (which we're working on together).

Here's a bit about herself from her blog:

Hi! I’m Alice. I studied Mathematics in college. I was doing a Master’s degree in Mathematical Economics before getting serious about programming.

She is also a graduate from Hacker's School.

Even though Alice has not been a programmer for many years, she has shown already lots of potential. For instance, she wrote a script to generate scheduling relations for buildbot; for this and many other reasons I tip my hat.

adusca will initially help me out with creating a generic pulse listener to handle job cancellations and retriggers for Treeheder. The intent is to create a way for Mozilla CI tools to manage scheduling on behalf of TH, make the way for more sophisticated Mozilla CI actions and allow other people to piggy back to this pulse service and trigger their own actions.

If you have not yet had a chance to welcome her and getting to know her, I highly encourage you to do so.

Welcome Alice!

EDIT: Added correct hyper-link to Hacke's School: https://www.recurse.com
Removed labels to prevent re-posting the various planets.

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

mozci 0.6.0 - Trigger based on Treeherder filters, Windows support, flexible and encrypted password managament

In this release of mozci we have a lot of developer facing improvements like Windows support or flexibility on password management.
We also have our latest experimental script mozci-triggerbyfilters (http://mozilla-ci-tools.readthedocs.org/en/latest/scripts.html#triggerbyfilters-py).

How to update

Run "pip install -U mozci" to update.

Notice

We have move all scripts from scripts/ to mozci/scripts/.
Note that you can now use "pip install" and have all scripts available as mozci-name_of_script_here in your PATH.

Contributions

We want to welcome @KWierso as our latest contributor!
Our gratitude @Gijs for reporting the Windows issues and for all his feedback.
Congratulations to @parkouss for making https://github.com/parkouss/mozbattue the first project using mozci as its dependency.
In this release we had @adusca and @vaibhavmagarwal as our main and very active contributors.

Major highlights

• Added script to trigger jobs based on Treeherder filters
  • This allows using filters like --include "web-platform-tests" and that will trigger all matching builders
  • You can also use --exclude to exclude builders you don't want
• With the new trigger by filters script you can preview what will be triggered:

233 jobs will be triggered, do you wish to continue? y/n/d (d=show details) d
05/15/2015 02:58:17 INFO: The following jobs will be triggered:
 Android 4.0 armv7 API 11+ try opt test mochitest-1
 Android 4.0 armv7 API 11+ try opt test mochitest-2

• Remove storing passwords in plain-text (Sorry!)
  • We now prompt the user if he/she wants to store their password enctrypted
• When you use "pip install" we will also install the main scripts as mozci-name_of_script_here binaries
  • This makes it easier to use the binaries in any location
• Windows issues
• The python module gzip.py is uncapable of decompressing large binaries
• Do not store buildjson on a temp file and then move

Minor improvements

• Updated docs
• Improve wording when triggering a build instead of a test job
• Loosened up the python requirements from == to >=
• Added filters to alltalos.py

All changes

You can see all changes in here:
0.5.0...0.6.0

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

"Thank you!"

This week I had a co-worker thank me again for a project I worked on at the end of 2014.
It touched my heart to hear it so I encourage you to do the same if someone has enabled you recently.

I have failed again and again to use the power of these two powerful words and I hope to use them more often.

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

mozci 0.5.0 released - Store password in keyring, prevent corrupted data, progress bar and many small improvements

In this release we have many small improvements that help with issues we have found.

The main improvement is that we now don't store credentials in plain-text (sorry!) but use keyring to store it encrypted.

We also prevent partially downloading any data (corrupted data) and added progress bar to downloads.

Congrats to @chmanchester as our latest contributor!
Our usual and very appreciated contributions are by @adusca @jmaher and @vaibhavmagarwal

Minor improvements:

• Lots of test changes and increased coverage
• Do not use the root logger but a mozci logger
• Allow passing custom files to a triggered job
• Work around buildbot status corruptions (Issue 167)
• Allow passing buildernames with lower case and removing trailing spaces (since we sometimes copy/paste from TH)
• Added support to use build a buildername based on trychooser syntax
• Allow passing extra properties when scheduling a job on Buildbot

You can see all changes in here:
0.4.0...0.5.0

Link to official release notes.

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

mozci hackday - Friday May 1st, 2015

I recently blogged about mozci and I was gladly surprised that people have curiosity about it.

I want to spend Friday fixing some issues on the tool and I wonder if you would like to join me to learn more about it and help me fix some of them.

I will be available as armenzg_mozci from 9 to 5pm EDT on IRC (#ateam channel).
I'm happy to jump on Vidyo to give you a hand understanding mozci.

I hand picked some issues that I could get a hand with.

• Prompt for credentials as soon as possible
• Do not prompt if --dry-run
• https://github.com/armenzg/mozilla_ci_tools/issues/158
• Do not store password on disk:
• https://github.com/armenzg/mozilla_ci_tools/issues/119
• Tackle known bugs
• https://github.com/armenzg/mozilla_ci_tools/labels/bug
• Minimize reading data from disk (Performance improvements)
• https://github.com/armenzg/mozilla_ci_tools/issues/143
• Add the ability to inspect the reason of a failure "inspect structured logs"
• https://github.com/armenzg/mozilla_ci_tools/issues/128

Documentation and definition of the project in readthedocs.

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

What Mozilla CI tools is and what it can do for you (aka mozci)

Mozci (Mozilla CI tools) is a python library, scripts and package which allows you to trigger jobs on treeherder.mozilla.org.
Not all jobs can be triggered but those that are run on Release Engineering's Buildbot setup. Most (if not all) Firefox desktop and Firefox for Android jobs can be triggered. I believe some B2G jobs can still be triggered.

NOTE: Most B2G jobs are not supported yet since they run on TaskCluster. Support for it will be given on this quarter.

Using it

Once you check out the code:

git clone https://github.com/armenzg/mozilla_ci_tools.git
python setup.py develop

you can run scripts like this one (click here for other scripts):

python scripts/trigger.py \
  --buildername "Rev5 MacOSX Yosemite 10.10 fx-team talos dromaeojs" \
  --rev e16054134e12 --times 10

which would trigger a specific job 10 times.

NOTE: This is independent if a build job exist to trigger the test job. mozci will trigger everything which is required to get you what you need.

One of the many other options is if you want to trigger the same job for the last X revisions, this would require you to use --back-revisions X.

There are many use cases and options listed in here.

A use case for developers

One use case which could be useful to developers (thanks @mike_conley!) is if you pushed to try and used this try syntax: "try: -b o -p win32 -u mochitests -t none". Unfortunately, you later determine that you really need this one: "try: -b o -p linux64,macosx64,win32 -u reftest,mochitests -t none".

In normal circumstances you would go and push again to the try server, however, with mozci (once someone implements this), we could simply pass the new syntax to a script (or with ./mach) and trigger everything that you need rather than having to push again and waster resources and your time!

If you have other use cases, please file an issue in here.

If you want to read about the definition of the project, vision, use cases or FAQ please visit the documentation.

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Firefox UI update testing

We currently trigger manually UI update tests for Firefox releases. There are automated headless update verification tests but they don't test the UI of Firefox.

The goal is to integrate this UI update testing as part of the Firefox releases.

This will require changes to firefox-ui-tests, buildbot scheduling changes, Marionette changes and other Mozbase packages. The ultimate goal is to speed up our turn around on releases.

The update testing code was recently ported from Mozmill to use Marionette to drive the testing.

I've already written some documentation on how to run the update verification using Release Engineering configuration files. You can use my tools repository until the code lands (update_testing is the branch to be used).

My deliverable is to ensure that the update testing works reliably on Release Engineering infrastructure and there is existing scheduling code for it.

You can read more about this project in bug 1148546.

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

How to install pywin32 on Windows

In Mozilla's Release Engineering Windows machines we have pywin32 installed.
This dependency if you're going to run older scripts that are needed for the release process.
Unfortunately, at the moment, we can't get rid of this dependency and need to install it.

If you're not using Mozilla-build, you can easily install it with these steps:

• Use Python 2.7.9's 32-bit installer
• If you don't already have Python installed for Windows
• Use Pywin32 for Python 2.7 32-bit (Build 219) installer

NOTE:These are 32-bit binary installers. 64-bit binaries are also available.

In Mozilla we use Mozilla-build which brings most of the tools you need to build Firefox.

Python is included on it, however, pywin32 is currently not pat of it (bug to fix this).

Since the process was a bit painful for me, I will take note of it for future reference.
I tried few approaches until I figured out that we need to use easy_install instead of pip and we need to point to an .exe file rather than a normal Python package.

Use easy_install

Here it is:

easy_install http://hivelocity.dl.sourceforge.net/project/pywin32/pywin32/Build%20219/pywin32-219.win32-py2.7.exe

You will know that it worked if you can run this without any errors:

python -c "import win32api" 

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

mozci 0.4.0 released - Many bug fixes and improved performance

For the release notes with all there hyper-links go here.

NOTE: I did a 0.3.1 release but the right number should have been 0.4.0

This release does not add any major features, however, it fixes many issues and has much better performance.

Many thanks to @adusca, @jmaher and @vaibhavmagarwal for their contributions.

Features:

• An alltalos.py script has been added
• Issue #69 - Generate graph of builds to testers
• Added flake8 support - Remove pyflakes and pep8
• Allow skipping revisions on a list (09f7138)
• Issue #61 - Rename trigger_range.py to trigger.py

Fixes:

• All the documentation and roadmap have been polished
• Issue #90 - Do not trigger builds multiple times if we are intending the test jobs to be triggered multiple times
• Issue #94 - Load list of repositories from disk only once
• Issue #117 - gaia-try builders are always upstream builders
• Determine a running job correctly (068b5ee)
• Issue #142 - Loading buildjson files from disk is now only done once
• Issue #135 - Remove buildjson files which have fallen out of date
• Issue #146 - If the buildapi information about a build is corrupted, trigger that build again
• Some DONTBUILD pushes can have buildapi support (dcb942f)
• Issue #120 - Prevent triggering more build jobs than necessary

For all changes visit: 0.3.0...0.4.0

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

How to generate data potentially useful to a dynamically generated trychooser UI

If you're interested on generating an up-to-date trychooser, I would love to hear from you.
adusca has helped me generate data similar to what a dynamic trychooser UI could use.
If you would like to help, please visit bug 983802 and let us know.

In order to generate the data all you have to do is:

git clone https://github.com/armenzg/mozilla_ci_tools.git
cd mozilla_ci_tools
python setup.py develop
python scripts/misc/write_tests_per_platform_graph.py

That's it! You will then have a graphs.json dictionary with some of the pieces needed. Once we have an idea on how to generate the UI and what we're missing we can modify this script.

Here's some of the output:

{
    "android": [
        "cppunit",
        "crashtest",
        "crashtest-1",
        "crashtest-2",
        "jsreftest-1",
        "jsreftest-2",
...

Here are the remaining keys:

[u'android', u'android-api-11', u'android-api-9', u'android-armv6', u'android-x86', u'emulator', u'emulator-jb', u'emulator-kk', u'linux', u'linux-pgo', u'linux32_gecko', u'linux64', u'linux64-asan', u'linux64-cc', u'linux64-mulet', u'linux64-pgo', u'linux64_gecko', u'macosx64', u'win32', u'win32-pgo', u'win64', u'win64-pgo']

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

mozci 0.3.0 - Support for backfilling jobs on treeherder added

Sometime on treeherder, jobs get coalesced (a.k.a. we run the tests on the most recent revision) in order to handle load. This is good so we can catch up when many pushes are committed on a tree.

However, when a job run on the most recent code comes back failing we need to find out which revision introduced the the regression. This is when we need to backfill up to the last good run.

In this release of mozci we have added the ability to --backfill:

python scripts/trigger_range.py --buildername "b2g_ubuntu64_vm cedar debug test gaia-js-integration-5" --dry-run --revision 2dea8b3c6c91 --backfill

This should be useful specially for sheriffs.

You can start using mozci as long as you have LDAP credentials. Follow these steps to get started:

git clone https://github.com/armenzg/mozilla_ci_tools.git
python setup.py develop (or install)

Release notes

Thanks again to vaibhav1994 and adusca for their many contributions in this release.

Major changes

• Issue #75 - Added the ability to backfill changes until last good is found
• No need to use --repo-name anymore
• Issue #83 - Look for request_ids from a better place
• Add interface to get status information instead of scheduling info

Minor fixes:

• Fixes to make livehtml documentation
• Make determine_upstream_builder() case insensitive

Release notes: https://github.com/armenzg/mozilla_ci_tools/releases/tag/0.3.0
PyPi package: https://pypi.python.org/pypi/mozci/0.3.0
Changes: https://github.com/armenzg/mozilla_ci_tools/compare/0.2.5...0.3.0

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

mozci 0.2.5 released - major bug fixes + many improvements

Big thanks again to vaibhav1994, adusca and valeriat for their many contributions in this release.

Release notes

Major bug fixes:

• Bug fix: Sort pushid_range numerically rather than alphabetically
• Calculation of hours_ago would not take days into consideration

Others:

• Added coveralls/coverage support
• Added "make livehtml" for live documentation changes
• Improved FAQ
• Updated roadmap
• Large documentation refactoring
• Automatically document scripts
• Added partial testing of mozci.mozci
• Streamed fetching of allthethings.json and verify integrity
• Clickable treeherder links
• Added support for zest.releaser

Release notes: https://github.com/armenzg/mozilla_ci_tools/releases/tag/0.2.5
PyPi package: https://pypi.python.org/pypi/mozci/0.2.5
Changes: https://github.com/armenzg/mozilla_ci_tools/compare/0.2.4...0.2.5

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

How to generate allthethings.json

It's this easy!
    hg clone https://hg.mozilla.org/build/braindump
    cd braindump/community
    ./generate_allthethings_json.sh

allthethings.json is generated based on data from buildbot-configs.
It contains data about builders, schedulers, masters and slavepools.

If you want to extract information from allthethings.json feel free to use mozci to help you!
https://mozilla-ci-tools.readthedocs.org/en/latest/allthethings.html

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

mozci 0.2.4 released - Support for intermittent oranges + major bug fixes

Big thanks to vaibhav1994 for his intermittent orange script contribution.

Also thanks to adusca and valeriat for their many contributions in this release.

Release notes

The major feature is being able to analyze data about reported intermittent oranges on bugzilla and give the user the ability to trigger jobs to spot where the regression started (generate_triggercli.py).

A lot of bug fixed and optimizations. I'm only highligthing some from this list: 0.2.3...0.2.4

Highlighted fixes:

• Fixed/improved issues with jobs completed today
• Added builds-4hr support
• allthethings.json gets clobbered after 24 hours
• This prevents relying on an old file
• Drop the need to use --repo-name for all scripts
• his prevents the user having to add a redundant option

Release notes: https://github.com/armenzg/mozilla_ci_tools/releases/tag/0.2.4
PyPi package: https://pypi.python.org/pypi/mozci/0.2.4
Changes: https://github.com/armenzg/mozilla_ci_tools/compare/0.2.3...0.2.4

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Listing builder differences for a buildbot-configs patch improved

Up until now, we updated the buildbot-configs repository to the "default" branch instead of "production" since we normally write patches against that branch.

However, there is a problem with this, buildbot-configs is always to be on the same branch as buildbotcustom. Otherwise, we can have changes land in one repository which require changes on the other one.

The fix was to simply make sure that both repositories are either on default or their associated production branches.

Besides this fix, I have landed two more changes:

1. Use the production branches instead of 'default'
• Use -p
2. Clobber our whole set up (e.g. ~/.mozilla/releng)
• Use -c

Here are the two changes:
https://hg.mozilla.org/build/braindump/rev/7b93c7b7c46a
https://hg.mozilla.org/build/braindump/rev/bbb5c54a7d42

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Pinning of mozharness completed

Yesterday we switched all jobs that use mozharness to use a specific changeset of mozharness instead of the production branch.

If you land mozharness code this affects you as you will have to:

• land on the default branch
• land a change on an integration branch to bump the revision of mozharness
• only at that moment your code will be in use

In other words, buildduty is not anymore in charge to make your changes live.

Original post: Pinning mozharness from in-tree (aka mozharness.json)

Bug: Bug 1110286

Notice to newsgroups: dev.platform - dev.b2g

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Mozilla CI tools 0.2.1 released - Trigger multiple jobs for a range of revisions

Today I have released a major release of mozci which includes the following:

• trigger_range(buildername, repo_name, start_revision, end_revision, times, dry_run=False)
• It allows you to trigger a buildername between two revisions as many times as you indicate.
• This is handy if you want to backfill jobs
• It fixes a major issue when trying to relate scheduling information with job execution data
• We needed to use UTC instead of localtime
• We needed to use the completion time of a job instead of the claiming time
• Added pushlog support
• Added buildbot job status interpretation 
• Found in mozci.sources.buildapi.query_job_status()

PyPi:       https://pypi.python.org/pypi/mozci
Source:   https://github.com/armenzg/mozilla_ci_tools

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

What the current list of buildbot builders is

This becomes very easy with mozilla_ci_tools (aka mozci):

>>> from mozci import mozci
>>> builders = mozci.list_builders()
>>> len(builders)
15736
>>> builders[0]
u'Linux x86-64 mozilla-inbound leak test build'

This and many other ways to interact with our CI will be showing up in the repository.

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Backed out - Pinning for Mozharness is enabled for the fx-team integration tree

EDIT=We had to back out this change since it caused issues for PGO talos jobs. We will try again after further testing.

Pinning for Mozharness [1] has been enabled for the fx-team integration tree.
Nothing should be changing. This is a no-op change.

We're still using the default mozharness repository and the "production" branch is what is being checked out. This has been enabled on Try and Ash for almost two months and all issues have been ironed out. You can know if a job is using pinning of Mozharness if you see "repostory_manifest.py" in its log.

If you notice anything odd please let me know in bug 1110286.

If by Monday we don't see anything odd happening, I would like to enable it for mozilla-central for few days before enabling it on all trunk trees.

Again, this is a no-op change, however, I want people to be aware of it.

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Name wanted for a retrigger-based bisection tool

Hello all,
I'm looking for a name for the tool that I will be working on this quarter.

This quarter I will be working on creating a prototype of a command-line tool that can be used by sheriffs and others to automate retrigger-based bisection. This could be used to help bisect new intermittent oranges, and to backfill jobs that have been skipped due to coalescing. Integration with Treeherder or other service will be done later.

I'm proposing "TriggerCI" since it shows what it does regardless of what you use it for.

If this works for you, please let me know.
If you have another suggestion please let me know. I'm interested on fun and creative names since that part of my brain is dysfunctional :P

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

How to uplift changes through the various Mozilla release trees

One of my co-workers asked me a bit ago what is a clean way of uplifting patches through the different Mozilla release trees. In my case the patch is self-approved since it only affects test jobs and not the product (a=testing).

Once my change landed on one of the integration trees (e.g. mozilla-inbound, mozilla-central), I waited for it to be merged to mozilla-central (a sheriff made a note on the bug). At that point we can start the uplift process.

In particular, the changeset I want is:
https://hg.mozilla.org/integration/fx-team/rev/a13aa1af5d75

Note

I believe I have the named branches because I have the firefoxtree extension enabled.
You can read more about it in here.

I have a unified checkout from which I can manage all the various release trees (aurora, beta, release, esr31 et al).

Update the tree

We update to the right brach:

hg pull beta
hg update beta

Graft changeset

NOTE: Usage explained in here.

NOTE: Make sure you use --edit if you need to add a=testing to your commit message.

We graft the changeset, carefull check that we're pushing to the right place and push:

armenzg@armenzg-thinkpad:~/repos/branches/release_branches$ hg id
774733a9b2ad beta/tip

armenzg@armenzg-thinkpad:~/repos/branches/release_branches$ hg graft --edit -r a13aa1af5d75
grafting revision 249741
merging testing/config/mozharness/android_arm_config.py

armenzg@armenzg-thinkpad:~/repos/branches/release_branches$ hg out -r tip beta
comparing with beta
searching for changes
changeset:   249997:93587eeda731
tag:         tip
user:        Armen Zambrano Gasparnian 
date:        Mon Jan 12 09:53:51 2015 -0500
summary:     Bug 1064002 - Fix removal of --log-raw from xpcshell. r=chmanchester. a=testing

armenzg@armenzg-thinkpad:~/repos/branches/release_branches$ hg push -r tip beta
pushing to ssh://hg.mozilla.org/releases/mozilla-beta
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 1 changes to 1 files
remote: Trying to insert into pushlog.
remote: Inserted into the pushlog db successfully.
remote: You can view your change at the following URL:
remote:   https://hg.mozilla.org/releases/mozilla-beta/rev/93587eeda731

Repeat the same process for the remaining trees.

UPDATE: Fixed typo

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.