Planet Mozilla Automation

August 27, 2015

Andrew Halberstadt

Looking beyond Try Syntax

Today marks the 5 year anniversary of try syntax. For the uninitiated, try syntax is a string that you put into your commit message which a parser then uses to determine the set of builds and tests to run on your try push. A common try syntax might look like this:

try: -b o -p linux -u mochitest -t none

Since inception, it has been a core part of the Mozilla development workflow. For many years it has served us well, and even today it serves us passably. But it is almost time for try syntax to don the wooden overcoat, and this post will explain why.

A brief history on try syntax

In the old days, pushing to try involved a web interface called sendchange.cgi. Pushing is probably the wrong word to use, as at no point did the process involve version control. Instead, patches were uploaded to the web service, which in turn invoked a buildbot sendchange with all the required arguments. Like today try server was often overloaded, sometimes taking over 4 hours for results to come back. Unlike today there was no way to pick and choose which builds and tests you wanted, every try push ran the full set.

The obvious solution was to create a mechanism for people to do that. It was while brainstorming this problem that ted, bhearsum and jorendorff came up with the idea of encoding this information in the commit message. Try syntax was first implemented by lsblakk in bug 473184 and landed on August 27th, 2010. It was a simple time; the list of valid builders could fit into a single 30 line config file; Fennec still hadn't picked up full steam; and B2G wasn't even a figment of anyone's wildest imagination.

It's probably not a surprise to anyone that as time went on, things got more complicated. As more build types, platforms and test jobs were added, the try syntax got harder to memorize. To help deal with this, lsblakk created the trychooser syntax builder just a few months later. In 2011, pbiggar created the trychooser mercurial extension (which was later forked and improved by sfink). These tools were (and still are) the canonical way to build a try syntax string. Little has changed since then, with the exception of the mach try command that chmanchester implemented around June 2015.

One step forward, two steps back

Since around 2013, the number of platforms and test configurations have grown at an unprecendented rate. So much so, that the various trychooser tools have been perpetually out of date. Any time someone got around to adding a new job to the tools, two other jobs had already sprung up in its place. Another problem caused by this rapid growth, was that try syntax became finicky. There were a lot of edge cases, exceptions to the rule and arbitrary aliases. Often jobs would mysteriously not show up when they should, or mysteriously show up when they shouldn't.

Both of those problems were exacerbated by the fact that the actual try parser code has never had a definite owner. Since it was first created, there have never been more than 11 commits in a year. There have been only two commits to date in 2015.

Two key insights

At this point, there are two things that are worth calling out:

  1. Generating try strings from memory is getting harder and harder, and for many cases is nigh impossible. We rely more and more on tools like trychooser.
  2. Try syntax is sort of like an API on which these tools are built on top of.

What this means is that primary generators of try syntax have shifted from humans to tools. A command line encoded in a commit message is convenient if you're a human generating the syntax manually. But as far as tooling goes, try syntax is one god awful API. Not only do the tools need to figure out the magic strings, they need to interact with version control, create an empty commit and push it to a remote repository.

There is also tooling on the other side of the see saw, things that process the try syntax post push. We've already seen buildbot's try parser but taskcluster has a separate try parser as well. This means that your try push has different behaviour, depending on whether the jobs are scheduled in buildbot or taskcluster. There are other one off tools that do some try syntax parsing as well, including but not limited to try tools in mozharness, the try re-trigger bot and the AWSY dashboard. These tools are all forced to share and parse the same try syntax string, so they have to be careful not to step on each other's toes.

The takeaway here is that for tools, a string encoded as a commit message is quite limiting and a lot less convenient than say, calling a function in a library.

Despair not, young Padawan

So far we've seen how try syntax is finicky, how the tools that use it are often outdated and how it fails as an API. But what is the alternative? Fortunately, over the course of 2015 a lot of progress has been made on projects that for the first time, give us a viable alternative to try syntax.

First and foremost, is mozci. Mozci, created by armenzg and adusca, is a tool that hooks into the build api (with early support for taskcluster as well). It can do things like schedule builds and tests against any arbitrary pushes, and is being used on the backend for tools like adusca's try-extender with integration directly into treeherder planned.

Another project that improves the situation is taskcluster itself. With taskcluster, job configuration and scheduling all lives in tree. Thanks to bhearsum's buildbot bridge, we can even use taskcluster to schedule jobs that still live in buildbot. There's an opportunity here to leverage these new tools in conjunction with mozci to gain complete and total control over how jobs are scheduled on try.

Finally I'd like to call out mach try once more. It is more than a thin wrapper around try syntax that handles your push for you. It actually lets you control how the harness gets run within a job. For now this is limited to test paths and tags, but there is a lot of potential to do some cool things here. One of the current limiting factors is the unexpressiveness of the try syntax API. Hopefully this won't be a problem too much longer. Oh yeah, and mach try also works with git.

A glimpse into the crystal ball

So we have several different projects all coming together at once. The hard part is figuring out how they all tie in together. What do we want to tackle first? How might the future look? I want to be clear that none of this is imminent. This is a look into what might be, not what will be.

There are two places we mainly care about scheduling jobs on try.

First imagine you push your change to try. You open up treeherder, except no jobs are scheduled. Instead you see every possible job in a distinct greyed out colour. Scheduling what you want is as simple as clicking the desired job icons. Hold on a sec, you don't have to imagine it. Adusca already has a prototype of what this might look like. Being able to schedule your try jobs this way has a huge benefit: you don't need to mentally correlate job symbols to job names. It's as easy as point and click.

Second, is pushing a predefined set of jobs to try from the command line, similar to how things work now. It's often handy to have the try command for a specific job set in your shell history and it's a pain to open up treeherder for a simple push that you've memorized and run dozens of times. There are a few improvements we can do here:

Finally for those who are stuck in their ways, it should still be possible to have a "classic try syntax" front-end to the new mozci backend. As large as this change sounds, it could be mostly transparent to the user. While I'm certainly not a fan of the current try syntax, there's no reason to begrudge the people who are.

Closing words

Try syntax has served us well for 5 long years. But it's almost time to move on to something better. Soon a lot of new avenues will be open and tools will be created that none of us have thought of yet. I'd like to thank all of the people mentioned in this post for their contributions in this area and I'm very excited for what the future holds.

The future is bright, and change is for the better.

August 27, 2015 06:03 PM

August 26, 2015

Jonathan Griffin

Engineering Productivity Update, August 26, 2015

It’s PTO season and many people have taken a few days or week off.  While they’re away, the team continues making progress on a variety of fronts.  Planning also continues for GoFaster and addon-signing, which will both likely be significant projects for the team in Q4.


Treeherder: camd rolled out a change which collapses chunked jobs on Treeherder, reducing visual noise.  In the future, we plan on significantly increasing the number of chunks of many jobs in order to reduce runtimes, so this change makes that work more practical.  See camd’s blog post.  emorley has landed a change which allows TaskCluster job errors that occur outside of mozharness to be properly handled by Treeherder.

Automatic Starring: jgraham has developed a basic backend which supports recognizing simple intermittent failures, and is working on integrating that into Treeherder; mdoglio is landing some related database changes. ekyle has received sheriff training from RyanVM, and plans to use this to help improve the automated failure recognition algorithm.

Perfherder and Performance Testing: Datazilla has finally been decommissioned (R.I.P.), in favor of our newest performance analysis tool, Perfherder.  A lot of Talos documentation updates have been made at, including details about how we perform calculations on data produced by Talos.  wlach performed a useful post-mortem of Eideticker, with several takeaways which should be applicable to many other projects.

MozReview and Autoland: There’s a MozReview meetup underway, so expect some cool updates next time!

TaskCluster Support: ted has made a successful cross-compiled OSX build using TaskCluster!  Take it for a spin.  More work is needed before we can move OSX builds from the mac mini builders to the cloud.

Mobile Automation: gbrown continues to make improvements on the new |mach emulator| command which makes running Android tests locally on emulator very simple.

General Automation: run-by-dir is live on opt mochitest-plain; debug and ASAN coming soon.  This reduces test “bleed-through” and makes it easier to change chunking.  adusca, our Outreachy intern, is working to integrate the try extender into Treeherder.  And ahal has merged the mozharness “in-tree” configs with the regular mozharness config files, now that mozharness lives in the tree.

Firefox Automation: YouTube ad detection has been improved for firefox-media-tests by maja, which fixes the source of the top intermittent failure in this suite.

Bughunter: bc has got asan-opt builds running in production, and is working on gtk3 support. gps has enabled syntax highlighting in hgweb, and has added a new JSON API as well.  See gps’ blog post.

The Details
Perfherder/Performance Testing
TaskCluster Support
Mobile Automation
Firefox and Media Automation
General Automation
  •  run-by-dir is live for mochitest-plain (opt only); debug is coming soon, followed by ASAN.
  • Mozilla CI tools is moving from using BuildAPI as the scheduling entry point to use TaskCluster’ scheduling. This work will allow us to schedule a graph of buildbot jobs and their dependencies in one shot.
  • adusca is integrating into treeherder the ability to extend the jobs run for any push. This is based on the prototype. Follow along in
  • Git was deployed to the test machines. This is necessary to make the Firefox UI update tests work on them.
  • [ahal] merge mozharness in-tree configs with the main mozharness configs
  • Bug fixes to the ETL – fix bad lookups on hg repo, mostly l10n builds 
  • More error reporting on ETL – Structured logging has changed a little, handle the new variations, and be more elegant when it comes to unknowns, an complain when there is non-conformance.
  • Some work on adding hg `repo` table – acts as a cache for ETL, but can be used to calculate ‘per push’ statistics on OrangeFactor data.
  • Added Talos to the `perf` table – used the old Datazilla ETL code to fill the ES cluster.  This may speed up extracting the replicates, for exploring the behaviour of a test.
  • Enable deep queries – Effectively performing SQL join on Elasticsearch – first attempt did too much refactoring.  Second attempt is simpler, but still slogging through all the resulting test breakage
  • Updated 
  • [ahal] helped review and finish contributor patch for switching marionette_client from optparse to argparse
  • Corrected UUID used for session ID and element IDs
  • Updated dispatching of various marionette calls in Gecko
  • [bc] Have asan-opt builds running in production. Finalizing patch. Still need to build gtk3 for rhel6 32bit in order to stop using custom builds and support opt in addition to debug.
  • Updated the hierarchical burndowns to EPM’s important metabugs that track features 
  • More config changes

August 26, 2015 11:47 PM

August 25, 2015

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

August 25, 2015 06:25 AM

August 18, 2015

Cameron Dawson

Treeherder Job Counts and Chunks

The Treeherder production instance has a significant front-end change arriving in the coming days. We are condensing job groups and chunks (a job which, for individual team or project reasons, were separated into numerous multiple jobs) into a ‘+n’ count representation in the job group.

To see the motivation for this change, one need only look at this before/after example of the same job-sets.  The number of chunks has begun (and will continue to) grow.  As this happens, the Treeherder page is becoming more cluttered.  The job aggregation to counts will streamline the initial view, while still giving easy access to individual job chunks as you need them.


Screenshot 2015-08-18 08.17.13

Wow.  That’s a lot of chunks.


Screenshot 2015-08-18 08.17.30

Whew!  To be sure, this is an extreme example.  But even in less extreme chunk counts, the difference is significant overall.


Counts can be expanded/collapsed Globally via our new navbar button:


Counts can also be expanded Locally either by clicking the ‘+n’ job count…


…or counts can be expanded and collapsed Locally by clicking the job group:


When a job is selected and its count collapsed, the ‘+n’ count receives a selected style until either a different job is selected, or the selection is cleared.

Classification and unclassified failure and next/previous job navigation should remain unchanged.  As new jobs are created or their state changes, you will notice the counts updating accordingly.


I received opinions and feedback from most of the Treeherder team, Sheriffs and Blake Winton.  But I wanted to give a special shout-out to Jonathan French.  His testing and design help GREATLY improved the quality of this feature!  Thanks Jon!

Contact us

If you have any comments/suggestions about this change, please feel free to visit us on IRC in the #treeherder channel.


-Cam [:camd]

August 18, 2015 04:30 PM

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

August 18, 2015 07:58 AM

August 13, 2015

Jonathan Griffin

Engineering Productivity Update, August 13, 2015

From Automation and Tools to Engineering Productivity

Automation and Tools” has been our name for a long time, but it is a catch-all name which can mean anything, everything, or nothing, depending on the context. Furthermore, it’s often unclear to others which “Automation” we should own or help with.

For these reasons, we are adopting the name “Engineering Productivity”. This name embodies the diverse range of work we do, reinforces our mission (, promotes immediate recognition of the value we provide to the organization, and encourages a re-commitment to the reason this team was originally created—to help developers move faster and be more effective through automation.

The “A-Team” nickname will very much still live on, even though our official name no longer begins with an “A”; the “get it done” spirit associated with that nickname remains a core part of our identity and culture, so you’ll still find us in #ateam, brainstorming and implementing ways to make the lives of Mozilla’s developers better.


Treeherder: Most of the backend work to support automatic starring of intermittent failures has been done. On the front end, several features were added to make it easier for sheriffs and others to retrigger jobs to assist with bisection: the ability to fill in all missing jobs for a particular push, the ability to trigger Talos jobs N times, the ability to backfill all the coalesced jobs of a specific type, and the ability to retrigger all pinned jobs. These changes should make bug hunting much easier.  Several improvements were made to the Logviewer as well, which should increase its usefulness.

Perfherder and performance testing: Lots of Perfherder improvements have landed in the last couple of weeks. See details at wlach’s blog post.  Meanwhile, lots of Talos cleanup is underway in preparation for moving it into the tree.

MozReview: Some upcoming auth changes are explained in mcote’s blog post.

Mobile automation: gbrown has converted a set of robocop tests to the newly enabled mochitest-chrome on Android. This is a much more efficient harness and converting just 20 tests has resulted in a reduction of 30 minutes of machine time per push.

Developer workflow: chmanchester is working on building annotations into files that will automatically select or prioritize tests based on files changed in a commit. See his blog post for more details. Meanwhile, armenzg and adusca have implemented an initial version of a Try Extender app, which allows people to add more jobs on an existing try push. Additional improvements for this are planned.

Firefox automation: whimboo has written a Q2 Firefox Automation Report detailing recent work on Firefox Update and UI tests. Maja has improved the integration of Firefox media tests with Treeherder so that they now officially support all the Tier 2 job requirements.

WebDriver and Marionette: WebDriver is now officially a living standard. Congratulations to David Burns, Andreas Tolfsen, and James Graham who have contributed to this standard. dburns has created some documentation which describes which WebDriver endpoints are implemented in Marionette.

Version control: The ability to read and extra metadata from files has been added to This opens the door to cool future features, like the ability auto file bugs in the proper component and automatically selecting appropriate reviewers when pushing to MozReview. gps has also blogged about some operational changes to which enables easier end-to-end testing of new features, among other things.

The Details
Treeherder/Automatic Starring
Treeherder/Front End
  • Several retrigger features were added to Treeherder to make merging and bisections easier:  auto fill all missing/coalesced jobs in a push; trigger all Talos jobs N times; backfill a specific job by triggering it on all skipped commits between this commit and the commit that previously ran the job, retrigger all pinned jobs in treeherder.  This should improve bug hunting for sheriffs and developers alike.
  • [jfrench] Logviewer ‘action buttons’ are now centralized in a Treeherder style navbar
  • [jfrench] Logviewer skipped steps are now recognized as non-failures and presented as blue info steps,
  • [jfrench] Middle-mouse-clicking on a job in treeherder now launches the Logviewer
  • [vaibhav] Added the ability to retrigger all pinned jobs (bug
  • Camd’s job chunking management will likely land next week
Perfherder/Performance Testing
  • [wlach] / [jmaher] Lots of perfherder updates, details here: Highlights below
  • [wlach] The compare pushes view in Perfherder has been improved to highlight the most important information.
  • [wlach] If your try push contains Talos jobs, you’ll get a url for the Perfherder comparison view when pushing (
  • [jmaher/wlach] Talos generates suite and test level metrics and perfherder now ingests those data points. This fixes results from internal benchmarks which do their own summarization to report proper numbers.
  • [jmaher/parkouss] Big talos updates (thanks to :parkouss), major refactoring, cleanup, and preparation to move talos in tree.
Mobile Automation
Dev Workflow
  • [ahal] Created patch to clobber compiled python files in srcdir
  • [ahal] More progress on mach/mozlog patch
  • [chmanchester] Fix to allow ‘mach try’ to work without test arguments (bug 1192484)
Media Automation
  • [maja_zf] firefox-media-tests ‘log steps’ and ‘failure summaries’ are now compatible with Treeherder’s log viewer, making them much easier to browse. This means the jobs now satisfy all Tier-2 Treeherder requirements.
  • [sydpolk] Refactoring of tests after fixing stall detection is complete. I can now take my network bandwidth prototype and merge it in.
Firefox Automation
General Automation
  • Finished adapting mozregression ( and mozdownload ( to S3.
  • (Henrik) Isn’t only a temporary solution, before we move to TC?
  • (armenzg) I believe so but for the time being I believe we’re out of the woods
  • The manifestparser dependency was removed from mozprofile (bug 1189858)
  • [ahal] Fix for
  • [sydpolk] Platform Jenkins migration to the SCL data center has not yet begun in earnest due to PTO. Hope to start making that transition this week.
  • [chmanchester] work in progress to build annotations in to files to automatically select or prioritize tests based on what changed in a commit. Strawman implementation posted in , blog post about this work at
  • [adusca/armenzg] Try Extender ( ) is open for business, however, a new plan will soon be released to make a better version that integrates well with Treeherder and solves some technichal difficulties we’re facing
  • [armenzg] Code has landed on mozci to allow re-triggering tasks on TaskCluster. This allows re-triggering TaskCluster tasks on the try server when they fail.
  • [armenzg] Work to move Firefox UI tests to the test machines instead of build machines is solving some of the crash issues we were facing
  • [ahal] re-implemented test-informant to use ActiveData:
  • [ekyle] Work on stability: Monitoring added to the rest of the ActiveData machines.  
  • [ekyle] Problem:  ES was not balancing the import workload on the cluster; probably because ES assumes symmetric nodes, and we do not have that.  The architecture was changed to prefer a better distribution of work (and query load) – There now appears to be less OutOfMemoryExceptions, despite test-informant’s queries.
  • [ekyle] More Problems:  Two servers in the ActiveData complex failed: The first was the ActiveData web server; which became unresponsive, even to SSH.  The machine was terminated.  The second server was the ‘master’ node of the ES cluster: This resulted in total data loss, but it was expected to happen eventually given the cheap configuration we have.   Contingency was in place:  The master was rebooted, the  configuration was verified, and data re-indexed from S3.   More nodes would help with this, but given the rarity of the event, the contingency plan in place, and the low number of users, it is not yet worth paying for. 
WebDriver (highlights)
  • [ato] WebDriver is now officially a living standard (
  • [ato] Rewrote chapter on extending the WebDriver protocol with vendor-specific commands
  • [ato] Defined the Get Element CSS Value command in specification
  • [ato] Get Element Attribute no longer conflates DOM attributes and properties; introduces new command Get Element Property
  • [ato] Several significant infrastructural issues with the specification was fixed
  • Project managers for FxOS have a renewed interest in the project tracking, and overall status dashboards.   Talk only, no coding yet.

August 13, 2015 11:17 PM

August 12, 2015

Andreas Tolfsen

The sorry state of women in tech

I read the anonymous blog post Your Pipeline Argument is Bullshit with great interest. While it’s not very diplomatic and contains a great deal of ranting, it speaks about something very important: Women in tech. I read it more as an expression of frustration, than a mindless flamatory rant as others have indicated.

On the disproportion of women granted CS degrees in the US:

I use this data because I graduated with a CS degree in 2000. Convenient. The 2000s is also when the proportion of women getting CS degrees started falling even faster. My first job out of school had an older woman working there who let me know back in the 80s there were a lot more women. Turns out, in the early 80s, almost 40% of CS degrees in the US were granted to women.

In 2009 only about 18% women graduated with CS degrees.

Although purely anecdotal evidence, my experience after working seven years as a professional programmer is that more women than men leave the industry to pursue other careers. In fact the numbers show that the cumulative pool of talent (meaning all who hold CS degrees) is close to 25% women.

We have to recognise what a sorry state our industry is in. Callous and utilitarian focus on technology alone, as opposed to building a healthy discourse and community around computing that is inclusive to women, does more harm than we can imagine.

Because men hire men, and especially those men similar to ourselves, I’m in favour of positive action to balance the numbers.

Whoever is saying that we should hire based on talent alone, there’s little evidence to support that 25% of the available CS talent is any worse than the remaining three quarters.

August 12, 2015 06:27 PM

mozregression updates

Release 0.41

mozregression 0.41 fixes bug 1185756, which caused the following warning on Windows:

WindowsError: [Error 32] The process cannot access the file
because it is being used by another process

Also, mozregression is now more tolerant when facing failures like a build not installable or a crash from the tested application (see bug 1192485).

And finally, when using mozregression with a custom profile it won’t be modified anymore (see bug 999009). This was a longstanding bug - thanks to Jonathan Pigree for his help in fixing it!

August 12, 2015 12:00 AM

August 11, 2015

Henrik Skupin

Firefox Automation report – Q2 2015

It’s been a while since I wrote my last Firefox automation report, so lets do another one to wrap up everything happened in Q2 this year. As you may remember from my last report the team has been cut down to only myself, and beside that I was away the whole April. Means only 2 months worth of work will be covered this time.

In general it was a tough quarter for me. Working alone and having to maintain all of our infrastructure and keeping both of our tests (Mozmill and Marionette) green was more work as expected. But it still gave me enough time to finish my two deliverables:

Firefox UI Test Results on Treeherder

With the transition of Mozmill tests to Marionette we no longer needed the mozmill-dashboard. Especially not since nearly no job in our Mozmill CI is running Mozmill anymore. Given that we also weren’t happy with the dashboard solution and the usage of couchdb under the hood, I was happy that a great replacement exist and our Marionette tests can make use of.

Treeherder is the new reporting system for tests being run in buildbot continuous integration. Because of that it covers all products and their appropriate supported branches. That’s why it it’s kinda perfect for us.

Before I was able to get started a bit of investigation had to be done, and I also checked some other projects which make use of treeherder reporting. That information was kinda helpful even with lots of undocumented code. Given that our tests are located outside of the development tree in its own Github repository some integration can be handled more loosely compared to in-tree tests. With that respect we do not appear as Tier-1 or Tier-2 but results are listed under the Tier-3 level, which is hidden by default. But that’s fine given that our goal was to bring up reports to treeherder first. Going into details will be a follow-up task.

For reporting results to Treeherder the Python package treeherder-client can be used. It’s a collection of different classes which help with authentication, collecting job details, and finally uploading the data to the server. It’s documentation can be found on readthedocs and meanwhile contains a good number of example code which makes it easier to get your code implemented.

Before you can actually upload reports the names and symbols of the groups and jobs have to be defined. For our tests I have chosen the three group symbols “Fu”, “Ff”, and “Fr”. Each of those stays for “(F)irefox UI Tests” and the first letter of the testrun name. We currently have “updates”, “functional”, and “remote”. As job name the locale of Firefox will be used. That results in an output like the following:

Treeherder Results

Whether jobs are passing or failing it is recommended to always add the log files as generated by running the tests to the report. It will not happen via data URLs but as artifacts with a link to an upload location. In our case we make use of Amazon S3 as storage service.

With all the pieces implemented we can now report to treeherder and covering Nightly, Aurora (Developer Edition), Beta, and Release builds. As of now reporting only happens against the staging instance of Treeherder, but soon we will be able to report to production. If you want to have a sneak peak how it works, just follow this link for Nightly builds.

More details about Treeherder reporting I will do later this quarter when the next pieces have been implemented.

Firefox UI Update Tests under RelEng

My second deliverable was to assist Armen Zambrano in getting the Firefox UI Update tests run for beta and release builds on Release Engineering infrastructure. This was a kinda important goal for us given that until now it was a manually triggered process with lots of human errors on our own infrastructure. That means lots of failures if you do not correctly setup the configuration for the tests, and a slower processing of builds due to our limited available infrastructure. So moving this out of our area made total sense.

Given that Armen had already done a fair amount of work when I came back from my PTO, I majorly fixed issues for the tests and the libraries as pointed out by him. All that allowed us to finally run our tests on Release Engineering infrastructure even with a couple of failures at the beginning for the first beta. But those were smaller issues and got fixed quickly. Since then we seem to have good results. If you want to have a look in how that works, you should check the Marionette update tests wiki page.

Sadly some of the requirements haven’t been completely finished yet. So the Quality Engineering team cannot stop running the tests themselves. But that will happen once bug 1182796 has been fixed and deployed to production.

Oh, and if you wonder where the results are located… Those are not getting sent to Treeherder but to an internal mailing list as used for every other automation results.

Other Work

Beside the deliverables I got some more work done. Mainly for the firefox-ui-tests and mozmill-ci.

While the test coverage has not really been increased, I had a couple of regressions to fix as caused by changes in Firefox. But we also landed some new features thankfully as contributed by community members. Once all that was done and we agreed to have kinda stable tests, new branches have been created in the repository. That was necessary to add support for each supported version of Firefox down to ESR 38.0, and to be able to run the tests in our Mozmill CI via Jenkins. More about that you will find below. The only task I still haven’t had time for yet was the creation of proper documentation about our tests. I hope that I will find the time in Q3.

Mozmill CI got the most changes in Q2 compared to all the former quarters. This is related to the transition from Mozmill tests to Marionette tests. More details why we got rid of Mozmill tests can be found in this post. With that we decided to get rid of most of the tests and mainly start from scratch by only porting the security and update tests over to Marionette. The complete replacement in Mozmill and all its jobs can be seen on issue 576 on Github. In detail we have the following major changes:

All changes in Mozmill CI can be seen on Github.

Last but not least we also had two releases of mozdownload in Q2. Both had a good amount of features included. For details you can check the changelog.

I hope that gave you a good quick read on the stuff I was working on last quarter. Maybe in Q3 I will find the time to blog more often and in more detail. Lets see.

August 11, 2015 02:20 PM

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

August 11, 2015 07:41 AM

August 10, 2015

Alice Scarpa

Extending Try pushes

Are you tired of having to do a new try push just because you forgot a test? Your problems are over! Meet Try Extender. This new web app allows you to add new jobs to existing try pushes, all you have to do is log in, choose a revision and pick the jobs you want!

OK, now I’m done with my parody informercial.

Try Extender is a web app running on Heroku that uses mozci to trigger new jobs for a try push. It is still a prototype, but if you want to extend a push you are welcomed to try it out! Currently it can only add new test jobs to completed build jobs or add new build jobs.

Right now the main goal is just to validate the idea. Depending on how this experiment goes we might work on integrating it into Treeherder, or leave it as a stand-alone web app.

August 10, 2015 12:00 AM

August 07, 2015

William Lachance

More Perfherder updates

Since my last update, we’ve been trucking along with improvements to Perfherder, the project for making Firefox performance sheriffing and analysis easier.

Compare visualization improvements

I’ve been spending quite a bit of time trying to fix up the display of information in the compare view, to address feedback from developers and hopefully generally streamline things. Vladan (from the perf team) referred me to Blake Winton, who provided tons of awesome suggestions on how to present things more concisely.

Here’s an old versus new picture:

Screen Shot 2015-07-14 at 3.53.20 PM Screen Shot 2015-08-07 at 1.57.39 PM

Summary of significant changes in this view:

The point of these changes isn’t necessarily to make everything “immediately obvious” to people. We’re not building general purpose software here: Perfherder will always be a rather specialized tool which presumes significant domain knowledge on the part of the people using it. However, even for our audience, it turns out that there’s a lot of room to improve how our presentation: reducing the amount of extraneous noise helps people zero in on the things they really need to care about.

Special thanks to everyone who took time out of their schedules to provide so much good feedback, in particular Avi Halmachi, Glandium, and Joel Maher.

Of course more suggestions are always welcome. Please give it a try and file bugs against the perfherder component if you find anything you’d like to see changed or improved.

Getting the word out

Hammersmith:mozilla-central wlach$ hg push -f try
pushing to ssh://
no revisions specified to push; using . to avoid pushing multiple heads
searching for changes
remote: waiting for lock on repository /repo/hg/mozilla/try held by ''
remote: got lock after 4 seconds
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 1 changes to 1 files
remote: Trying to insert into pushlog.
remote: Inserted into the pushlog db successfully.
remote: View your change here:
remote: Follow the progress of your build on Treeherder:
remote: It looks like this try push has talos jobs. Compare performance against a baseline revision:

Try pushes incorporating Talos jobs now automatically link to perfherder’s compare view, both in the output from mercurial and in the emails the system sends. One of the challenges we’ve been facing up to this point is just letting developers know that Perfherder exists and it can help them either avoid or resolve performance regressions. I believe this will help.

Data quality and ingestion improvements

Over the past couple weeks, we’ve been comparing our regression detection code when run against Graphserver data to Perfherder data. In doing so, we discovered that we’ve sometimes been using the wrong algorithm (geometric mean) to summarize some of our tests, leading to unexpected and less meaningful results. For example, the v8_7 benchmark uses a custom weighting algorithm for its score, to account for the fact that the things it tests have a particular range of expected values.

To hopefully prevent this from happening again in the future, we’ve decided to move the test summarization code out of Perfherder back into Talos (bug 1184966). This has the additional benefit of creating a stronger connection between the content of the Talos logs and what Perfherder displays in its comparison and graph views, which has thrown people off in the past.

Continuing data challenges

Having better tools for visualizing this stuff is great, but it also highlights some continuing problems we’ve had with data quality. It turns out that our automation setup often produces qualitatively different performance results for the exact same set of data, depending on when and how the tests are run.

A certain amount of random noise is always expected when running performance tests. As much as we might try to make them uniform, our testing machines and environments are just not 100% identical. That we expect and can deal with: our standard approach is just to retrigger runs, to make sure we get a representative sample of data from our population of machines.

The problem comes when there’s a pattern to the noise: we’ve already noticed that tests run on the weekends produce different results (see Joel’s post from a year ago, “A case of the weekends”) but it seems as if there’s other circumstances where one set of results will be different from another, depending on the time that each set of tests was run. Some tests and platforms (e.g. the a11yr suite, MacOS X 10.10) seem particularly susceptible to this issue.

We need to find better ways of dealing with this problem, as it can result in a lot of wasted time and energy, for both sheriffs and developers. See for example bug 1190877, which concerned a completely spurious regression on the tresize benchmark that was initially blamed on some changes to the media code– in this case, Joel speculates that the linux64 test machines we use might have changed from under us in some way, but we really don’t know yet.

I see two approaches possible here:

  1. Figure out what’s causing the same machines to produce qualitatively different result distributions and address that. This is of course the ideal solution, but it requires coordination with other parts of the organization who are likely quite busy and might be hard.
  2. Figure out better ways of detecting and managing these sorts of case. I have noticed that the standard deviation inside the results when we have spurious regressions/improvements tends to be higher (see for example this compare view for the aforementioned “regression”). Knowing what we do, maybe there’s some statistical methods we can use to detect bad data?

For now, I’m leaning towards (2). I don’t think we’ll ever completely solve this problem and I think coming up with better approaches to understanding and managing it will pay the largest dividends. Open to other opinions of course!

August 07, 2015 08:04 PM

Geoff Brown

mochitest-chrome tests for Android

Support for mochitest-chrome tests on Android was recently improved and mochitest-chrome tests can now be seen on treeherder (the M(c) job) for all Firefox for Android builds. Most mochitest-chrome tests are desktop-specific or cross-platform; most Firefox for Android tests have been written for Robocop, our Robotium-based Android UI test framework.

I noticed that some Robocop tests were implemented almost entirely in Javascript and could easily be converted to mochitest-chrome, where tests would run much more efficiently. Bug 1184186 converted about 20 such Robocop tests to mochitest-chrome, reducing our Robocop load by about 30 minutes while increasing mochitest-chrome load by only about 3 minutes. (We save time by not starting and stopping the browser between mochitests, and not waiting around for state changes, as frequently required in UI tests.)

The “new” mochitest-chrome tests are all located in mobile/android/tests/browser/chrome and can also be run locally from mach. Just make sure you have:

Here is a screen recording showing the new tests in action:

Want to write your own mochitest-chrome test for Firefox for Android? Be sure to see for general advice. If you are looking for an example, is one of the simplest tests — a great place to start.

As always, be sure to see for detailed information about running tests for Firefox for Android, and ping me on irc if you run into any trouble.

August 07, 2015 04:50 PM

August 06, 2015

Christopher Manchester

Defining Semi-Automatic Test Prioritization

We run a lot of tests against Firefox, and running them takes a long time. Ideally, a best effort test selection is done on try, and if nothing obvious shakes out of that a change is landed on inbound. Inbound runs a selection of test jobs pared down based on coarse historical data (SETA). Modulo a handful of periodic build types, the remaining integration branches run a full set of tests and builds.

With the exception of the (entirely optional) selection made by try syntax, none of these steps codify the idea that some tests might fail given a particular change, and others are unlikely to fail or never will.

This post outlines one approach to implementing smarter (semi-automatic) test selection for Gecko.



Here’s a sketch of how this could work. This list doesn’t have a lot of regard for what I think is feasible in the short term. In particular, code coverage data for Mozilla’s code base is not readily available.

After this process, platforms and build types are still not known. Build types are things like opt, debug, and pgo, while platforms are things like Windows or B2G. In general we can’t expect a set of changed files to suggest a test platform and build type (many changes will impact code that runs across platforms and build types), but when all changes are in a subdirectory known to impact a single platform, we can be confident selecting only that platform. I expect these classifications are coarse and stable enough to capture effectively with annotations in files.

This is the framework I’m planning to use to get this off the ground in the coming months. Initial work around fleshing out file metadata is tracked in bug 1184405.

Feedback is welcome and appreciated.

August 06, 2015 05:50 PM

August 04, 2015

Mark Côté

MozReview auth changes

MozReview will soon be using Bugzilla’s new OAuth-like1 API keys and auth delegation. This is long overdue, and, in addition to providing security benefits, will eliminate all those confusing session-expired errors (e.g. bug 1178814).

After we deploy the change, all users will need to log back into MozReview’s Review Board2 instance. This time, rather than entering your Bugzilla credentials directly into Review Board, when you go to the “Log In” page, you’ll be redirected to BMO. If you don’t have a current BMO session, you’ll have to log into BMO. After you log in, or immediately after being redirected to BMO if you do have a session, you’ll be redirected back to Review Board and logged in. This is because, unlike most third-party apps, MozReview’s Review Board is a trusted app that is tightly integrated to BMO, so you won’t be confronted with the standard “Auth Delegation Request” intermediate page.

This is the first stage of conversion to API keys. For pushing review requests with Mercurial, you will still have to have either your Bugzilla username and password or your cookies in your .hgrc file or enter them on the command line at push time. However, Review Board will no longer store cookies; the username/password or login cookies will only be passed to BMO for verification and then discarded. We’ll be moving to API key usage on the command line in a subsequent patch.

Through API keys, Review Board will only have access to the specific BMO APIs required by MozReview. Those actions are mainly restricted to creating and updating attachments and posting comments; however, it will also need access to the login API until we support API keys on the command line. As noted, this will be used solely for identification, and no login tokens will be stored in MozReview.

Another big benefit of API keys is the elimination of those annoying and confusing expired-session errors. The BMO cookies used by MozReview have a limited lifespan, but API keys are good until explicitly revoked by the user. You can see the API key that is transferred to MozReview, as well as any other API keys you’ve manually or automatically created, in the API Keys tab in your BMO preferences. Revoking the API key won’t automatically log you out of Review Board, but you won’t be able to do any actions that interact with BMO (most actions) unless you log out and back in again (thus generating a new key).

You can follow along progress in bug 993233.

  1. No, it’s not exactly OAuth, but it’s based on similar ideas. We haven’t found a good OAuth 2 library for use with BMO, but we’re looking around.

  2. A note about names: MozReview generally refers to the full code-review system, which is primarily an hg server and a Review Board installation with extensions that we’ve developed. It also includes BMO, Autoland, Pulse, LDAP, and little things like code-review bots. When we say “Review Board”, we are referring specifically to the web app, which is the primary user interface to MozReview.

August 04, 2015 05:18 PM

Andreas Tolfsen

WebDriver now a living standard

The WebDriver specification is now officially a living standard. Practically this means that all changes are automatically published to

This brings an end to the era of forever outdated (two years in our case!) technical reports. It also helps bridge the disconnect many readers were having when they looked for information on our specification.

This is made possible with the Echidna tool that has recently been developed at the W3C. It integrates with Github and Travis, and lets you trigger the publishing steps when changes land on a specific branch in your source repository.

A possible future enhancement is abandoning the now superfluous master branch in favour of making the autopublishing gh-pages the default. The two-step landing process seems more tuned towards a levelled Editor’s Draft-to-Working Draft model.

Thanks to tripu and Michael[tm] Smith for doing the legwork.

August 04, 2015 09:49 AM

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

August 04, 2015 07:41 AM

July 31, 2015

Vaibhav Agrawal

First Month of Mozilla Internship

It has been a month since I started my Mozilla internship in San Francisco, but it feels like I had just started yesterday. I have been interning with the Automation and Tools team this summer and it has been a wonderful experience. As an intern in Mozilla, one gets goodies, new laptop, free food, and there are various intern events in which one gets to take part in. My mentor @chmanchester also gave me the freedom to decide what I wanted to work on which is quite unlike some other companies.

I chose to work on tools to improve developer productivity and making libraries that I have worked on in the past, more relevant. In the past month, I have been working on getting various triggering options inside Treeherder, which is a reporting dashboard for checkins to Mozilla projects. This involved writing AngularJS code for the front-end UI and python code for the backend. The process involved publishing the “triggering actions” to pulse, listening to those actions and then use the mozci library on the backend to trigger jobs. Currently if developers, particularly the sheriffs, want to trigger a lot of jobs, they have to do it via command line, and it involves context switching plus typing out the syntax.  To improve that, this week we have deployed three new options in Treeherder that will hopefully save time and make the process easier. They are:

* Trigger_Missing_Jobs: This button is to ensure that there is one job for every build in a push. The problem is that on integration branches we coalesce jobs to minimize load during peak hours, but many times there is a regression and sheriffs need to trigger all jobs for the push. That is when this button will come in handy, and one will be able to trigger jobs easily.

* Trigger_All_Talos_Jobs: As the name suggests, this button is to trigger all the talos jobs on a particular push. Being a perf sheriff, I need to trigger all talos jobs a certain number of times for a regressing push to get more data, and this button will aid me and others in doing that.

Screen Shot 2015-07-30 at 22.30.56Fig: shows “Trigger Missing Jobs” and “Trigger All Talos Jobs” buttons in the Treeherder UI

* Backfill_Job: This button is to trigger a particular type of job till the last known push on which that job was run. Due to coalescing certain jobs are not run on a series of pushes, and when an intermittent or bustage is detected, sheriffs need to find the root cause and thus manually trigger those jobs for the pushes. This button should aid them in narrowing down the root regressing revision for a job that they are investigating.

Screen Shot 2015-07-31 at 09.39.43Fig: shows “Backfill this job” button in the Treeherder UI

All of the above features right now only work for jobs that are on buildapi, but when mozci will have the ability to trigger tasks on taskcluster, they will be able to work on those jobs too. Also, right now all these buttons trigger the jobs which have a completed build, in future I plan to make these buttons to also trigger jobs which don’t have an existing build. These features are in beta and have been released for sheriffs, I would love to hear your feedback! A big shout out to all the people who have reviewed, tested and given feedback on these features.

July 31, 2015 04:48 PM

mozregression updates

Release 0.40

This is it, now mozregression uses fully the new build storage!

mozregression now downloads builds from In many cases this should be faster than before. Good news!

Also this 0.40 release fixes an intermittent windows bug introduced with 0.37, see bug 1185756.

July 31, 2015 12:00 AM

July 30, 2015

David Burns

Another Marionette release! Now with Windows Support!

If you have been wanting to use Marionette but couldn't because you only work on Windows, now is your chance to do so! All the latest downloads are available from our development github repository releases page

There is also a new page on MDN that walks you through the process of setting up Marionette and using it. I have only updated the python bindings so I can get a fell for how people are using it

Since you are awesome early adopters it would be great if we could raise bugs.

I am not expecting everything to work but below is a quick list that I know doesn't work.

Switching of Frames needs to be done with either a WebElement or an index. Windows can only be switched by window handles. This is currently how it has been discussed in the specification.

If in doubt, raise bugs!

Thanks for being an early adopter and thanks for raising bugs as you find them!

July 30, 2015 11:45 AM

Jonathan Griffin

A-Team Update, July 29, 2015


Treeherder: We’ve added to mozlog the ability to create error summaries which will be used as the basis for automatic starring.  The Treeherder team is working on implementing database changes which will make it easier to add support for that.  On the front end, there’s now a “What’s Deployed” link in the footer of the help page, to make it easier to see what commits have been applied to staging and production.  Job details are now shown in the Logviewer, and a mockup has been created of additional Logviewer enhancements; see bug 1183872.

MozReview and Autoland: Work continues to allow autoland to work on inbound; MozReview has been changed to carry forward r+ on revised commits.

Bugzilla: The ability to search attachments by content has been turned off; BMO documentation has been started at

Perfherder/Performance Testing: We’re working towards landing Talos in-tree.  A new Talos test measuring tab-switching performance has been created (TPS, or Talos Page Switch); e10s Talos has been enabled on all platforms for PGO builds on mozilla-central.  Some usability improvements have been made to Perfherder –

TaskCluster: Successful OSX cross-compilation has been achieved; working on the ability to trigger these on Try and sorting out details related to packaging and symbols.  Work on porting Linux tests to TaskCluster is blocked due to problems with the builds.

Marionette: The Marionette-WebDriver proxy now works on Windows.  Documentation on using this has been added at

Developer Workflow: A kill_and_get_minidump method has been added to mozcrash, which allows us to get stack traces out of Windows mochitests in more situations, particularly plugin hangs.  Linux xpcshell debug tests have been split into two chunks in buildbot in order to reduce E2E times, and chunks of mochitest-browser-chrome and mochitest-devtools-chrome have been re-normalized by runtime across all platforms.  Now that mozharness lives in the tree, we’re planning on removing the “in-tree configs”, and consolidating them with the previously out-of-tree mozharness configs (bug 1181261).

Tools: We’re testing an auto-backfill tool which will automatically retrigger coalesced jobs in Treeherder that precede a failing job.  The goal is to reduce the turnaround time required for this currently manual process, which should in turn reduce tree closure times related to test failures

The Details

Treeherder/Automatic Starring

Treeherder/Front End

Perfherder/Performance Testing


TaskCluster Support

Mobile Automation

Dev Workflow

Media Automation

General Automation



July 30, 2015 12:31 AM

July 29, 2015

Joel Maher

Lost in data – episode 2 – bisecting and comparing

This week on Lost in Data, we tackle yet another pile of alerts.  This time we have a set of changes which landed together and we push to try for bisection.  In addition we have an e10s only failure which happened when we broke talos uploading to perfherder.  See how I get one step closer to figuring out the root cause of the regressions.

July 29, 2015 11:53 PM

July 28, 2015

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

July 28, 2015 05:21 AM

July 27, 2015

Armen Zambrano G. (@armenzg)

Enabling automated back-filling on mozilla-inbound for few hours

tl;dr; we're going to enable automated back-filling tomorrow Tuesday for
few hours on mozilla-inbound.

We were aiming for Monday but pushed it to Tuesday to help publicize this more.

If on Wednesday there are no fall-outs we will leave it running for m-i for a week before enabling it on other places.

Posted on various mailing lists including

> Hello all,
> We are planning to turn on a service that automatically backfills
> failed test jobs on m-i. If there are no concerns, we would like to
> turn this on experimentally for a couple of hours on [Tuesday]. We
> hope this will make it easier to identify which revision broke a
> test. Suggestions are welcome.
> The backfilling works like this: - It triggers the job that failed
> one extra time - Then it looks for a successful run of the job on the
> previous 5 revisions. If a good run is found, it triggers the job
> only on revisions up to this good run. If not, it triggers the job on
> every one of the previous 5 revisions. Previous jobs will be
> triggered one time.
> The tracking bug is:
> Best, Alice

Zambrano Gasparnian, Armen
Automation & Tools Engineer

Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

July 27, 2015 06:56 PM

Christopher Manchester

Introducing mach try

This is a short introduction to mach try, a mach command that simplifies the process of selecting tests and pushing them to the try server.

To follow along at home you’ll either need to be using git cinnabar or have a modern mercurial and the hg “push-to-try” extension (available from |mach mercurial-setup|). Append —no-push to commands to keep them from pushing to the try server, and -n to see verbose messages associated with the results of commands.

# mach try is a command that takes try syntax and automates the steps
# to push the current tree to try with that syntax.
# For instance:

$ ./mach try -p win32 -u mochitest-bc

# ... will result in pushing "try: -b do -p win32 -u mochitest-bc -t none"
# to try. This saves dealing with mq or other ways of generating the try
# message commit. (An in-memory commit is generated with the appropriate
# message -- mq is not invoked at any stage).

# The more novel feature exposed by mach try is the ability to select
# specific test directories containing xpcshell, mochitests or reftests
# to run on the try server.

# For instance, if I've just made a change to one of the python libraries
# used by our test harnesses, and I'd like to quickly check that this
# feature works on windows. I can run:

$ ./mach try -p win64 testing/xpcshell testing/mochitest/tests

# This will result in the small number of xpcshell and mochitest tests
# that live next to their harnesses being run (in a single chunk) on
# try, so I can get my results without waiting for the entire suite,
# and I don't need to sift through logs to figure out which chunk a
# test lives in when I only care about running certain tests.

For more details run ./mach help try. As the command will inform you, this feature is under development — bugs should be filed blocking bug 1149670).

July 27, 2015 01:41 AM

July 26, 2015

Alice Scarpa


Thanks to Mozilla and Outreachy, I was able to spend one week visiting Mozilla’s Toronto office and it was awesome. My mentor, armenzg, really went out of his way to make me feel welcomed. It was a great way to feel like I’m part of the Mozilla community.

armenzg introduced me to Toronto mozillians so I could learn more about what is going on in the organization: ekyle gave me an ActiveData tour (so much data to play with!), mconley explained the e10s project to me (this time it is going to happen!), secretrobotron talked with me about community building (I want to get more involved in the Brazilian Mozilla community).

Thursday I went with the other interns and their mentors to a real-life Escape-the-room game. It was pretty fun and a great chance to meet people from a lot of different teams.

The best part of the whole week was Friday. armenzg invited me, wlach, ekyle and rwood to a work-from-his-house day and it was very pleasant. It was a great ending for an amazing week!

July 26, 2015 12:00 AM

July 23, 2015

Joel Maher

lost in data – episode 1, tackling a bunch of alerts

Today I recorded a session of me investigating talos alerts, It is ~35 minutes, sort of long, but doable in the background whilst grabbing breakfast or lunch!

I am looking forward to more sessions and answering questions that come up.

July 23, 2015 11:22 PM

David Burns

Microsoft ship a WebDriver implementation

Microsoft, the people still claim to be evil (who are actually big proponents of the the The Open Web), have... (wait for it...) SHIPPED. AN. IMPLEMENTATION. OF. WEBDRIVER!

At GTAC in California in 2011, Simon Stewart and I discussed that the Selenium project was at a crossroads (I am pretty sure beer was involved). We could ,and should, move this to the browser vendors. We had seen how in April the Chrome team had shipped their implementation and the Selenium project then deleted its implementation. I am pretty sure Daniel (Wagner-Hall) was glad to see it go.

In that time to now we have seen Mozilla get Marionette into Firefox and into release branches since Firefox 24 while slowly working on it (as well as Firefox OS support). We have seen Blackberry ship a version for the browser on their devices. We have seen mobile implementation with iOS-Driver, Selendroid and Appium.

The Spec is on track to be put forward for Recommendation by the end of the year. All the dreams that we (the Selenium Development team (my BFFs)) had are slowly coming true. This ship might be slow moving but it's mostly because some companies haven't always seen the value.


P.s. There is an open bug on the WebKit tracker for Safari support (and it is getting some internal push so I am hopeful!)

July 23, 2015 08:19 PM

Andreas Tolfsen

Optimised Rust code in Gecko

Rust Rust wave by Glen Scott (CC BY-NC 2.0)

Following bug 1177608, Rust code in Gecko is now compiled with optimisation by default.

Compilation of Rust components can be enabled by adding ac_add_options --enable-rust to your mozconfig file. For now there’s only limited usage of Rust in Gecko, but you can take a look at the bindings for the MP4 encoder if you’re interested. This is much thanks to the work of Ralph Giles.

Since optimised compiles are the default, you will now also get optimised output from rustc. You can disable this by setting ac_add_options --disable-optimize. This disables compilation for all cc, c++, and rustc compilers.

Next up is adding debug symbols, assertions, and adding some Rust autoconf macros.

July 23, 2015 06:10 PM

Alice Scarpa

Ijson is magic

I love Python, but it can be a little resource-hungry at times.

For example, allthethings.json is a 12MB json file. How much memory does a Python script take to load it? Let’s profile the memory usage of a script that just loads it.

import json

def load_json(filename):
    with open(filename, 'r') as f:
        return json.load(f)

if __name__ == "__main__":

Running memory_profiler on the script above:

$ python -m memory_profiler

Line #    Mem usage    Increment   Line Contents
     3   13.430 MiB    0.000 MiB   @profile
     4                             def load_json(filename):
     5   13.434 MiB    0.004 MiB       with open(filename, 'r') as f:
     6   86.773 MiB   73.340 MiB           return json.load(f)

73MB, more then 6 times the size of the original file.

This is not a big problem when loading a 12MB json on my desktop. But we needed to load a 200MB file on a Heroku standard-2x dyno, so we ended up having problems. What can help us in this case is that we only needed a subset of the data.

To use our allthethings.json example, imagine that the data we need is just the ‘shortname’ key for builders. Searching for a way to get just the data we needed without having to load the whole json in memory, I learned about ijson.

According to it’s documentation ijson is an:

Iterative JSON parser with a standard Python iterator interface

That means that instead of loading the whole file into memory and parsing everything at once, it uses iterators to lazily load the data. In that way, when we pass by a key that we don’t need, we can just ignore it and the generated object can be removed from memory.

import ijson

def load_json(filename):
    with open(filename, 'r') as fd:
        parser = ijson.parse(fd)
        ret = {'builders': {}}
        for prefix, event, value in parser:
            if (prefix, event) == ('builders', 'map_key'):
                buildername = value
                ret['builders'][buildername] = {}
            elif prefix.endswith('.shortname'):
                ret['builders'][buildername]['shortname'] = value

        return ret

if __name__ == "__main__":

Profiling its memory usage:

(venv)$ python -m memory_profiler

Line #    Mem usage    Increment   Line Contents
     5   15.109 MiB    0.000 MiB   @profile
     6                             def load_json(filename):
     7   15.109 MiB    0.000 MiB       with open(filename, 'r') as fd:
     8   15.109 MiB    0.000 MiB           parser = ijson.parse(fd)
     9   15.113 MiB    0.004 MiB           ret = {'builders': {}}
    10   26.012 MiB   10.898 MiB           for prefix, event, value in parser:
    11   26.012 MiB    0.000 MiB               if (prefix, event) == ('builders', 'map_key'):
    12   25.824 MiB   -0.188 MiB                   buildername = value
    13   25.824 MiB    0.000 MiB                   ret['builders'][buildername] = {}
    14   26.012 MiB    0.188 MiB               elif prefix.endswith('.shortname'):
    15   25.824 MiB   -0.188 MiB                   ret['builders'][buildername]['shortname'] = value
    17   26.012 MiB    0.188 MiB           return ret

Now loading the file takes 11MB and the script total memory usage went from 86MB to 26MB. On our real production example, total memory usage went from 4.5GB to 300MB.


Ijson gave us a magic level of memory-saving, but there was a speed trade-off. Using the yalj2 backend helped to reduce our loss in speed. To use the yajl backend all we had to do was:

import ijson.backends.yajl2 as ijson


$ time python

real    0m2.966s
user    0m2.944s
sys    0m0.020s


$ time python

real    0m1.626s
user    0m1.613s
sys    0m0.013s

The downside of using the yajl2 backend is that it requires users to have a C library (libyajl2) installed on their systems. Luckily, it is possible to install it on Heroku.

July 23, 2015 12:00 AM

July 22, 2015

Armen Zambrano G. (@armenzg)

Few mozci releases + reduced memory usage

While I was away adusca released few releases of mozci.

From the latest release I want to highlight that we're replacing the standard json library for ijson since it solves some memory leak issues we were facing for pulse_actions (bug 1186232).

This was important to fix since our Heroku instance for pulse_actions has an upper limit of 1GB of RAM.

Here are the release notes and the highlights of them:

Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

July 22, 2015 02:21 PM

July 21, 2015

Byron Jones

happy bmo push day!

the following changes have been pushed to

to improve security now serves attachments from a different domain – – instead of from a subdomain of  all existing links should continue to work, and will redirect to a url.

discuss these changes on

Filed under: bmo, mozilla

July 21, 2015 08:13 AM

July 17, 2015

mozregression updates

Release 0.38 and 0.39

The 0.38 release fixes a minor regression introduced with the 0.37 release (bug 1184915).

Just after that mozlog 3.0 was released, breaking compatibility with older mozbase packages, so we decided to ensure fully working dependencies and released mozregression 0.39! :)

July 17, 2015 12:00 AM

July 16, 2015

Jonathan Griffin

Automation and Tools Team Update, July 16 2015

The Automation and Tools Team (the A-Team, for short) is a large team that oversees a diverse set of services, tools and test harnesses used by nearly everyone at Mozilla.  We’re borrowing a page from Release Engineering and publishing a series of updates to inform people about what we’re up to, in the hopes of fostering better visibility and inter-team coordination.


Treeherder and Automatic Starring: Our focus for Treeherder in Q3 will be improving the signal-to-noise ratio for dealing with intermittent oranges. An overall design has been agreed to for the “automatic starring” project, and work has begun; final rollout is likely in Q4. This quarter, we’ll also stop spamming Bugzilla with comments for each intermittent, but we will put in place an alternate notification system for people who rely on Bugzilla orange comments to determine when an intermittent needs attention. We’ve also agreed on a redesign for the Logviewer that should result in a more useful and intuitive interface.
MozReview and Autoland:  MozReview now offers to publish review requests when you push, so it isn’t necessary to visit the MozReview’s UI. Work has started on adding support for autoland-to-inbound, which will allow developers to push changes to inbound directly from MozReview… no more battling tree closures!
Performance: Work continues on Perfherder’s “comparison mode”, a view that compares Talos performance data between two revisions. See wlach’s blog post for more details.
TaskCluster Support: We’re helping Release Engineering migrate from Buildbot to TaskCluster; this quarter we’re standing up Linux tests in TaskCluster and getting OS X cross-compilation to work so that we can move those builds to the cloud.
BMO now has tests running in continuous integration using TaskCluster and reporting to Treeherder.
Mobile Automation: mochitest-chrome for Android is now live! Work is also underway to enable debug reftests on Android emulators, and significant reliability improvements have been landed in Autophone.
Desktop Automation: Work is in progress to get Thread Sanitizer (TSan) builds running on try and to split gTest into its own test chunk. We’re also working towards applying –run-by-dir to mochitest-plain, in order to improve isolation and enable smarter chunking in CI.
Developer Workflow: We’re adding test-selection flexibility to the reftest harness as a prelude to making ‘mach try’ work with more test types.

The Details
Treeherder/Automatic Starring
  • Work has started on backend work needed to support automatic starring, including db simplification, and db unification (so each tree doesn’t have its own database).  Bug 1179263 tracks this work.  As a side effect of this work, Treeherder code should become less complex and easier to maintain.
  • Work has started on identifying what needs to happen in order to turn off Bugzilla comments for intermittents, and to create an alternative notification mechanism instead.  Bug 1179310 tracks this work.
Treeherder/Front End
  • New shortcuts for Logviewer, Delete Classification plus improved classification save
  • Design work is in progress for collapsing chunks in Treeherder in order to reduce visual noise in bug 1163064
Perfherder/Performance Testing
  • Evaluating alerts generated from PerfHerder
  • Improvements to compare chooser and viewer inside of PerfHerder
  • Work towards building a new tab switching test (bug 1166132)
  • Automatic publishing of reviews upon pushing
  • Known bug: people using cookie auth may experience bug 1181886
  • Better error message when MozReview’s Bugzilla session has expired (bug 1178811)
  • Pruned user database to improve user searching (bug 1171274)
  • Work is progressing on autoland-to-inbound (bug 1128039)
TaskCluster Support
  • Working on OSX cross-compilation, which will allow us to move OSX builds to the cloud; this will make OSX builds much faster in CI.
Mobile Automation
  • Autophone detects USB lock-ups and gracefully restarts. This is a huge improvement in system reliability.
  • Continued work on getting Android Talos tests ported to Autophone (bug 1170685)
  • Updated manifests and mozharness configs for mochitest-chrome (bug 1026290)
  • Determined total-chunks requirements for Android 4.3 Debug reftests (bug 1140471)
  • Re-wrote robocop harness to significantly improve run-time efficiency (bug 1179981)
Dev Workflow
  • Helped RelEng resolve some problems that were preventing them from landing mozharness in the tree.  This opens the door to a lot of future dev workflow improvements, including better unification of the ways we run automated tests in continuous integration and locally.  We’ve wanted this for years and it’s great to see it finally happen.
  • Did some work on top of jgraham’s patch to make mach use mozlog structured logging
Platform QA
  • We had to respond to the breakup of into several files to keep our Jenkins instance running.
  • Getting firefox-media-tests to satisfy Tier-2 Treeherder visibility requirements involves changing how Treeherder accommodates non-buildbot jobs (e.g bug 1182299)
General Automation
  • Working on running multiple tests/manifests through reftests harness as a prelude for supporting |mach try| for more test types.
  • Created patch to move mozlog.structured to the top level package (and what was previously there to mozlog.unstructured)
  • Figured out the series of steps needed to produce a usable Thread Sanitizer enabled linux build on our infra
  • Separating out gTest into a separate job in CI – bug 1179955.
  • More memory optimizations (motivation: releng query for Chris Atlee:  query slow tests)
    • run staging environment as stability test for production
    • change etl procedure so pushing changes to prod are easier (moving toward standard procedure)
  • import treeherder data markup to active data (motivation: characterizing test failures
    • ateam query: summary of test failures, stars and resolutions (bug 1161268bug 1172048)
    • subtests are too large for download of more than one day – working on code to only pull what’s required


July 16, 2015 05:11 PM

July 15, 2015

Christopher Manchester

Automatic triggering on try server

If you’ve pushed to Mozilla’s try server recently, you’ve probably noticed some of your failures being re-triggered automatically. This happens courtesy of the trigger bot tool implemented as a pulse listener that invokes self-serve in response to certain events.

Two trigerring rules are currently implemented:

Finally, if --rebuild is specified, nothing gets triggered when a test fails, and if --no-retry is found in try syntax, no triggering happens for the push.

This was enabled at the end of June, and I’ve gotten a bit of feedback (almost mostly positive). My own curiosity is about the overall volume of these jobs, so I pulled some stats from self-serve to see how much we’ve ended up triggering. For the first 2 weeks of July, there were 373,772 jobs on try, 13,099 of which were initiated by the trigger-bot (3.5%).

This is a larger proportion than I think it should be, but we’ve recently landed changes that mean we will trigger fewer hidden jobs. Hidden jobs are usually hidden due to failing persistently, so this should reduce overall triggering.

The rudimentary script I used to gather these stats is available on the trigger-bot repo

July 15, 2015 09:44 PM

July 14, 2015

William Lachance

Perfherder update

Haven’t been doing enough blogging about Perfherder (our project to make Talos and other per-checkin performance data more useful) recently. Let’s fix that. We’ve been making some good progress, helped in part by a group of new contributors that joined us through an experimental “summer of contribution” program.

Comparison mode

Inspired by Compare Talos, we’ve designed something similar which hooks into the perfherder backend. This has already gotten some interest: see this post on dev.tree-management and this one on dev.platform. We’re working towards building something that will be really useful both for (1) illustrating that the performance regressions we detect are real and (2) helping developers figure out the impact of their changes before they land them.

Screen Shot 2015-07-14 at 3.54.57 PM Screen Shot 2015-07-14 at 3.53.20 PM

Most of the initial work was done by Joel Maher with lots of review for aesthetics and correctness by me. Avi Halmachi from the Performance Team also helped out with the t-test model for detecting the confidence that we have that a difference in performance was real. Lately myself and Mike Ling (one of our summer of contribution members) have been working on further improving the interface for usability — I’m hopeful that we’ll soon have something implemented that’s broadly usable and comprehensible to the Mozilla Firefox and Platform developer community.

Graphs improvements

Although it’s received slightly less attention lately than the comparison view above, we’ve been making steady progress on the graphs view of performance series. Aside from demonstrations and presentations, the primary use case for this is being able to detect visually sustained changes in the result distribution for talos tests, which is often necessary to be able to confirm regressions. Notable recent changes include a much easier way of selecting tests to add to the graph from Mike Ling and more readable/parseable urls from Akhilesh Pillai (another summer of contribution participant).

Screen Shot 2015-07-14 at 4.09.45 PM

Performance alerts

I’ve also been steadily working on making Perfherder generate alerts when there is a significant discontinuity in the performance numbers, similar to what GraphServer does now. Currently we have an option to generate a static CSV file of these alerts, but the eventual plan is to insert these things into a peristent database. After that’s done, we can actually work on creating a UI inside Perfherder to replace alertmanager (which currently uses GraphServer data) and start using this thing to sheriff performance regressions — putting the herder into perfherder.

As part of this, I’ve converted the graphserver alert generation code into a standalone python library, which has already proven useful as a component in the Raptor project for FirefoxOS. Yay modularity and reusability.

Python API

I’ve also been working on creating and improving a python API to access Treeherder data, which includes Perfherder. This lets you do interesting things, like dynamically run various types of statistical analysis on the data stored in the production instance of Perfherder (no need to ask me for a database dump or other credentials). I’ve been using this to perform validation of the data we’re storing and debug various tricky problems. For example, I found out last week that we were storing up to duplicate 200 entries in each performance series due to double data ingestion — oops.

You can also use this API to dynamically create interesting graphs and visualizations using ipython notebook, here’s a simple example of me plotting the last 7 days of pageload data inline in a notebook:

Screen Shot 2015-07-14 at 4.43.55 PM


July 14, 2015 08:51 PM

Julien Pagès

mozregression 0.37 release

Yesterday we just released mozregression (command line regression range finder for Mozilla nightly and inbound builds) 0.37!

This release include some new features and fixes, and uses the new TaskCluster system from Mozilla Release Engineering to retrieve integration branch information. This considerably speeds up the bisection of inbound builds.

Try it out!

July 14, 2015 04:29 PM

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

July 14, 2015 07:17 AM

July 13, 2015

Julien Pagès

Python [aggressive] pep8 conversion using autopep8

Some time ago, I started to look at Talos, the python performance testing framework for firefox that is usable on Windows, Mac and Linux.

Talos has been around for a long time, and has seen many contributors! Unfortunately the codebase reflects that, and as an example there was simply no common code style around for all the Python files.

Sometimes a 2 spaces based indentation, sometimes 4 spaces based, sometimes something else. No limit for the lines length. This end up with something that is really hard to read, understand and maintain. So one of my first goal was to try to clean this up.

And I heard about autopep8. As the name suggests, this is a tool (command line) that will automatically do some pep8 conversion on Python source files.

Awesome! I highly recommend that to you if you are trying to adapt the coding style of some Python code to pep8 convention, this worked really well for me. This is a pain to do that by hand, and almost impracticable when the indentation needs to be fixed, but autopep8 makes that for you in a safe way.

Tip: the –aggressive flag is really nice for fixing long lines.

So, as an example this is how I used autopep8 on talos:

autopep8 --recursive --in-place --aggressive --aggressive /path/to/talos

Simple and effective. See the usage documentation to see with an example what it really does and other command line flags.

July 13, 2015 09:21 AM

mozregression updates

Release 0.37

I’m excited to announce that mozregression now supports using taskcluster for inbound bisection with this release!

This considerably speeds up the process of downloading builds from integration branches (like mozilla-inbound) and also supports work by Mozilla Release Engineering to migrate build storage from ftp to s3.


New features:

July 13, 2015 12:00 AM

July 12, 2015

Julien Pagès

RunSnakeRun – graphical visualisation of dumped python profiling data

If you are a Python developer like me, you probably know the profile and cProfile modules that provides deterministic profiling of Python programs.

These modules are awesome – however, when it comes to analysing the data to improve your program, the provided pstats module is generally not powerful enough if you have quite a large codebase.

And here graphical tools comes in handy! I tried RunSnakeRun, and this is a really great program that allows you to analyse the profiling data under multiple angles (a nice view is by file), so you can find easily the bottlenecks and fix them.

RunSnakeRun helped me to improve the “mach help” command. It is a cross platform tool. Note that if you are on GNU/Linux and that you have the KDE desktop (or don’t mind to install the required KDE dependencies), KCacheGrind can be used with Python also.

July 12, 2015 04:12 PM

July 07, 2015

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

July 07, 2015 06:11 AM

July 06, 2015

Armen Zambrano G. (@armenzg)

mozci 0.8.2 - Allow using TreeHerder as a query source

In this release we have added an experimental feature where you can use Treeherder as your source for jobs' information instead of using BuildApi/Buildjson.
My apologies as this should have been a minor release (0.9.0) instead of a security release (0.8.2).


Thanks to @adusca @vaibhavmagarwal and @chmanchester for their contributions.
Our latest new contributor is @priyanklodha - thank you!

How to update

Run "pip install -U mozci" to update

Major highlights

  • Added --query-source option to get data from Treeherder or Buildapi
  • Improved usage of OOP to allow for different data sources seamlessly

Minor improvements

  • Better documentation of --times
  • Cleaning up old builds-*.js files
  • Enforced line character limit

All changes

You can see all changes in here:

Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

July 06, 2015 02:58 PM

July 02, 2015

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

July 02, 2015 05:59 AM

July 01, 2015

David Burns

Who wants to be an alpha tester for Marionette?

Are you an early adopter type? Are you an avid user of WebDriver and want to use the latest and great technology? Then you are most definitely in luck.

Marionette, the Mozilla implementation of the FirefoxDriver, is ready for a very limited outing. There is a lot of things that have not been implemented or, since we are implementing things agains the WebDriver Specification they might not have enough prose to implement (This has been a great way to iron out spec bugs).

Getting Started

At the moment, since things are still being developed and we are trying to do things with new technologies (like writing part this project using Rust) we are starting out with supporting Linux and OS X first. Windows support will be coming in the future!

Getting the driver

We have binaries that you can download. For Linux and for OS X . The only bindings currently updated to work are the python bindings that are available in a branch on my fork of the Selenium project. Do the following to get it into a virtualenv:
  1. Create a virtualenv
  2. activate your virtualenv
  3. cd to where you have cloned my repository
  4. In a terminal type the following: ./go py_install

Running tests

Running tests against marionette requires that you do the following changes (which hopefully remains small)

Update the desired capability to have marionette:true and add binary:/path/to/Firefox/DeveloperEdition/or/Nightly. We are only supporting those two versions of at the moment because we have had a couple in compatible issues that we have fixed which means speaking to Marionette in the browser in the beta or release versions quite difficult.

Since you are awesome early adopters it would be great if we could raise bugs. I am not expecting everything to work but below is a quick list that I know doesn't work.

Thanks for being an early adopter and thanks for raising bugs as you find them!

July 01, 2015 07:57 PM

June 30, 2015

Geoff Brown

Firefox for Android Performance Measures – Q2 Check-up

This review of Android performance measurements covers the period April 1 – June 30: the second quarter of 2015. I will write these summary posts on a quarterly basis from now on.


– Most tests fairly steady over the quarter.


This section tracks Perfomatic graphs from for mozilla-central builds of Firefox for Android, for Talos tests run on Android 4.0 Opt. The test names shown are those used on treeherder. See for background on Talos.


Measure of “checkerboarding” during simulation of real user interaction with page. Lower values are better.


19 (start of period) – 20 (end of period)

Small regression on June 10 (no bug?). This test is exhibiting some inconsistent behavior, as noted in bug 1149567.


An svg-only number that measures SVG rendering performance. About half of the tests are animations or iterations of rendering. This ASAP test (tsvgx) iterates in unlimited frame-rate mode thus reflecting the maximum rendering throughput of each test. The reported value is the page load time, or, for animations/iterations – overall duration the sequence/animation took to complete. Lower values are better.


720 (start of period) – 680 (end of period).

Small improvement on May 27.


Generic page load test. Lower values are better.


680 (start of period) – 670 (end of period).

Small improvement on May 18.

Throbber Start / Throbber Stop

These graphs are taken from  Browser startup performance is measured on real phones (a variety of popular devices).

Unfortunately, I could not find any devices for mozilla-central which reported consistently throughout this period.




These graphs are taken from Eideticker is a performance harness that measures user perceived performance of web browsers by video capturing them in action and subsequently running image analysis on the raw result.

More info at:




Most other tests have incomplete data for this time period.


These graphs are taken from the mozbench dashboard at which includes some comparisons involving Firefox for Android. More info at





June 30, 2015 10:41 PM

June 29, 2015

Alice Scarpa

Changing goals

I started my Outreachy internship with Mozilla on May 25th, working with mozci. A lot happened in the first five weeks:

Part of the application process was writing a project proposal. Mozci was about 2 months old at the time and so much changed since then that strictly following the original proposal would have been a bad idea.

Our current plan for the rest of my internship is to focus on using Pulse Actions to add new functionality to Treeherder, but a lot can happen until then (for example, we depend a lot on feedback from the sheriffs to guide us). I’m not nervous because I know I can count with the help of people from Outreachy, the Recurse Center and the A-team, especially my mentor Armen who has been awesome since I started contributing to mozci a long long time ago (four whole months!).

June 29, 2015 12:00 AM

June 16, 2015

Armen Zambrano G. (@armenzg)

mozci 0.8.0 - New feature -- Trigger coalesced jobs + improved performance

Beware! This release is full awesome!! However, at the same time new bugs might pop up so please let us know :)
You can trigger now all coalesced jobs on a revision:
mozci-trigger --coalesced --repo-name REPO -r REVISION


Thanks to @adusca @glandium @vaibhavmagarwal @chmanchester for their contributions on this release.

How to update

Run "pip install -U mozci" to update

Major highlights

  • #259 - New feature - Allow triggering
  • #227 - Cache files as gzip files instead of uncompressed
    • Less disk print
  • #227 - Allow using multiple builders
  • 1e591bf - Make sure that we do not fetch files if they are not newer
    • We were failing to double check that the last modification date of a file was the same as the one in the server
    • Hence, we were downloading files more often than needed
  • Caching builds-4hr on memory for improved performance

Minor improvements

  • f72135d - Backfilling did not accept dry_run or to trigger more than once
  • More tests and documents
  • Support for split tests (test packages json file)
  • Some OOP refactoring

All changes

You can see all changes in here:

Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

June 16, 2015 05:38 PM

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

June 16, 2015 06:42 AM

June 13, 2015

Syd Polk

WWDC Day 4 - ...and that's a wrap

And the last day...

I had lunch with Emily Toop on the Firefox iOS team. She is new to Mozilla, but an experienced iOS developer out of the UK. Lovely venue at the Samovar Tea Lounge in Yerba Buena Gardens.

Overall impressions of WWDC:
I would be happy to come back next year, but I am not going to push again unless programming the Mac or iOS is my primary job. Lots of good stuff. Also, there are some sessions from last year I need to watch.

June 13, 2015 12:35 AM

June 12, 2015

Syd Polk

WWDC Day 3 - Sessions are starting to run together...

We are in the thick of things now; next to last day. Sessions I attended:

I had lunch with my good buddy Jim Ingham. We worked together at Sun, Cygnus, Red Hat and Apple, off and on from 1997-2006. It was good to see him. I wish I could remember the last name of the other Apple engineer who joined us, Greg. He had been there a few months when I left, and he had moved from Austin (!). It was good to eat a good lunch away from the show.

I also went to the Apple Bash.  They had the band Walk the Moon, and I was pleasantly surprised. There was actually musicianship on display. I am not going to buy anything, but for once, I wasn't counting time until the band stopped.

It was not a dinner outing. There were small entree-like snacks, but it's not the same. So Paul and I found a late-night place near Union Square.

June 12, 2015 04:56 PM

June 11, 2015

Syd Polk

WWDC Day 2 - more sessions and a party

Lots of sessions today. Let's tear right in:

After the main conference, I went to see James Dempsey and the Breakpoints. Uneven songs musically, but the words are all about programming in Cocoa, and it's pretty funny. Check it out on iTunes if you are into that sort of thing.

And I closed the night having an excellent cocktail with my best friend, Paul Tien-Shih Lee, whose birthday it was. The View bar in the Marriot is wonderful.

June 11, 2015 03:56 PM

June 10, 2015

Syd Polk

WWDC Day 1 - Sessions, Sessions, and some old friends

Tuesday was filled with sessions, with breaks in the Developer Tools lab. Sessions I attended:

I also spent some time in the Developer Tools Lab catching up with former coworkers. Plus, I discovered an Xcode 7 bug launching the simulator, and I talked to them about that.

After the show, I met some friends from college and/or my very first projects for dinner in San Mateo. I was truly great to see them.

June 10, 2015 04:21 PM

Alice Scarpa

Tracking down a non-deterministic coverage failure

At Mozilla CI Tools, we use Coveralls to track our test coverage progress. In the past month we noticed that sometimes the coverage changes even in PRs that only changed files that are not tracked by coveralls. How come?

Comparing coverage reports for builds in which this problem happens, I noticed that this line would sometimes be marked as covered, sometimes as not covered. The line belongs to a function that builds a graph from a list. An interesting thing about this function is that the order of the list does not affect the final graph, but it does affect the code path taken inside the function. In the test for this function, the input comes from a call to a dict.keys() method. Also, an interesting thing about the .keys() method is that it always produces a list containing the same elements, but the order varies. Mystery solved!

So a single sort() command on the test input could fix those annoying random coverage failures!

June 10, 2015 12:00 AM

June 09, 2015

Joel Maher

Please welcome the Dashboard Hacker team

A few weeks ago we announced that we would be looking for committed contributors for 8+ weeks on Perfherder.  We have found a few great individuals, all of whom show a lot of potential and make a great team.

They are becoming familiar with the code base and are already making a dent in the initial list of work set aside.  Let me introduce them (alphabetical order by nicks):

akhileshpillai – Akhilesh has jumped right in and started fixing bugs in Perfherder.  He is new to Mozilla and will fit right in.  With how fast he has come up to speed, we are looking forward to what he will be delivering in the coming weeks.  We have a lot of UI workflow as well as backend data refactoring work on our list, all of which he will be a great contributor towards.

mikeling – mikeling has been around Mozilla for roughly two years, recently he started out helping with a few projects on the A*Team.  He is very detailed oriented, easy to work with and is willing to tackle big things.

theterabyte – Tushar is excited about this program as an opportunity to grow his skills as a python developer and experiencing how software is built outside of a classroom.  Tushar will get a chance to grow his UI skills on Perfherder by making the graphs and compare view more polished and complete, while helping out with an interface for the alerts.

Perfherder will soon become the primary dashboard for all our performance needs.

I am looking forward to the ideas and solutions these new team members bring to the table.  Please join me in welcoming them!

June 09, 2015 04:55 PM

Please join me in welcoming the DX Team

A few weeks ago, I posted a call out for people to reach out and commit to participate for 8+ weeks.  There were two projects and one of them was Developer Experience.  Since then we have had some great interest, there are 5 awesome contributors participating (sorted by irc nicks).

BYK – I met BYK 3+ years ago on IRC- he is a great person and very ambitious.  As a more senior developer he will be focused primarily on improving interactions with mach.  While there are a lot of little things to make mach better, BYK proposed a system to collect information about how mach is used.

gma_fav – I met gma_fav on IRC when she heard about the program.  She has a lot of energy, seems very detail oriented, asks good questions, and brings fresh ideas to the team!  She is a graduate of the Ascend project and is looking to continue her growth in development and open source.  Her primary focus will be on the interface to try server (think the try chooser page, extension, and taking other experiments further).

kaustabh93 – I met Kaustabh on IRC about a year ago and since then he has been a consistent friend and hacker.  He attends university.  In fact I do owe him credit for large portions of alert manager.  While working on this team, he will be focused on making run-by-dir a reality.  There are two parts: getting the tests to run green, and reducing the overhead of the harness.

sehgalvibhor – I met Vibhor on IRC about 2 weeks ago.  He was excited about the possibility of working on this project and jumped right in.  Like Kaustabh, he is a student who is just finishing up exams this week.  His primary focus this summer will be working in a similar role to Stanley in making our test harnesses act the same and more useful.

stanley – When this program was announced Stanley was the first person to ping me on IRC.  I have found him to be very organized, a pleasure to chat with and he understands code quite well.  Coding and open source are both new things to Stanley and we have the opportunity to give him a great view of it.  Stanley will be focusing on making the commands we have for running tests via mach easier to use and more unified between harnesses.

Personally I am looking forward to seeing the ambition folks have translate into great solutions, learning more about each person, and sharing with Mozilla as a whole the great work they are doing.

Take a few moments to say hi to them online.

June 09, 2015 04:32 PM

Syd Polk

WWDC Day 0, Part 2 - Platforms State of the Union

The Platforms State of the Union session was much more interesting than the keynote. Highlights applicable to what I work on:
You should do those things anyway.
Great session.

June 09, 2015 04:14 PM

David Burns

WebDriver Specification - Have you read it lately?

A lot of work has gone into the WebDriver Specification this year. The methods in there have had a major make over to make them more specific in the steps that are required as well as having the relevant links. Go have a read of it and feel free to raise bugs against it, we will be updating it quite regularly. You can see all the work that is happening on Github. We do everything via pull requests so you can read things before they land.

My team have also been working hard at making sure that our implementation is following the specification and are making some great leaps with it. I will be releasing a development version of the python bindings soon that use the httpd, like InternetExplorerDriver and ChromeDriver, to drive the browser. Currently our httpd only works against Nightly but there is a merge to Aurora happening soon when we will be sending out links for you to start playing with it all. I am actually looking forward to some of the feedback that we get about it.

June 09, 2015 02:42 PM

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

June 09, 2015 06:37 AM

June 08, 2015

Syd Polk

WWDC Day 0, Part 1 - Keynote

First of all, there are a lot of people here. Last night, my buddy told me that I might want to line up early; otherwise, I might have ended up in the overflow room.

And my Mozilla colleagues, Emily Toop and Darrin Henin, wanted to hook up early. So I set out to Moscone West and arrived at 6:30. I initially went to the back of the line. It went north on 4th, all of the way west on Minna, back South on 5th, and then almost completing the circle halfway back up Howard. 

Fortunately, the other Mozillians found some of their friends who work at Facebook, and got us a place at the front of the line. Hooray.

They let us into the building at 8. We went up the escalators, and then walked all of the way around the second floor, where we waited in the (hot, crowded) hallway for another hour.

At nine, they let us crowd around the base of the escalator to the 3rd floor. We were there for about half an hour when we were finally let into the main hall.

The keynote itself was a little disappointing. New versions of OS X and iOS with some whizzy features. My favorite was the multiapp features in iPad; looking forward to playing with those. My single favorite announcement was that Swift is now going to be Open Source and run on Linux. They then talked about watchOS, which I currently have no interest in.

And then they wasted 45 minutes with the Apple Music announcement. OK, I can stream all of Apple's catalog. But I have to either stream or play my library? Wasn't clear. And the performance did nothing for me.

So. OS X El Capitain and iOS 9: Yay. Looks good. Carry on. 
watchOS: Cool, I guess. 
MUSIC: meh.

June 08, 2015 08:45 PM

Dave Hunt

Joining Web QA

Dylan with origami foxI’m excited to announce that as of last week I am officially on Mozilla’s Web QA team! Despite working closely with the team since I started at Mozilla over four years ago, I’ve always reported to another team. Originally I had a hybrid role, where I reported to the Director of QA and assisted with automation of both Web and Desktop products. Later, we formed the QA Automation Services team, which existed to provide automation services to any QA team. This was mostly absorbed into the A-Team, which shares a lot of the same goals but is not just focused on QA. During my time with the A-Team a lot of my work started to shift towards Firefox OS, so it made sense during a organisational shift towards vertical stacks for me to officially join the Firefox OS automation team.

Many times since I started at Mozilla I’ve felt that I had more to offer the Web QA team, and I’ve always been a keen contributor to the team. I can’t say it was an easy decision to move away from Firefox OS, as it’s a terrifically exciting project, but the thought of joining Web QA just had me bursting with enthusiasm! In a time where there’s been a number of departures, I’m pleased to say that I feel like I’m coming home. Look out web – you’re about to get even more automated!

Last week Stephen Donner interviewed me on being a Mozilla Web QA contributor. You can read the blog post over on the Web QA blog.

I’d like to take this opportunity to thank Stephen Donner, Matt Evans, Clint Talbert, Jonathan Griffin, and James Lal for their support and leadership. I’ve made so many great friendships at Mozilla, and with our Whistler work week just around the corner I’m so looking forward to catching up with many of them in person!

June 08, 2015 09:42 AM

Syd Polk

WWDC - D -1

When WWDC signups went up, I decided to take a chance and see if I could score a ticket. Much to my surprise, I did get one. So, here I am, in San Francisco, the night before it starts, and I am about to go for the first time since 2005.

In that WWDC, I was an Apple employee. I worked on the Developer Tools team. We were shuffled into the Apple Employee room for the Keynote.  At the end of the keynote, Steve Jobs announced that Apple was moving the Mac product line to Intel. I was on the team that was "in the know" about the transition. SJ asked us all to stand up, and the rest of the employees gave us a standing O. Was one of the proudest moments of my career.

Since then, I have moved to Austin and have worked several jobs. Currently, I work in automation for Mozilla. I hope to do some development work someday, and Mozilla has an effort underway for iOS, so I am going to be attending sessions and talking with people here about that effort. I am also going to sessions for my personal projects.

I checked in today, and they did something really smart. They gave us a windbreaker. Anybody who has been to San Francisco in June knows that this is really smart. There are probably many attendees who have never been here and don't know that June and July are cold, foggy and windy here.

This is not the first year they have done this (I saw somebody with a "14" on his back), but it is really brilliant.

Tomorrow is dominated by the Keynote, and in the afternoon, the only things to see are Platforms State of the Union and the Apple Design awards.

I'm stoked!

June 08, 2015 01:44 AM

June 07, 2015

mozregression updates

Release 0.37

I’m excited to announce that mozregression now supports using taskcluster for inbound bisection with this release!

This considerably speeds up the process of downloading builds from integration branches (like mozilla-inbound) and also supports work by Mozilla Release Engineering to migrate build storage from ftp to s3.


New features:

June 07, 2015 12:00 AM

June 06, 2015

Julien Pagès

Python code cleanup – vulture

I’m pretty sure you all already know about flake8, a great tool that combine other tools (PyFlakes, pep8 and Ned Batchelder’s McCabe script) to check style issues and code errors.

Working on Talos, I needed to find a way to find dead code on the codebase (the less code, the better!). I found a pretty good tool that helped me: vulture. This is a one shot tool – it can not be automated like flake8 as there is some false positives, but still running it once on an old codebase may be a big win to find and remove useless stuff.

Obviously it is easier to use on python programs – you will have to whitelist a lot of things if you run this on a python library that offers a public API.

If you want to seek and destoy dead code on your python program, give it a try!

June 06, 2015 09:46 AM

June 05, 2015

Alice Scarpa


Recently we had an ‘UnboundLocalError’ in mozci. A local variable was being referenced before assignment. Turns out we were only defining the variable in one branch of an ‘if’ statement. It was very straightforward to fix. What bothered me was that this is the exact sort of error that would have been caught at compile time in other languages.

Is there a Python static analysis tool that would have caught this? I wrote a very simple module with the same bug and tried with different tools1:

"""This module bakes pizzas."""

def bake_pizza():
    """Bake a pizza."""
    number = int(raw_input())

    if number > 3:
        additional = 2
        number += 1
    print number + additional

if __name__ == '__main__':

Neither Pylint, Pyflakes nor PyChecker were able to notify me about the possibility of an ‘UnboundLocalError’.

ekyle suggested that PyCharm might be able to solve this problem, and indeed, PyCharm code inspection shows the message I wanted:

>  Local variable 'additional' might be referenced before assignment

I spent a couple of hours today going through every PyCharm warning in mozci to see if there were any other bugs that it could catch. It didn’t find other serious issues (I don’t know if that’s fortunate or not).

  1. It ended up being the only module I’ve ever written that scored 10.0/10 in pylint.

June 05, 2015 12:00 AM

June 03, 2015

Armen Zambrano G. (@armenzg)

mozci 0.7.2 - Support b2g jobs that still run on Buildbot

There are a lot of b2g (aka Firefox OS) jobs that still run on Buildbot .
Interestingly enough we had not tried before to trigger one with mozci.
This release adds support for it.
This should have been a minor release (0.8.0) rather than a security release (0.7.2). My apologies!
All jobs that start with "b2g_" in all_builders.txt are b2g jobs that still run on Buildbot instead of TaskCluster (docs - TC jobs on treeherder).

Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

June 03, 2015 07:23 PM

mozci 0.7.1 - regression fix - do not look for files for running jobs

This release mainly fixes a regression we introduced in the release 0.7.0.
The change (#220) we introduced checked completed and running jobs for files that have been uploaded in order to trigger tests.
The problem is that running jobs do not have any metadata until they actually complete.
We fixed this on #234.


Thanks to @adusca and @glandium for their contributions on this release.

How to update

Run "pip install -U mozci" to update

Major highlights

  • #234 - (bug fix) - Do not try to find files for running jobs
  • #228 - For try, only trigger talos jobs on existing build jobs ** rather than triggering builds for platforms that were not requested
  • #238 - Read credentials through environment variables

Minor improvements

  • #226 - (bug fix) Properly cache downloaded files
  • #228 - (refactor) Move SCHEDULING_MANAGER
  • #231 - Doc fixes

All changes

You can see all changes in here:

Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

June 03, 2015 07:14 PM

Joel Maher

re-triggering for a [root] cause- version 2.0 – a single bug!

Today the orange factor email came out- the top 10 bugs with stars on them :)  Last week we had no bugs that we could take action on, and the week before we had a few bugs to take action on.

This week I looked at each bug again, annotated them with some notes as to what I did or why I didn’t do anything, here are the bugs:

It is nice to find a bug that we can take action on.  What is interesting is the bug has been around for a while, but we noticed about May 21 that the rate of failures went up from a couple a day to >5/day.  Details:

I might try a full experiment soon blindly looking at bugs instead of using orange factor.

June 03, 2015 10:46 AM

June 02, 2015

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

June 02, 2015 08:03 AM

May 28, 2015

Joel Maher

the orange factor – no need to retrigger this week

last week I did another round of re-triggering for a root cause and found some root causes!  This week I got an email from orange factor outlining the top 10 failures on the trees (as we do every week).

Unfortunately as of this morning there is no work for me to do- maybe next week I can hunt.

Here is the breakdown of bugs:

As you can see there isn’t much to do here.  Maybe next week we will have some actions we can take.  Once I have about 10 bugs investigated I will summarize the bugs, related dates, and status, etc.

May 28, 2015 04:15 PM

Armen Zambrano G. (@armenzg)

mozci 0.7.0 - Less network fetches - great speed improvements!

This release is not large in scope but it has many performance improvements.
The main improvement is to have reduced the number of times that we fetch for information and use a cache where possible. The network cost was very high.
You can read more about in here:


Thanks to @adusca @parkouss @vaibhavmagarwal for their contributions on this release.

How to update

Run "pip install -U mozci" to update

Major highlights

  • Reduce drastically the number of requests by caching where possible
  • If a failed build has uploaded good files let's use them
  • Added support for retriggering and cancelling jobs
  • Retrigger a job once with a count of N instead of triggering individually N times

Minor improvements

  • Documenation updates
  • Add badge

All changes

You can see all changes in here:

Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

May 28, 2015 02:01 PM

Byron Jones

happy bmo push day!

it appears to be “bmo push week“.  today’s push addresses some issues caused by yesterday’s push.

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

May 28, 2015 07:06 AM

Alice Scarpa

Acting upon Pulse messages

Pulse Actions is a service that listens to retrigger/cancel messages on exchange/treeherder/v1/job-actions and acts upon them using Mozilla CI Tools.

The project was deployed to Heroku today in dry-run mode, i.e., instead of actually making retrigger/cancel requests, Pulse Actions is just logging the requests it would have made.

If this experiment works, it might be possible for Treeherder to use the service for retrigger/cancel requests. That would eliminate the need for users signed in with Persona to enter LDAP credentials to make TH requests (bug 1077053) and would also open a lot of new possibilities (e.g. adding a backfill button to TH using mozci).

This project’s progress is being tracked in bug 1168148.

May 28, 2015 12:00 AM

May 27, 2015

Armen Zambrano G. (@armenzg)

Welcome adusca!

It is my privilege to announce that adusca (blog) joined Mozilla (since Monday) as an Outreachy intern for the next 4 months.

adusca has an outstanding number of contributions over the last few months including Mozilla CI Tools (which we're working on together).

Here's a bit about herself from her blog:
Hi! I’m Alice. I studied Mathematics in college. I was doing a Master’s degree in Mathematical Economics before getting serious about programming.
She is also a graduate from Hacker's School.

Even though Alice has not been a programmer for many years, she has shown already lots of potential. For instance, she wrote a script to generate scheduling relations for buildbot; for this and many other reasons I tip my hat.

adusca will initially help me out with creating a generic pulse listener to handle job cancellations and retriggers for Treeheder. The intent is to create a way for Mozilla CI tools to manage scheduling on behalf of TH, make the way for more sophisticated Mozilla CI actions and allow other people to piggy back to this pulse service and trigger their own actions.

If you have not yet had a chance to welcome her and getting to know her, I highly encourage you to do so.

Welcome Alice!

Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

May 27, 2015 05:10 PM

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

May 27, 2015 07:11 AM