Planet Mozilla Automation

July 31, 2015

Vaibhav Agrawal

First Month of Mozilla Internship

It has been a month since I started my Mozilla internship in San Francisco, but it feels like I had just started yesterday. I have been interning with the Automation and Tools team this summer and it has been a wonderful experience. As an intern in Mozilla, one gets goodies, new laptop, free food, and there are various intern events in which one gets to take part in. My mentor @chmanchester also gave me the freedom to decide what I wanted to work on which is quite unlike some other companies.

I chose to work on tools to improve developer productivity and making libraries that I have worked on in the past, more relevant. In the past month, I have been working on getting various triggering options inside Treeherder, which is a reporting dashboard for checkins to Mozilla projects. This involved writing AngularJS code for the front-end UI and python code for the backend. The process involved publishing the “triggering actions” to pulse, listening to those actions and then use the mozci library on the backend to trigger jobs. Currently if developers, particularly the sheriffs, want to trigger a lot of jobs, they have to do it via command line, and it involves context switching plus typing out the syntax.  To improve that, this week we have deployed three new options in Treeherder that will hopefully save time and make the process easier. They are:

* Trigger_Missing_Jobs: This button is to ensure that there is one job for every build in a push. The problem is that on integration branches we coalesce jobs to minimize load during peak hours, but many times there is a regression and sheriffs need to trigger all jobs for the push. That is when this button will come in handy, and one will be able to trigger jobs easily.

* Trigger_All_Talos_Jobs: As the name suggests, this button is to trigger all the talos jobs on a particular push. Being a perf sheriff, I need to trigger all talos jobs a certain number of times for a regressing push to get more data, and this button will aid me and others in doing that.

Screen Shot 2015-07-30 at 22.30.56Fig: shows “Trigger Missing Jobs” and “Trigger All Talos Jobs” buttons in the Treeherder UI

* Backfill_Job: This button is to trigger a particular type of job till the last known push on which that job was run. Due to coalescing certain jobs are not run on a series of pushes, and when an intermittent or bustage is detected, sheriffs need to find the root cause and thus manually trigger those jobs for the pushes. This button should aid them in narrowing down the root regressing revision for a job that they are investigating.

Screen Shot 2015-07-31 at 09.39.43Fig: shows “Backfill this job” button in the Treeherder UI

All of the above features right now only work for jobs that are on buildapi, but when mozci will have the ability to trigger tasks on taskcluster, they will be able to work on those jobs too. Also, right now all these buttons trigger the jobs which have a completed build, in future I plan to make these buttons to also trigger jobs which don’t have an existing build. These features are in beta and have been released for sheriffs, I would love to hear your feedback! A big shout out to all the people who have reviewed, tested and given feedback on these features.

July 31, 2015 04:48 PM

mozregression updates

Release 0.40

This is it, now mozregression uses fully the new build storage!

mozregression now downloads builds from In many cases this should be faster than before. Good news!

Also this 0.40 release fixes an intermittent windows bug introduced with 0.37, see bug 1185756.

July 31, 2015 12:00 AM

July 30, 2015

David Burns

Another Marionette release! Now with Windows Support!

If you have been wanting to use Marionette but couldn't because you only work on Windows, now is your chance to do so! All the latest downloads are available from our development github repository releases page

There is also a new page on MDN that walks you through the process of setting up Marionette and using it. I have only updated the python bindings so I can get a fell for how people are using it

Since you are awesome early adopters it would be great if we could raise bugs.

I am not expecting everything to work but below is a quick list that I know doesn't work.

Switching of Frames needs to be done with either a WebElement or an index. Windows can only be switched by window handles. This is currently how it has been discussed in the specification.

If in doubt, raise bugs!

Thanks for being an early adopter and thanks for raising bugs as you find them!

July 30, 2015 11:45 AM

Jonathan Griffin

A-Team Update, July 29, 2015


Treeherder: We’ve added to mozlog the ability to create error summaries which will be used as the basis for automatic starring.  The Treeherder team is working on implementing database changes which will make it easier to add support for that.  On the front end, there’s now a “What’s Deployed” link in the footer of the help page, to make it easier to see what commits have been applied to staging and production.  Job details are now shown in the Logviewer, and a mockup has been created of additional Logviewer enhancements; see bug 1183872.

MozReview and Autoland: Work continues to allow autoland to work on inbound; MozReview has been changed to carry forward r+ on revised commits.

Bugzilla: The ability to search attachments by content has been turned off; BMO documentation has been started at

Perfherder/Performance Testing: We’re working towards landing Talos in-tree.  A new Talos test measuring tab-switching performance has been created (TPS, or Talos Page Switch); e10s Talos has been enabled on all platforms for PGO builds on mozilla-central.  Some usability improvements have been made to Perfherder –

TaskCluster: Successful OSX cross-compilation has been achieved; working on the ability to trigger these on Try and sorting out details related to packaging and symbols.  Work on porting Linux tests to TaskCluster is blocked due to problems with the builds.

Marionette: The Marionette-WebDriver proxy now works on Windows.  Documentation on using this has been added at

Developer Workflow: A kill_and_get_minidump method has been added to mozcrash, which allows us to get stack traces out of Windows mochitests in more situations, particularly plugin hangs.  Linux xpcshell debug tests have been split into two chunks in buildbot in order to reduce E2E times, and chunks of mochitest-browser-chrome and mochitest-devtools-chrome have been re-normalized by runtime across all platforms.  Now that mozharness lives in the tree, we’re planning on removing the “in-tree configs”, and consolidating them with the previously out-of-tree mozharness configs (bug 1181261).

Tools: We’re testing an auto-backfill tool which will automatically retrigger coalesced jobs in Treeherder that precede a failing job.  The goal is to reduce the turnaround time required for this currently manual process, which should in turn reduce tree closure times related to test failures

The Details

Treeherder/Automatic Starring

Treeherder/Front End

Perfherder/Performance Testing


TaskCluster Support

Mobile Automation

Dev Workflow

Media Automation

General Automation



July 30, 2015 12:31 AM

July 29, 2015

Joel Maher

Lost in data – episode 2 – bisecting and comparing

This week on Lost in Data, we tackle yet another pile of alerts.  This time we have a set of changes which landed together and we push to try for bisection.  In addition we have an e10s only failure which happened when we broke talos uploading to perfherder.  See how I get one step closer to figuring out the root cause of the regressions.

July 29, 2015 11:53 PM

July 28, 2015

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

July 28, 2015 05:21 AM

July 27, 2015

Armen Zambrano G. (@armenzg)

Enabling automated back-filling on mozilla-inbound for few hours

tl;dr; we're going to enable automated back-filling tomorrow Tuesday for
few hours on mozilla-inbound.

We were aiming for Monday but pushed it to Tuesday to help publicize this more.

If on Wednesday there are no fall-outs we will leave it running for m-i for a week before enabling it on other places.

Posted on various mailing lists including

> Hello all,
> We are planning to turn on a service that automatically backfills
> failed test jobs on m-i. If there are no concerns, we would like to
> turn this on experimentally for a couple of hours on [Tuesday]. We
> hope this will make it easier to identify which revision broke a
> test. Suggestions are welcome.
> The backfilling works like this: - It triggers the job that failed
> one extra time - Then it looks for a successful run of the job on the
> previous 5 revisions. If a good run is found, it triggers the job
> only on revisions up to this good run. If not, it triggers the job on
> every one of the previous 5 revisions. Previous jobs will be
> triggered one time.
> The tracking bug is:
> Best, Alice

Zambrano Gasparnian, Armen
Automation & Tools Engineer

Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

July 27, 2015 06:56 PM

July 26, 2015

Alice Scarpa


Thanks to Mozilla and Outreachy, I was able to spend one week visiting Mozilla’s Toronto office and it was awesome. My mentor, armenzg, really went out of his way to make me feel welcomed. It was a great way to feel like I’m part of the Mozilla community.

armenzg introduced me to Toronto mozillians so I could learn more about what is going on in the organization: ekyle gave me an ActiveData tour (so much data to play with!), mconley explained the e10s project to me (this time it is going to happen!), secretrobotron talked with me about community building (I want to get more involved in the Brazilian Mozilla community).

Thursday I went with the other interns and their mentors to a real-life Escape-the-room game. It was pretty fun and a great chance to meet people from a lot of different teams.

The best part of the whole week was Friday. armenzg invited me, wlach, ekyle and rwood to a work-from-his-house day and it was very pleasant. It was a great ending for an amazing week!

July 26, 2015 12:00 AM

July 23, 2015

Joel Maher

lost in data – episode 1, tackling a bunch of alerts

Today I recorded a session of me investigating talos alerts, It is ~35 minutes, sort of long, but doable in the background whilst grabbing breakfast or lunch!

I am looking forward to more sessions and answering questions that come up.

July 23, 2015 11:22 PM

David Burns

Microsoft ship a WebDriver implementation

Microsoft, the people still claim to be evil (who are actually big proponents of the the The Open Web), have... (wait for it...) SHIPPED. AN. IMPLEMENTATION. OF. WEBDRIVER!

At GTAC in California in 2011, Simon Stewart and I discussed that the Selenium project was at a crossroads (I am pretty sure beer was involved). We could ,and should, move this to the browser vendors. We had seen how in April the Chrome team had shipped their implementation and the Selenium project then deleted its implementation. I am pretty sure Daniel (Wagner-Hall) was glad to see it go.

In that time to now we have seen Mozilla get Marionette into Firefox and into release branches since Firefox 24 while slowly working on it (as well as Firefox OS support). We have seen Blackberry ship a version for the browser on their devices. We have seen mobile implementation with iOS-Driver, Selendroid and Appium.

The Spec is on track to be put forward for Recommendation by the end of the year. All the dreams that we (the Selenium Development team (my BFFs)) had are slowly coming true. This ship might be slow moving but it's mostly because some companies haven't always seen the value.


P.s. There is an open bug on the WebKit tracker for Safari support (and it is getting some internal push so I am hopeful!)

July 23, 2015 08:19 PM

Andreas Tolfsen

Optimised Rust code in Gecko

Rust Rust wave by Glen Scott (CC BY-NC 2.0)

Optimised Rust code in Gecko

Following bug 1177608, Rust code in Gecko is now compiled with optimisation by default.

Compilation of Rust components can be enabled by adding ac_add_options --enable-rust to your mozconfig file. For now there’s only limited usage of Rust in Gecko, but you can take a look at the bindings for the MP4 encoder if you’re interested. This is much thanks to the work of Ralph Giles.

Since optimised compiles are the default, you will now also get optimised output from rustc. You can disable this by setting ac_add_options --disable-optimize. This disables compilation for all cc, c++, and rustc compilers.

Next up is adding debug symbols, assertions, and adding some Rust autoconf macros.

July 23, 2015 06:10 PM

Alice Scarpa

Ijson is magic

I love Python, but it can be a little resource-hungry at times.

For example, allthethings.json is a 12MB json file. How much memory does a Python script take to load it? Let’s profile the memory usage of a script that just loads it.

import json

def load_json(filename):
    with open(filename, 'r') as f:
        return json.load(f)

if __name__ == "__main__":

Running memory_profiler on the script above:

$ python -m memory_profiler

Line #    Mem usage    Increment   Line Contents
     3   13.430 MiB    0.000 MiB   @profile
     4                             def load_json(filename):
     5   13.434 MiB    0.004 MiB       with open(filename, 'r') as f:
     6   86.773 MiB   73.340 MiB           return json.load(f)

73MB, more then 6 times the size of the original file.

This is not a big problem when loading a 12MB json on my desktop. But we needed to load a 200MB file on a Heroku standard-2x dyno, so we ended up having problems. What can help us in this case is that we only needed a subset of the data.

To use our allthethings.json example, imagine that the data we need is just the ‘shortname’ key for builders. Searching for a way to get just the data we needed without having to load the whole json in memory, I learned about ijson.

According to it’s documentation ijson is an:

Iterative JSON parser with a standard Python iterator interface

That means that instead of loading the whole file into memory and parsing everything at once, it uses iterators to lazily load the data. In that way, when we pass by a key that we don’t need, we can just ignore it and the generated object can be removed from memory.

import ijson

def load_json(filename):
    with open(filename, 'r') as fd:
        parser = ijson.parse(fd)
        ret = {'builders': {}}
        for prefix, event, value in parser:
            if (prefix, event) == ('builders', 'map_key'):
                buildername = value
                ret['builders'][buildername] = {}
            elif prefix.endswith('.shortname'):
                ret['builders'][buildername]['shortname'] = value

        return ret

if __name__ == "__main__":

Profiling its memory usage:

(venv)$ python -m memory_profiler

Line #    Mem usage    Increment   Line Contents
     5   15.109 MiB    0.000 MiB   @profile
     6                             def load_json(filename):
     7   15.109 MiB    0.000 MiB       with open(filename, 'r') as fd:
     8   15.109 MiB    0.000 MiB           parser = ijson.parse(fd)
     9   15.113 MiB    0.004 MiB           ret = {'builders': {}}
    10   26.012 MiB   10.898 MiB           for prefix, event, value in parser:
    11   26.012 MiB    0.000 MiB               if (prefix, event) == ('builders', 'map_key'):
    12   25.824 MiB   -0.188 MiB                   buildername = value
    13   25.824 MiB    0.000 MiB                   ret['builders'][buildername] = {}
    14   26.012 MiB    0.188 MiB               elif prefix.endswith('.shortname'):
    15   25.824 MiB   -0.188 MiB                   ret['builders'][buildername]['shortname'] = value
    17   26.012 MiB    0.188 MiB           return ret

Now loading the file takes 11MB and the script total memory usage went from 86MB to 26MB. On our real production example, total memory usage went from 4.5GB to 300MB.


Ijson gave us a magic level of memory-saving, but there was a speed trade-off. Using the yalj2 backend helped to reduce our loss in speed. To use the yajl backend all we had to do was:

import ijson.backends.yajl2 as ijson


$ time python

real    0m2.966s
user    0m2.944s
sys    0m0.020s


$ time python

real    0m1.626s
user    0m1.613s
sys    0m0.013s

The downside of using the yajl2 backend is that it requires users to have a C library (libyajl2) installed on their systems. Luckily, it is possible to install it on Heroku.

July 23, 2015 12:00 AM

July 22, 2015

Armen Zambrano G. (@armenzg)

Few mozci releases + reduced memory usage

While I was away adusca released few releases of mozci.

From the latest release I want to highlight that we're replacing the standard json library for ijson since it solves some memory leak issues we were facing for pulse_actions (bug 1186232).

This was important to fix since our Heroku instance for pulse_actions has an upper limit of 1GB of RAM.

Here are the release notes and the highlights of them:

Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

July 22, 2015 02:21 PM

July 21, 2015

Byron Jones

happy bmo push day!

the following changes have been pushed to

to improve security now serves attachments from a different domain – – instead of from a subdomain of  all existing links should continue to work, and will redirect to a url.

discuss these changes on

Filed under: bmo, mozilla

July 21, 2015 08:13 AM

July 17, 2015

mozregression updates

Release 0.38 and 0.39

The 0.38 release fixes a minor regression introduced with the 0.37 release (bug 1184915).

Just after that mozlog 3.0 was released, breaking compatibility with older mozbase packages, so we decided to ensure fully working dependencies and released mozregression 0.39! :)

July 17, 2015 12:00 AM

July 16, 2015

Jonathan Griffin

Automation and Tools Team Update, July 16 2015

The Automation and Tools Team (the A-Team, for short) is a large team that oversees a diverse set of services, tools and test harnesses used by nearly everyone at Mozilla.  We’re borrowing a page from Release Engineering and publishing a series of updates to inform people about what we’re up to, in the hopes of fostering better visibility and inter-team coordination.


Treeherder and Automatic Starring: Our focus for Treeherder in Q3 will be improving the signal-to-noise ratio for dealing with intermittent oranges. An overall design has been agreed to for the “automatic starring” project, and work has begun; final rollout is likely in Q4. This quarter, we’ll also stop spamming Bugzilla with comments for each intermittent, but we will put in place an alternate notification system for people who rely on Bugzilla orange comments to determine when an intermittent needs attention. We’ve also agreed on a redesign for the Logviewer that should result in a more useful and intuitive interface.
MozReview and Autoland:  MozReview now offers to publish review requests when you push, so it isn’t necessary to visit the MozReview’s UI. Work has started on adding support for autoland-to-inbound, which will allow developers to push changes to inbound directly from MozReview… no more battling tree closures!
Performance: Work continues on Perfherder’s “comparison mode”, a view that compares Talos performance data between two revisions. See wlach’s blog post for more details.
TaskCluster Support: We’re helping Release Engineering migrate from Buildbot to TaskCluster; this quarter we’re standing up Linux tests in TaskCluster and getting OS X cross-compilation to work so that we can move those builds to the cloud.
BMO now has tests running in continuous integration using TaskCluster and reporting to Treeherder.
Mobile Automation: mochitest-chrome for Android is now live! Work is also underway to enable debug reftests on Android emulators, and significant reliability improvements have been landed in Autophone.
Desktop Automation: Work is in progress to get Thread Sanitizer (TSan) builds running on try and to split gTest into its own test chunk. We’re also working towards applying –run-by-dir to mochitest-plain, in order to improve isolation and enable smarter chunking in CI.
Developer Workflow: We’re adding test-selection flexibility to the reftest harness as a prelude to making ‘mach try’ work with more test types.

The Details
Treeherder/Automatic Starring
  • Work has started on backend work needed to support automatic starring, including db simplification, and db unification (so each tree doesn’t have its own database).  Bug 1179263 tracks this work.  As a side effect of this work, Treeherder code should become less complex and easier to maintain.
  • Work has started on identifying what needs to happen in order to turn off Bugzilla comments for intermittents, and to create an alternative notification mechanism instead.  Bug 1179310 tracks this work.
Treeherder/Front End
  • New shortcuts for Logviewer, Delete Classification plus improved classification save
  • Design work is in progress for collapsing chunks in Treeherder in order to reduce visual noise in bug 1163064
Perfherder/Performance Testing
  • Evaluating alerts generated from PerfHerder
  • Improvements to compare chooser and viewer inside of PerfHerder
  • Work towards building a new tab switching test (bug 1166132)
  • Automatic publishing of reviews upon pushing
  • Known bug: people using cookie auth may experience bug 1181886
  • Better error message when MozReview’s Bugzilla session has expired (bug 1178811)
  • Pruned user database to improve user searching (bug 1171274)
  • Work is progressing on autoland-to-inbound (bug 1128039)
TaskCluster Support
  • Working on OSX cross-compilation, which will allow us to move OSX builds to the cloud; this will make OSX builds much faster in CI.
Mobile Automation
  • Autophone detects USB lock-ups and gracefully restarts. This is a huge improvement in system reliability.
  • Continued work on getting Android Talos tests ported to Autophone (bug 1170685)
  • Updated manifests and mozharness configs for mochitest-chrome (bug 1026290)
  • Determined total-chunks requirements for Android 4.3 Debug reftests (bug 1140471)
  • Re-wrote robocop harness to significantly improve run-time efficiency (bug 1179981)
Dev Workflow
  • Helped RelEng resolve some problems that were preventing them from landing mozharness in the tree.  This opens the door to a lot of future dev workflow improvements, including better unification of the ways we run automated tests in continuous integration and locally.  We’ve wanted this for years and it’s great to see it finally happen.
  • Did some work on top of jgraham’s patch to make mach use mozlog structured logging
Platform QA
  • We had to respond to the breakup of into several files to keep our Jenkins instance running.
  • Getting firefox-media-tests to satisfy Tier-2 Treeherder visibility requirements involves changing how Treeherder accommodates non-buildbot jobs (e.g bug 1182299)
General Automation
  • Working on running multiple tests/manifests through reftests harness as a prelude for supporting |mach try| for more test types.
  • Created patch to move mozlog.structured to the top level package (and what was previously there to mozlog.unstructured)
  • Figured out the series of steps needed to produce a usable Thread Sanitizer enabled linux build on our infra
  • Separating out gTest into a separate job in CI – bug 1179955.
  • More memory optimizations (motivation: releng query for Chris Atlee:  query slow tests)
    • run staging environment as stability test for production
    • change etl procedure so pushing changes to prod are easier (moving toward standard procedure)
  • import treeherder data markup to active data (motivation: characterizing test failures
    • ateam query: summary of test failures, stars and resolutions (bug 1161268bug 1172048)
    • subtests are too large for download of more than one day – working on code to only pull what’s required


July 16, 2015 05:11 PM

July 14, 2015

William Lachance

Perfherder update

Haven’t been doing enough blogging about Perfherder (our project to make Talos and other per-checkin performance data more useful) recently. Let’s fix that. We’ve been making some good progress, helped in part by a group of new contributors that joined us through an experimental “summer of contribution” program.

Comparison mode

Inspired by Compare Talos, we’ve designed something similar which hooks into the perfherder backend. This has already gotten some interest: see this post on dev.tree-management and this one on dev.platform. We’re working towards building something that will be really useful both for (1) illustrating that the performance regressions we detect are real and (2) helping developers figure out the impact of their changes before they land them.

Screen Shot 2015-07-14 at 3.54.57 PM Screen Shot 2015-07-14 at 3.53.20 PM

Most of the initial work was done by Joel Maher with lots of review for aesthetics and correctness by me. Avi Halmachi from the Performance Team also helped out with the t-test model for detecting the confidence that we have that a difference in performance was real. Lately myself and Mike Ling (one of our summer of contribution members) have been working on further improving the interface for usability — I’m hopeful that we’ll soon have something implemented that’s broadly usable and comprehensible to the Mozilla Firefox and Platform developer community.

Graphs improvements

Although it’s received slightly less attention lately than the comparison view above, we’ve been making steady progress on the graphs view of performance series. Aside from demonstrations and presentations, the primary use case for this is being able to detect visually sustained changes in the result distribution for talos tests, which is often necessary to be able to confirm regressions. Notable recent changes include a much easier way of selecting tests to add to the graph from Mike Ling and more readable/parseable urls from Akhilesh Pillai (another summer of contribution participant).

Screen Shot 2015-07-14 at 4.09.45 PM

Performance alerts

I’ve also been steadily working on making Perfherder generate alerts when there is a significant discontinuity in the performance numbers, similar to what GraphServer does now. Currently we have an option to generate a static CSV file of these alerts, but the eventual plan is to insert these things into a peristent database. After that’s done, we can actually work on creating a UI inside Perfherder to replace alertmanager (which currently uses GraphServer data) and start using this thing to sheriff performance regressions — putting the herder into perfherder.

As part of this, I’ve converted the graphserver alert generation code into a standalone python library, which has already proven useful as a component in the Raptor project for FirefoxOS. Yay modularity and reusability.

Python API

I’ve also been working on creating and improving a python API to access Treeherder data, which includes Perfherder. This lets you do interesting things, like dynamically run various types of statistical analysis on the data stored in the production instance of Perfherder (no need to ask me for a database dump or other credentials). I’ve been using this to perform validation of the data we’re storing and debug various tricky problems. For example, I found out last week that we were storing up to duplicate 200 entries in each performance series due to double data ingestion — oops.

You can also use this API to dynamically create interesting graphs and visualizations using ipython notebook, here’s a simple example of me plotting the last 7 days of pageload data inline in a notebook:

Screen Shot 2015-07-14 at 4.43.55 PM


July 14, 2015 08:51 PM

Julien Pagès

mozregression 0.37 release

Yesterday we just released mozregression (command line regression range finder for Mozilla nightly and inbound builds) 0.37!

This release include some new features and fixes, and uses the new TaskCluster system from Mozilla Release Engineering to retrieve integration branch information. This considerably speeds up the bisection of inbound builds.

Try it out!

July 14, 2015 04:29 PM

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

July 14, 2015 07:17 AM

July 13, 2015

Julien Pagès

Python [aggressive] pep8 conversion using autopep8

Some time ago, I started to look at Talos, the python performance testing framework for firefox that is usable on Windows, Mac and Linux.

Talos has been around for a long time, and has seen many contributors! Unfortunately the codebase reflects that, and as an example there was simply no common code style around for all the Python files.

Sometimes a 2 spaces based indentation, sometimes 4 spaces based, sometimes something else. No limit for the lines length. This end up with something that is really hard to read, understand and maintain. So one of my first goal was to try to clean this up.

And I heard about autopep8. As the name suggests, this is a tool (command line) that will automatically do some pep8 conversion on Python source files.

Awesome! I highly recommend that to you if you are trying to adapt the coding style of some Python code to pep8 convention, this worked really well for me. This is a pain to do that by hand, and almost impracticable when the indentation needs to be fixed, but autopep8 makes that for you in a safe way.

Tip: the –aggressive flag is really nice for fixing long lines.

So, as an example this is how I used autopep8 on talos:

autopep8 --recursive --in-place --aggressive --aggressive /path/to/talos

Simple and effective. See the usage documentation to see with an example what it really does and other command line flags.

July 13, 2015 09:21 AM

mozregression updates

Release 0.37

I’m excited to announce that mozregression now supports using taskcluster for inbound bisection with this release!

This considerably speeds up the process of downloading builds from integration branches (like mozilla-inbound) and also supports work by Mozilla Release Engineering to migrate build storage from ftp to s3.


New features:

July 13, 2015 12:00 AM

July 12, 2015

Julien Pagès

RunSnakeRun – graphical visualisation of dumped python profiling data

If you are a Python developer like me, you probably know the profile and cProfile modules that provides deterministic profiling of Python programs.

These modules are awesome – however, when it comes to analysing the data to improve your program, the provided pstats module is generally not powerful enough if you have quite a large codebase.

And here graphical tools comes in handy! I tried RunSnakeRun, and this is a really great program that allows you to analyse the profiling data under multiple angles (a nice view is by file), so you can find easily the bottlenecks and fix them.

RunSnakeRun helped me to improve the “mach help” command. It is a cross platform tool. Note that if you are on GNU/Linux and that you have the KDE desktop (or don’t mind to install the required KDE dependencies), KCacheGrind can be used with Python also.

July 12, 2015 04:12 PM

July 07, 2015

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

July 07, 2015 06:11 AM

July 06, 2015

Armen Zambrano G. (@armenzg)

mozci 0.8.2 - Allow using TreeHerder as a query source

In this release we have added an experimental feature where you can use Treeherder as your source for jobs' information instead of using BuildApi/Buildjson.
My apologies as this should have been a minor release (0.9.0) instead of a security release (0.8.2).


Thanks to @adusca @vaibhavmagarwal and @chmanchester for their contributions.
Our latest new contributor is @priyanklodha - thank you!

How to update

Run "pip install -U mozci" to update

Major highlights

  • Added --query-source option to get data from Treeherder or Buildapi
  • Improved usage of OOP to allow for different data sources seamlessly

Minor improvements

  • Better documentation of --times
  • Cleaning up old builds-*.js files
  • Enforced line character limit

All changes

You can see all changes in here:

Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

July 06, 2015 02:58 PM

July 02, 2015

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

July 02, 2015 05:59 AM

July 01, 2015

David Burns

Who wants to be an alpha tester for Marionette?

Are you an early adopter type? Are you an avid user of WebDriver and want to use the latest and great technology? Then you are most definitely in luck.

Marionette, the Mozilla implementation of the FirefoxDriver, is ready for a very limited outing. There is a lot of things that have not been implemented or, since we are implementing things agains the WebDriver Specification they might not have enough prose to implement (This has been a great way to iron out spec bugs).

Getting Started

At the moment, since things are still being developed and we are trying to do things with new technologies (like writing part this project using Rust) we are starting out with supporting Linux and OS X first. Windows support will be coming in the future!

Getting the driver

We have binaries that you can download. For Linux and for OS X . The only bindings currently updated to work are the python bindings that are available in a branch on my fork of the Selenium project. Do the following to get it into a virtualenv:
  1. Create a virtualenv
  2. activate your virtualenv
  3. cd to where you have cloned my repository
  4. In a terminal type the following: ./go py_install

Running tests

Running tests against marionette requires that you do the following changes (which hopefully remains small)

Update the desired capability to have marionette:true and add binary:/path/to/Firefox/DeveloperEdition/or/Nightly. We are only supporting those two versions of at the moment because we have had a couple in compatible issues that we have fixed which means speaking to Marionette in the browser in the beta or release versions quite difficult.

Since you are awesome early adopters it would be great if we could raise bugs. I am not expecting everything to work but below is a quick list that I know doesn't work.

Thanks for being an early adopter and thanks for raising bugs as you find them!

July 01, 2015 07:57 PM

June 30, 2015

Geoff Brown

Firefox for Android Performance Measures – Q2 Check-up

This review of Android performance measurements covers the period April 1 – June 30: the second quarter of 2015. I will write these summary posts on a quarterly basis from now on.


– Most tests fairly steady over the quarter.


This section tracks Perfomatic graphs from for mozilla-central builds of Firefox for Android, for Talos tests run on Android 4.0 Opt. The test names shown are those used on treeherder. See for background on Talos.


Measure of “checkerboarding” during simulation of real user interaction with page. Lower values are better.


19 (start of period) – 20 (end of period)

Small regression on June 10 (no bug?). This test is exhibiting some inconsistent behavior, as noted in bug 1149567.


An svg-only number that measures SVG rendering performance. About half of the tests are animations or iterations of rendering. This ASAP test (tsvgx) iterates in unlimited frame-rate mode thus reflecting the maximum rendering throughput of each test. The reported value is the page load time, or, for animations/iterations – overall duration the sequence/animation took to complete. Lower values are better.


720 (start of period) – 680 (end of period).

Small improvement on May 27.


Generic page load test. Lower values are better.


680 (start of period) – 670 (end of period).

Small improvement on May 18.

Throbber Start / Throbber Stop

These graphs are taken from  Browser startup performance is measured on real phones (a variety of popular devices).

Unfortunately, I could not find any devices for mozilla-central which reported consistently throughout this period.




These graphs are taken from Eideticker is a performance harness that measures user perceived performance of web browsers by video capturing them in action and subsequently running image analysis on the raw result.

More info at:




Most other tests have incomplete data for this time period.


These graphs are taken from the mozbench dashboard at which includes some comparisons involving Firefox for Android. More info at





June 30, 2015 10:41 PM

June 29, 2015

Alice Scarpa

Changing goals

I started my Outreachy internship with Mozilla on May 25th, working with mozci. A lot happened in the first five weeks:

Part of the application process was writing a project proposal. Mozci was about 2 months old at the time and so much changed since then that strictly following the original proposal would have been a bad idea.

Our current plan for the rest of my internship is to focus on using Pulse Actions to add new functionality to Treeherder, but a lot can happen until then (for example, we depend a lot on feedback from the sheriffs to guide us). I’m not nervous because I know I can count with the help of people from Outreachy, the Recurse Center and the A-team, especially my mentor Armen who has been awesome since I started contributing to mozci a long long time ago (four whole months!).

June 29, 2015 12:00 AM

June 16, 2015

Armen Zambrano G. (@armenzg)

mozci 0.8.0 - New feature -- Trigger coalesced jobs + improved performance

Beware! This release is full awesome!! However, at the same time new bugs might pop up so please let us know :)
You can trigger now all coalesced jobs on a revision:
mozci-trigger --coalesced --repo-name REPO -r REVISION


Thanks to @adusca @glandium @vaibhavmagarwal @chmanchester for their contributions on this release.

How to update

Run "pip install -U mozci" to update

Major highlights

  • #259 - New feature - Allow triggering
  • #227 - Cache files as gzip files instead of uncompressed
    • Less disk print
  • #227 - Allow using multiple builders
  • 1e591bf - Make sure that we do not fetch files if they are not newer
    • We were failing to double check that the last modification date of a file was the same as the one in the server
    • Hence, we were downloading files more often than needed
  • Caching builds-4hr on memory for improved performance

Minor improvements

  • f72135d - Backfilling did not accept dry_run or to trigger more than once
  • More tests and documents
  • Support for split tests (test packages json file)
  • Some OOP refactoring

All changes

You can see all changes in here:

Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

June 16, 2015 05:38 PM

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

June 16, 2015 06:42 AM

June 13, 2015

Syd Polk

WWDC Day 4 - ...and that's a wrap

And the last day...

I had lunch with Emily Toop on the Firefox iOS team. She is new to Mozilla, but an experienced iOS developer out of the UK. Lovely venue at the Samovar Tea Lounge in Yerba Buena Gardens.

Overall impressions of WWDC:
I would be happy to come back next year, but I am not going to push again unless programming the Mac or iOS is my primary job. Lots of good stuff. Also, there are some sessions from last year I need to watch.

June 13, 2015 12:35 AM

June 12, 2015

Syd Polk

WWDC Day 3 - Sessions are starting to run together...

We are in the thick of things now; next to last day. Sessions I attended:

I had lunch with my good buddy Jim Ingham. We worked together at Sun, Cygnus, Red Hat and Apple, off and on from 1997-2006. It was good to see him. I wish I could remember the last name of the other Apple engineer who joined us, Greg. He had been there a few months when I left, and he had moved from Austin (!). It was good to eat a good lunch away from the show.

I also went to the Apple Bash.  They had the band Walk the Moon, and I was pleasantly surprised. There was actually musicianship on display. I am not going to buy anything, but for once, I wasn't counting time until the band stopped.

It was not a dinner outing. There were small entree-like snacks, but it's not the same. So Paul and I found a late-night place near Union Square.

June 12, 2015 04:56 PM

June 11, 2015

Syd Polk

WWDC Day 2 - more sessions and a party

Lots of sessions today. Let's tear right in:

After the main conference, I went to see James Dempsey and the Breakpoints. Uneven songs musically, but the words are all about programming in Cocoa, and it's pretty funny. Check it out on iTunes if you are into that sort of thing.

And I closed the night having an excellent cocktail with my best friend, Paul Tien-Shih Lee, whose birthday it was. The View bar in the Marriot is wonderful.

June 11, 2015 03:56 PM

June 10, 2015

Syd Polk

WWDC Day 1 - Sessions, Sessions, and some old friends

Tuesday was filled with sessions, with breaks in the Developer Tools lab. Sessions I attended:

I also spent some time in the Developer Tools Lab catching up with former coworkers. Plus, I discovered an Xcode 7 bug launching the simulator, and I talked to them about that.

After the show, I met some friends from college and/or my very first projects for dinner in San Mateo. I was truly great to see them.

June 10, 2015 04:21 PM

Alice Scarpa

Tracking down a non-deterministic coverage failure

At Mozilla CI Tools, we use Coveralls to track our test coverage progress. In the past month we noticed that sometimes the coverage changes even in PRs that only changed files that are not tracked by coveralls. How come?

Comparing coverage reports for builds in which this problem happens, I noticed that this line would sometimes be marked as covered, sometimes as not covered. The line belongs to a function that builds a graph from a list. An interesting thing about this function is that the order of the list does not affect the final graph, but it does affect the code path taken inside the function. In the test for this function, the input comes from a call to a dict.keys() method. Also, an interesting thing about the .keys() method is that it always produces a list containing the same elements, but the order varies. Mystery solved!

So a single sort() command on the test input could fix those annoying random coverage failures!

June 10, 2015 12:00 AM

June 09, 2015

Joel Maher

Please welcome the Dashboard Hacker team

A few weeks ago we announced that we would be looking for committed contributors for 8+ weeks on Perfherder.  We have found a few great individuals, all of whom show a lot of potential and make a great team.

They are becoming familiar with the code base and are already making a dent in the initial list of work set aside.  Let me introduce them (alphabetical order by nicks):

akhileshpillai – Akhilesh has jumped right in and started fixing bugs in Perfherder.  He is new to Mozilla and will fit right in.  With how fast he has come up to speed, we are looking forward to what he will be delivering in the coming weeks.  We have a lot of UI workflow as well as backend data refactoring work on our list, all of which he will be a great contributor towards.

mikeling – mikeling has been around Mozilla for roughly two years, recently he started out helping with a few projects on the A*Team.  He is very detailed oriented, easy to work with and is willing to tackle big things.

theterabyte – Tushar is excited about this program as an opportunity to grow his skills as a python developer and experiencing how software is built outside of a classroom.  Tushar will get a chance to grow his UI skills on Perfherder by making the graphs and compare view more polished and complete, while helping out with an interface for the alerts.

Perfherder will soon become the primary dashboard for all our performance needs.

I am looking forward to the ideas and solutions these new team members bring to the table.  Please join me in welcoming them!

June 09, 2015 04:55 PM

Please join me in welcoming the DX Team

A few weeks ago, I posted a call out for people to reach out and commit to participate for 8+ weeks.  There were two projects and one of them was Developer Experience.  Since then we have had some great interest, there are 5 awesome contributors participating (sorted by irc nicks).

BYK – I met BYK 3+ years ago on IRC- he is a great person and very ambitious.  As a more senior developer he will be focused primarily on improving interactions with mach.  While there are a lot of little things to make mach better, BYK proposed a system to collect information about how mach is used.

gma_fav – I met gma_fav on IRC when she heard about the program.  She has a lot of energy, seems very detail oriented, asks good questions, and brings fresh ideas to the team!  She is a graduate of the Ascend project and is looking to continue her growth in development and open source.  Her primary focus will be on the interface to try server (think the try chooser page, extension, and taking other experiments further).

kaustabh93 – I met Kaustabh on IRC about a year ago and since then he has been a consistent friend and hacker.  He attends university.  In fact I do owe him credit for large portions of alert manager.  While working on this team, he will be focused on making run-by-dir a reality.  There are two parts: getting the tests to run green, and reducing the overhead of the harness.

sehgalvibhor – I met Vibhor on IRC about 2 weeks ago.  He was excited about the possibility of working on this project and jumped right in.  Like Kaustabh, he is a student who is just finishing up exams this week.  His primary focus this summer will be working in a similar role to Stanley in making our test harnesses act the same and more useful.

stanley – When this program was announced Stanley was the first person to ping me on IRC.  I have found him to be very organized, a pleasure to chat with and he understands code quite well.  Coding and open source are both new things to Stanley and we have the opportunity to give him a great view of it.  Stanley will be focusing on making the commands we have for running tests via mach easier to use and more unified between harnesses.

Personally I am looking forward to seeing the ambition folks have translate into great solutions, learning more about each person, and sharing with Mozilla as a whole the great work they are doing.

Take a few moments to say hi to them online.

June 09, 2015 04:32 PM

Syd Polk

WWDC Day 0, Part 2 - Platforms State of the Union

The Platforms State of the Union session was much more interesting than the keynote. Highlights applicable to what I work on:
You should do those things anyway.
Great session.

June 09, 2015 04:14 PM

David Burns

WebDriver Specification - Have you read it lately?

A lot of work has gone into the WebDriver Specification this year. The methods in there have had a major make over to make them more specific in the steps that are required as well as having the relevant links. Go have a read of it and feel free to raise bugs against it, we will be updating it quite regularly. You can see all the work that is happening on Github. We do everything via pull requests so you can read things before they land.

My team have also been working hard at making sure that our implementation is following the specification and are making some great leaps with it. I will be releasing a development version of the python bindings soon that use the httpd, like InternetExplorerDriver and ChromeDriver, to drive the browser. Currently our httpd only works against Nightly but there is a merge to Aurora happening soon when we will be sending out links for you to start playing with it all. I am actually looking forward to some of the feedback that we get about it.

June 09, 2015 02:42 PM

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

June 09, 2015 06:37 AM

June 08, 2015

Syd Polk

WWDC Day 0, Part 1 - Keynote

First of all, there are a lot of people here. Last night, my buddy told me that I might want to line up early; otherwise, I might have ended up in the overflow room.

And my Mozilla colleagues, Emily Toop and Darrin Henin, wanted to hook up early. So I set out to Moscone West and arrived at 6:30. I initially went to the back of the line. It went north on 4th, all of the way west on Minna, back South on 5th, and then almost completing the circle halfway back up Howard. 

Fortunately, the other Mozillians found some of their friends who work at Facebook, and got us a place at the front of the line. Hooray.

They let us into the building at 8. We went up the escalators, and then walked all of the way around the second floor, where we waited in the (hot, crowded) hallway for another hour.

At nine, they let us crowd around the base of the escalator to the 3rd floor. We were there for about half an hour when we were finally let into the main hall.

The keynote itself was a little disappointing. New versions of OS X and iOS with some whizzy features. My favorite was the multiapp features in iPad; looking forward to playing with those. My single favorite announcement was that Swift is now going to be Open Source and run on Linux. They then talked about watchOS, which I currently have no interest in.

And then they wasted 45 minutes with the Apple Music announcement. OK, I can stream all of Apple's catalog. But I have to either stream or play my library? Wasn't clear. And the performance did nothing for me.

So. OS X El Capitain and iOS 9: Yay. Looks good. Carry on. 
watchOS: Cool, I guess. 
MUSIC: meh.

June 08, 2015 08:45 PM

Dave Hunt

Joining Web QA

Dylan with origami foxI’m excited to announce that as of last week I am officially on Mozilla’s Web QA team! Despite working closely with the team since I started at Mozilla over four years ago, I’ve always reported to another team. Originally I had a hybrid role, where I reported to the Director of QA and assisted with automation of both Web and Desktop products. Later, we formed the QA Automation Services team, which existed to provide automation services to any QA team. This was mostly absorbed into the A-Team, which shares a lot of the same goals but is not just focused on QA. During my time with the A-Team a lot of my work started to shift towards Firefox OS, so it made sense during a organisational shift towards vertical stacks for me to officially join the Firefox OS automation team.

Many times since I started at Mozilla I’ve felt that I had more to offer the Web QA team, and I’ve always been a keen contributor to the team. I can’t say it was an easy decision to move away from Firefox OS, as it’s a terrifically exciting project, but the thought of joining Web QA just had me bursting with enthusiasm! In a time where there’s been a number of departures, I’m pleased to say that I feel like I’m coming home. Look out web – you’re about to get even more automated!

Last week Stephen Donner interviewed me on being a Mozilla Web QA contributor. You can read the blog post over on the Web QA blog.

I’d like to take this opportunity to thank Stephen Donner, Matt Evans, Clint Talbert, Jonathan Griffin, and James Lal for their support and leadership. I’ve made so many great friendships at Mozilla, and with our Whistler work week just around the corner I’m so looking forward to catching up with many of them in person!

June 08, 2015 09:42 AM

Syd Polk

WWDC - D -1

When WWDC signups went up, I decided to take a chance and see if I could score a ticket. Much to my surprise, I did get one. So, here I am, in San Francisco, the night before it starts, and I am about to go for the first time since 2005.

In that WWDC, I was an Apple employee. I worked on the Developer Tools team. We were shuffled into the Apple Employee room for the Keynote.  At the end of the keynote, Steve Jobs announced that Apple was moving the Mac product line to Intel. I was on the team that was "in the know" about the transition. SJ asked us all to stand up, and the rest of the employees gave us a standing O. Was one of the proudest moments of my career.

Since then, I have moved to Austin and have worked several jobs. Currently, I work in automation for Mozilla. I hope to do some development work someday, and Mozilla has an effort underway for iOS, so I am going to be attending sessions and talking with people here about that effort. I am also going to sessions for my personal projects.

I checked in today, and they did something really smart. They gave us a windbreaker. Anybody who has been to San Francisco in June knows that this is really smart. There are probably many attendees who have never been here and don't know that June and July are cold, foggy and windy here.

This is not the first year they have done this (I saw somebody with a "14" on his back), but it is really brilliant.

Tomorrow is dominated by the Keynote, and in the afternoon, the only things to see are Platforms State of the Union and the Apple Design awards.

I'm stoked!

June 08, 2015 01:44 AM

June 07, 2015

mozregression updates

Release 0.37

I’m excited to announce that mozregression now supports using taskcluster for inbound bisection with this release!

This considerably speeds up the process of downloading builds from integration branches (like mozilla-inbound) and also supports work by Mozilla Release Engineering to migrate build storage from ftp to s3.


New features:

June 07, 2015 12:00 AM

June 06, 2015

Julien Pagès

Python code cleanup – vulture

I’m pretty sure you all already know about flake8, a great tool that combine other tools (PyFlakes, pep8 and Ned Batchelder’s McCabe script) to check style issues and code errors.

Working on Talos, I needed to find a way to find dead code on the codebase (the less code, the better!). I found a pretty good tool that helped me: vulture. This is a one shot tool – it can not be automated like flake8 as there is some false positives, but still running it once on an old codebase may be a big win to find and remove useless stuff.

Obviously it is easier to use on python programs – you will have to whitelist a lot of things if you run this on a python library that offers a public API.

If you want to seek and destoy dead code on your python program, give it a try!

June 06, 2015 09:46 AM

June 05, 2015

Alice Scarpa


Recently we had an ‘UnboundLocalError’ in mozci. A local variable was being referenced before assignment. Turns out we were only defining the variable in one branch of an ‘if’ statement. It was very straightforward to fix. What bothered me was that this is the exact sort of error that would have been caught at compile time in other languages.

Is there a Python static analysis tool that would have caught this? I wrote a very simple module with the same bug and tried with different tools1:

"""This module bakes pizzas."""

def bake_pizza():
    """Bake a pizza."""
    number = int(raw_input())

    if number > 3:
        additional = 2
        number += 1
    print number + additional

if __name__ == '__main__':

Neither Pylint, Pyflakes nor PyChecker were able to notify me about the possibility of an ‘UnboundLocalError’.

ekyle suggested that PyCharm might be able to solve this problem, and indeed, PyCharm code inspection shows the message I wanted:

>  Local variable 'additional' might be referenced before assignment

I spent a couple of hours today going through every PyCharm warning in mozci to see if there were any other bugs that it could catch. It didn’t find other serious issues (I don’t know if that’s fortunate or not).

  1. It ended up being the only module I’ve ever written that scored 10.0/10 in pylint.

June 05, 2015 12:00 AM

June 03, 2015

Armen Zambrano G. (@armenzg)

mozci 0.7.2 - Support b2g jobs that still run on Buildbot

There are a lot of b2g (aka Firefox OS) jobs that still run on Buildbot .
Interestingly enough we had not tried before to trigger one with mozci.
This release adds support for it.
This should have been a minor release (0.8.0) rather than a security release (0.7.2). My apologies!
All jobs that start with "b2g_" in all_builders.txt are b2g jobs that still run on Buildbot instead of TaskCluster (docs - TC jobs on treeherder).

Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

June 03, 2015 07:23 PM

mozci 0.7.1 - regression fix - do not look for files for running jobs

This release mainly fixes a regression we introduced in the release 0.7.0.
The change (#220) we introduced checked completed and running jobs for files that have been uploaded in order to trigger tests.
The problem is that running jobs do not have any metadata until they actually complete.
We fixed this on #234.


Thanks to @adusca and @glandium for their contributions on this release.

How to update

Run "pip install -U mozci" to update

Major highlights

  • #234 - (bug fix) - Do not try to find files for running jobs
  • #228 - For try, only trigger talos jobs on existing build jobs ** rather than triggering builds for platforms that were not requested
  • #238 - Read credentials through environment variables

Minor improvements

  • #226 - (bug fix) Properly cache downloaded files
  • #228 - (refactor) Move SCHEDULING_MANAGER
  • #231 - Doc fixes

All changes

You can see all changes in here:

Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

June 03, 2015 07:14 PM

Joel Maher

re-triggering for a [root] cause- version 2.0 – a single bug!

Today the orange factor email came out- the top 10 bugs with stars on them :)  Last week we had no bugs that we could take action on, and the week before we had a few bugs to take action on.

This week I looked at each bug again, annotated them with some notes as to what I did or why I didn’t do anything, here are the bugs:

It is nice to find a bug that we can take action on.  What is interesting is the bug has been around for a while, but we noticed about May 21 that the rate of failures went up from a couple a day to >5/day.  Details:

I might try a full experiment soon blindly looking at bugs instead of using orange factor.

June 03, 2015 10:46 AM

June 02, 2015

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

June 02, 2015 08:03 AM

May 28, 2015

Joel Maher

the orange factor – no need to retrigger this week

last week I did another round of re-triggering for a root cause and found some root causes!  This week I got an email from orange factor outlining the top 10 failures on the trees (as we do every week).

Unfortunately as of this morning there is no work for me to do- maybe next week I can hunt.

Here is the breakdown of bugs:

As you can see there isn’t much to do here.  Maybe next week we will have some actions we can take.  Once I have about 10 bugs investigated I will summarize the bugs, related dates, and status, etc.

May 28, 2015 04:15 PM

Armen Zambrano G. (@armenzg)

mozci 0.7.0 - Less network fetches - great speed improvements!

This release is not large in scope but it has many performance improvements.
The main improvement is to have reduced the number of times that we fetch for information and use a cache where possible. The network cost was very high.
You can read more about in here:


Thanks to @adusca @parkouss @vaibhavmagarwal for their contributions on this release.

How to update

Run "pip install -U mozci" to update

Major highlights

  • Reduce drastically the number of requests by caching where possible
  • If a failed build has uploaded good files let's use them
  • Added support for retriggering and cancelling jobs
  • Retrigger a job once with a count of N instead of triggering individually N times

Minor improvements

  • Documenation updates
  • Add badge

All changes

You can see all changes in here:

Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

May 28, 2015 02:01 PM

Byron Jones

happy bmo push day!

it appears to be “bmo push week“.  today’s push addresses some issues caused by yesterday’s push.

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

May 28, 2015 07:06 AM

Alice Scarpa

Acting upon Pulse messages

Pulse Actions is a service that listens to retrigger/cancel messages on exchange/treeherder/v1/job-actions and acts upon them using Mozilla CI Tools.

The project was deployed to Heroku today in dry-run mode, i.e., instead of actually making retrigger/cancel requests, Pulse Actions is just logging the requests it would have made.

If this experiment works, it might be possible for Treeherder to use the service for retrigger/cancel requests. That would eliminate the need for users signed in with Persona to enter LDAP credentials to make TH requests (bug 1077053) and would also open a lot of new possibilities (e.g. adding a backfill button to TH using mozci).

This project’s progress is being tracked in bug 1168148.

May 28, 2015 12:00 AM

May 27, 2015

Armen Zambrano G. (@armenzg)

Welcome adusca!

It is my privilege to announce that adusca (blog) joined Mozilla (since Monday) as an Outreachy intern for the next 4 months.

adusca has an outstanding number of contributions over the last few months including Mozilla CI Tools (which we're working on together).

Here's a bit about herself from her blog:
Hi! I’m Alice. I studied Mathematics in college. I was doing a Master’s degree in Mathematical Economics before getting serious about programming.
She is also a graduate from Hacker's School.

Even though Alice has not been a programmer for many years, she has shown already lots of potential. For instance, she wrote a script to generate scheduling relations for buildbot; for this and many other reasons I tip my hat.

adusca will initially help me out with creating a generic pulse listener to handle job cancellations and retriggers for Treeheder. The intent is to create a way for Mozilla CI tools to manage scheduling on behalf of TH, make the way for more sophisticated Mozilla CI actions and allow other people to piggy back to this pulse service and trigger their own actions.

If you have not yet had a chance to welcome her and getting to know her, I highly encourage you to do so.

Welcome Alice!

Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

May 27, 2015 05:10 PM

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

May 27, 2015 07:11 AM

May 26, 2015

Geoff Brown

Handling intermittent test timeouts in long-running tests

Tests running on our new-ish Android 4.3 Opt emulator platform have recently been plagued by intermittent timeouts and I have been having a closer look at some of them (like bug 919246 and bug 1154505) .

A few of these tests normally run “quickly”. Think of a test that runs to completion in under 10 seconds most of the time but times out after 300+ seconds intermittently. In a case like this, it seems likely that there is an intermittent hang and the test needs debugging to determine the underlying cause.

But most of the recent Android 4.3 Opt test timeouts seem to be affecting what I classify as “long-running” tests. Think of a test that normally runs to completion in 250 to 299 seconds, but intermittently times out after 300 seconds. It seems likely that normal variations in test duration are intermittently pushing past the timeout threshold; if we can tolerate a longer time-out, or make the test run faster in general, we can probably eliminate the intermittent test failure.

We have a lot of options for dealing with long-running tests that sometimes timeout.

Option: Simplify or optimize the test

Long-running tests are usually doing a lot of work. A lot of assertions can be run in 300 seconds, even on a slow platform! Do we need to test all of those cases, or could some be eliminated? Is there some setup or tear down code being run repeatedly that could be run just once, or even just less often?

We usually don’t worry about optimizing tests but sometimes a little effort can help a test run a lot more efficiently, saving test time, money (think aws costs), and aggravation like intermittent time-outs.

Option: Split the test into 2 or more smaller tests

Some tests can be split into 2 or more smaller tests with minimal effort. Instead of testing 100 different cases in one test, we may be able to test 50 in each. There may be some loss of efficiency: Maybe some setup code will need to be run twice, and copied and pasted to the second test. But now each half runs faster, reducing the chance of a timeout. And when one test fails, the cause is – at least slightly – more isolated.

Option: Request a longer timeout for the test

Mochitests can call SimpleTest.requestLongerTimeout(2) to double the length of the timeout applied to the test. We currently have about 100 mochitests that use this feature.

For xpcshell tests, the same thing can be accomplished with a manifest annotation:

requesttimeoutfactor = 2

That’s a really simple “fix” and an effective way of declaring that a test is known to be long-running.

On the other hand, it is avoiding the problem and potentially covering up an issue that could be solved more effectively by splitting, optimizing, or simplifying. Also, long-running tests make our test job “chunking” less effective: It’s harder to split load evenly amongst jobs when some tests run 100 times longer than others.

Option: Skip the test on slow platforms

Sometimes it’s not worth the effort. Do we really need to run this test on Android as well as on all the desktop platforms? Do we get value from running this test on both Android 2.3 and Android 4.3? We may “disable our way to victory” too often, but this is a simple strategy, doesn’t affect other platforms and sometimes it feels like the right thing to do.

Option: Run on faster hardware

This usually isn’t practical, but in special circumstances it seems like the best way forward.

If you have a lot of timeouts from long-running tests on one platform and those tests don’t timeout on other platforms, it may be time to take a closer look at the platform.

Our Android arm emulator test platforms are infamous for slowness. In fairness, the emulator has a lot of work to do, Firefox is complex, our tests are often relentless (compared to human-driven browsing), and we normally run the emulator on the remarkably slow (and cheap!) m1.medium AWS instances.

If we are willing to pay for better cpu, memory, and I/O capabilities, we can easily speed up the emulator by running on a faster AWS instance type — but the cost must be justified.

I recently tried running Android 4.3 Debug mochitests on m1.medium and found that many tests timed out. Also, since all tests were taking longer, each test job (each “chunk”) needed 2 to 3 hours to complete — much longer than we can wait. Increasing chunks seemed impractical (we would need 50 or so) and we would still have all those individual timeouts to deal with. In this case, running the emulator on c3.xlarge instances for Debug mochitests made a big difference, allowing them to run in the same number of chunks as Opt on m1.medium and eliminating nearly all timeouts.

I’ve enjoyed investigating mochitest timeouts and found most of them to be easy to resolve. I’ll try to investigate more timeouts as I see them. Won’t you join me?

May 26, 2015 11:34 PM

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

May 26, 2015 04:49 AM

May 25, 2015

Alice Scarpa

Using cProfile with gprof2dot to identify bottlenecks

At Mozilla CI Tools we have a script that can sometimes be annoyingly slow. I used cProfile with gprof2dot to better understand how to improve it.

To get a pretty graph of the script’s behaviour I ran:

python -m cProfile -o timing
gprof2dot -f pstats timing -o
dot -Tsvg -o graph_master.svg

This gave me a very useful graph.

Looking at the graph I was able to identify two bottlenecks that were low-hanging fruit: query_jobs and valid_revision. These two functions are called a lot of times in the script with the same arguments. This means that by adding some simple caching I could improve the script’s speed. Preliminary results show a 2x speed-up. There is still a lot of room for improvement, but it’s a nice start.

May 25, 2015 12:00 AM

May 20, 2015

Joel Maher

re-triggering for a [root] cause – version 1.57

Last week I wrote some notes about re-triggering jobs to find a root cause.  This week I decided to look at the orange factor email of the top 10 bugs and see how I could help.  Looking at each of the 10 bugs, I had 3 worth investigating and 7 I ignored.



Looking at the bugs of interest, I jumped right in in retriggering.  This time around I did 20 retriggers for the original changeset, then went back to 30 revisions (every 5th) doing the same thing.  Effectively this was doing 20 retriggers for the 0, 5th, 10th, 15th, 20th, 25th, and 30th revisions in the history list (140 retriggers).

I ran into issues doing this, specifically on Bug 1073761.  The reason why is that for about 7 revisions in history the windows 8 builds failed!  Luckily the builds finished enough to get a binary+tests package so we could run tests, but mozci didn’t understand that the build was available.  That required some manual retriggering.  Actually a few cases on both retriggers were actual build failures which resulted in having to manually pick a different revision to retrigger on.  This was fairly easy to then run my tool again and fill in the 4 missing revisions using slightly different mozci parameters.

This was a bit frustrating as there was a lot of manual digging and retriggering due to build failures.  Luckily 2 of the top 10 bugs are the same root cause and we figured it out.  Including irc chatter and this blog post, I have roughly 3 hours invested into this experiment.

May 20, 2015 01:41 PM

May 19, 2015

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

May 19, 2015 06:19 AM

May 18, 2015

Joel Maher

A-Team contribution opportunity – Dashboard Hacker

I am excited to announce a new focused project for contribution – Dashboard Hacker.  Last week we gave a preview that today we would be announcing 2 contribution projects.  This is an unpaid program where we are looking for 1-2 contributors who will dedicate between 5-10 hours/week for at least 8 weeks.  More time is welcome, but not required.

What is a dashboard hacker?

When a developer is ready to land code, they want to test it. Getting the results and understanding the results is made a lot easier by good dashboards and tools. For this project, we have a starting point with our performance data view to fix up a series of nice to have polish features and then ensure that it is easy to use with a normal developer workflow. Part of the developer work flow is the regular job view, If time permits there are some fun experiments we would like to implement in the job view.  These bugs, features, projects are all smaller and self contained which make great projects for someone looking to contribute.

What is required of you to participate?

What we will guarantee from our end:

How do you apply?

Get in touch with us either by replying to the post, commenting in the bug or just contacting us on IRC (I am :jmaher in #ateam on, wlach on IRC will be the primary mentor).  We will point you at a starter bug and introduce you to the bugs and problems to solve.  If you have prior work (links to bugzilla, github, blogs, etc.) that would be useful to learn more about you that would be a plus.

How will you select the candidates?

There is no real criteria here.  One factor will be if you can meet the criteria outlined above and how well you do at picking up the problem space.  Ultimately it will be up to the mentor (for this project, it will be :wlach).  If you do apply and we already have a candidate picked or don’t choose you for other reasons, we do plan to repeat this every few months.

Looking forward to building great things!

May 18, 2015 07:43 PM

A-Team contribution opportunity – DX (Developer Ergonomics)

I am excited to announce a new focused project for contribution – Developer Ergonomics/Experience, otherwise known as DX.  Last week we gave a preview that today we would be announcing 2 contribution projects.  This is an unpaid program where we are looking for 1-2 contributors who will dedicate between 5-10 hours/week for at least 8 weeks.  More time is welcome, but not required.

What does DX mean?

We chose this project as we continue to experience frustration while fixing bugs and debugging test failures.  Many people suggest great ideas, in this case we have set aside a few ideas (look at the dependent bugs to clean up argument parsers, help our tests run in smarter chunks, make it easier to run tests locally or on server, etc.) which would clean up stuff and be harder than a good first bug, yet each issue by itself would be too easy for an internship.  Our goal is to clean up our test harnesses and tools and if time permits, add stuff to the workflow which makes it easier for developers to do their job!

What is required of you to participate?

What we will guarantee from our end:

How do you apply?

Get in touch with us either by replying to the post, commenting in the bug or just contacting us on IRC (I am :jmaher in #ateam on  We will point you at a starter bug and introduce you to the bugs and problems to solve.  If you have prior work (links to bugzilla, github, blogs, etc.) that would be useful to learn more about you that would be a plus.

How will you select the candidates?

There is no real criteria here.  One factor will be if you can meet the criteria outlined above and how well you do at picking up the problem space.  Ultimately it will be up to the mentor (for this project, it will be me).  If you do apply and we already have a candidate picked or don’t choose you for other reasons, we do plan to repeat this every few months.

Looking forward to building great things!

May 18, 2015 07:42 PM

Mark Côté

Project Isolation

The other day I read about another new Mozilla project that decided to go with GitHub issues instead of our Bugzilla installation (BMO). The author’s arguments make a lot of sense: GitHub issues are much simpler and faster, and if you keep your code in GitHub, you get tighter integration. The author notes that a downside is the inability to file security or confidential bugs, for which Bugzilla has a fine-grained permission system, and that he’d just put those (rare) issues on BMO.

The one downside he doesn’t mention is interdependencies with other Mozilla projects, e.g. the Depends On/Blocks fields. This is where Bugzilla gets into project, product, and perhaps even program management by allowing people to easily track dependency chains, which is invaluable in planning. Many people actually file bugs solely as trackers for a particular feature or project, hanging all the work items and bugs off of it, and sometimes that work crosses product boundaries. There are also a number of tracking flags and fields that managers use to prioritize work and decide which releases to target.

If I had to rebut my own point, I would argue that the projects that use GitHub issues are relatively isolated, and so dependency tracking is not particularly important. Why clutter up and slow down the UI with lots of features that I don’t need for my project? In particular, most of the tracking features are currently used only by, and thus designed for, the Firefox products (aside: this is one reason the new modal UI hides most of these fields by default if they have never been set).

This seems hard to refute, and I certainly wouldn’t want to force an admittedly complex tool on anyone who had much simpler needs. But something still wasn’t sitting right with me, and it took a while to figure out what it was. As usual, it was that a different question was going unasked, leading to unspoken assumptions: why do we have so many isolated projects, and what are we giving up by having such loose (or even no) integration amongst all our work?

Working on projects in isolation is comforting because you don’t have to think about all the other things going on in your organization—in other words, you don’t have to communicate with very many people. A lack of communication, however, leads to several problems:

By working in isolation, we can’t leverage each other’s strengths and accomplishments. We waste effort and lose great opportunities to deliver amazing things. We know that places like Twitter use monorepos to get some of these benefits, like a single build/test/deploy toolchain and coordination of breaking changes. This is what facilitates architectures like microservices and SOAs. Even if we don’t want to go down those paths, there is still a clear benefit to program management by at least integrating the tracking and planning of all of our various endeavours and directions. We need better organization-wide coordination.

We’re already taking some steps in this direction, like moving Firefox and Cloud Services to one division. But there are many other teams that could benefit from better integration, many teams that are duplicating effort and missing out on chances to work together. It’s a huge effort, but maybe we need to form a team to define a strategy and process—a Strategic Integration Team perhaps?

May 18, 2015 02:37 AM

May 15, 2015

Armen Zambrano G. (@armenzg)

mozci 0.6.0 - Trigger based on Treeherder filters, Windows support, flexible and encrypted password managament

In this release of mozci we have a lot of developer facing improvements like Windows support or flexibility on password management.
We also have our latest experimental script mozci-triggerbyfilters (

How to update

Run "pip install -U mozci" to update.


We have move all scripts from scripts/ to mozci/scripts/.
Note that you can now use "pip install" and have all scripts available as mozci-name_of_script_here in your PATH.


We want to welcome @KWierso as our latest contributor!
Our gratitude @Gijs for reporting the Windows issues and for all his feedback.
Congratulations to @parkouss for making the first project using mozci as its dependency.
In this release we had @adusca and @vaibhavmagarwal as our main and very active contributors.

Major highlights

  • Added script to trigger jobs based on Treeherder filters
    • This allows using filters like --include "web-platform-tests" and that will trigger all matching builders
    • You can also use --exclude to exclude builders you don't want
  • With the new trigger by filters script you can preview what will be triggered:
233 jobs will be triggered, do you wish to continue? y/n/d (d=show details) d
05/15/2015 02:58:17 INFO: The following jobs will be triggered:
Android 4.0 armv7 API 11+ try opt test mochitest-1
Android 4.0 armv7 API 11+ try opt test mochitest-2
  • Remove storing passwords in plain-text (Sorry!)
    • We now prompt the user if he/she wants to store their password enctrypted
  • When you use "pip install" we will also install the main scripts as mozci-name_of_script_here binaries
    • This makes it easier to use the binaries in any location
  • Windows issues
    • The python module is uncapable of decompressing large binaries
    • Do not store buildjson on a temp file and then move

Minor improvements

  • Updated docs
  • Improve wording when triggering a build instead of a test job
  • Loosened up the python requirements from == to >=
  • Added filters to

All changes

You can see all changes in here:

Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

May 15, 2015 08:13 PM

Joel Maher

Watching the watcher – Some data on the Talos alerts we generate

What are the performance regressions at Mozilla- who monitors them and what kind of regressions do we see?  I want to answer this question with a few peeks at the data.  There are plenty of previous blog posts I have done outlining stats, trends, and the process.  Lets recap what we do briefly, then look at the breakdown of alerts (not necessarily bugs).

When Talos uploads numbers to graph server they get stored and eventually run through a calculation loop to find regressions and improvements.  As of Jan 1, 2015, we upload these to as well as email to the offending patch author (if they can easily be identified).  There are a couple folks (performance sheriffs) who look at the alerts and triage them.  If necessary a bug is filed for further investigation.  Reading this brief recap of what happens to our performance numbers probably doesn’t inspire folks, what is interesting is looking at the actual data we have.

Lets start with some basic facts about alerts in the last 12 months:

As you can see this is not a casual hobby, it is a real system helping out in fixing and understanding hundreds of performance issues.

We generate alerts on a variety of branches, here is the breakdown of branches and alerts/branch;

number of regression alerts we have received per branch

number of regression alerts we have received per branch

There are a few things to keep in mind here, mobile/mozilla-central/Firefox are the same branch, and for non-pgo branches that is only linux/windows/android, not osx. 

Looking at that graph is sort of non inspiring, most of the alerts will land on fx-team and mozilla-inbound, then show up on the other branches as we merge code.  We run more tests/platforms and land/backout stuff more frequently on mozilla-inbound and fx-team, this is why we have a larger number of alerts.

Given the fact we have so many alerts and have manually triaged them, what state the the alerts end up in?

Current state of alerts

Current state of alerts

The interesting data point here is that 43% of our alerts are duplicates.  A few reasons for this:

The last piece of information that I would like to share is the break down of alerts per test:

Alerts per test

number of alerts per test (some excluded)

There are a few outliers, but we need to keep in mind that active work was being done in certain areas which would explain a lot of alerts for a given test.  There are 35 different test types which wouldn’t look good in an image, so I have excluded retired tests, counters, startup tests, and android tests.

Personally, I am looking forward to the next year as we transition some tools and do some hacking on the reporting, alert generation and overall process.  Thanks for reading!

May 15, 2015 12:40 PM

May 12, 2015

Joel Maher

community hacking – thoughts on what works for the automation and tools team

Community is a word that means a lot of things to different people.  When there is talk of community at an A*Team meeting, some people perk up and others tune out.  Taking a voluntary role in leading many community efforts on the A*Team over the last year, here are some thoughts I have towards accepting contributions, growing community, and making it work within the team.


Historically on the A*Team we would file bugs which are mentored (and discoverable via bugsahoy) and blog/advertise help wanted.  This is always met with great enthusiasm from a lot of contributors.  What does this mean for the mentor?  There are a few common axis here:

We need to appreciate all types of contributions and ensure we do our best to encourage folks to participate.  As a mentor if you have a lot of high-touch, low-reward, short-term contributors, it is exhausting and de-moralizing.  No wonder a lot of people don’t want to participate in mentoring folks as they contribute.  It is also unrealistic to expect a bunch of seasoned coders to show up and implement all the great features, then repeat for years on end.

The question remains, how do you find low-touch contributors or identify ones that are high-touch at the start and end up learning fast (some of the best contributors fall into this latter category).

Growing Community:

The simple answer here is file a bunch of bugs.  In fact whenever we do this they get fixed real fast.  This turns into a problem when you have 8 new contributors, 2 mentors, and 10 good first bugs.  Of course it is possible to find more mentors, and it is possible to file more bugs.  In reality this doesn’t work well for most people and projects.

The real question to ask is what kind of growth are you looking for?  To answer this is different for many people.  What we find of value is slowly growing our long-term/low-touch contributors by giving them more responsibility (i.e. ownership) and really depending on them for input on projects.  There is also a need to grow mentors and mentors can be contributors as well!  Lastly it is great to have a larger pool of short-term contributors who have ramped up on a few projects and enjoy pitching in once in a while.

How can we foster a better environment for both mentors and contributors?  Here are a few key areas:

Just focusing on the relationships and what comes after the good first bugs will go a long way in retaining new contributors and reducing the time spent helping out.

How we make it work in the A*Team:

The A*Team is not perfect.  We have few mentors and community is not built into the way we work.  Some of this is circumstantial, but a lot of it is within our control.  What do we do and what does and does not work for us.

Once a month we meet to discuss what is going on within the community on our team.  We have tackled topics such as project documentation, bootcamp tools/docs, discoverability, good next bugs, good first projects, and prioritizing our projects for encouraging new contributors.

While that sounds good, it is the work of a few people.  There is a lot of negative history of contributors fixing one bug and taking off.  Much frustration is expressed around helping someone with basic pull requests and patch management, over and over again.  While we can document stuff all day long, the reality is new contributors won’t read the docs and still ask questions.

The good news is in the last year we have seen a much larger impact of contributors to our respective projects.  Many great ideas were introduced, problems were solved, and experiments were conducted- all by the growing pool of contributors who associate themselves with the A*Team!

Recently, we discussed the most desirable attributes of contributors in trying to think about the problem in a different way.  It boiled down to a couple things, willingness to learn, and sticking around at least for the medium term.

Going forward we are working on growing our mentor pool, and focusing on key projects so the high-touch and timely learning curve only happens in areas where we can spread the love between domain experts and folks just getting started.

Keep an eye out for most posts in the coming week(s) outlining some new projects and opportunities to get involved.

May 12, 2015 07:41 PM

Henrik Skupin

Firefox Automation report – Q1 2015

As you may have noticed I was not able to come up with status reports of the Firefox Automation team during the whole last quarter. I feel sad about it, but there was simply no time to keep up with those blog posts. Even now I’m not sure how often I will be able to blog. So maybe I will aim to do it at least once a quarter or if possible once a month.

You may ask how it comes? The answer is simple. Our team faced some changes and finally a massive loss of core members. Which means from the former 6 people only myself are remaining. Since end of February all 5 former team members from Softvision are no longer participating in any of the maintained projects. Thanks to all of them for the great help over all the last months and years! But every project we own is now on my own shoulders. And this is kinda hell of work with downsides like not being able to do as many reviews as I want for side projects. One positive thing at least was that I got pulled back into the A-Team at the same time. With that move I’m once more closer again to all the people who care about the basics of all test infrastructure at Mozilla. I feel back home.

So what have I done the whole last quarter… First, it was always the ongoing daily work for maintaining our Mozmill CI system. This was usually a job for a dedicated person all the last months. The amount of work can sometimes eat up a whole day. Especially if several regressions have been found or incompatible changes in Firefox have been landed. Seeing my deliverables for Q1 it was clear that we have to cut down the time to spent on those failures. As result we started to partially skip tests which were failing. There was no time to get any of those fixed. Happily the latest version of Mozmill is still working kinda nicely so no other work had to be dedicated for this project.

Most of my time during the last quarter I actually had to spent on Marionette, especially building up wrapper scripts for being able to use Marionette as test framework for Firefox Desktop. This was a kinda large change for us but totally important in terms of maintenance burden and sustainability. The code base of Mozmill is kinda outdated and features like Electrolysis (e10s) will totally break it. Given that a rewrite of the test framework is too cost-intensive the decision has been made to transition our Mozmill tests over to Marionette. Side-effect was that a lot of missing features had to be implemented in Marionette to bring it at a level as what Mozmill offers. Thanks for the amazing work goes to Andrew Halberstadt, David Burns, Jonathan Griffin, and especially Chris Manchester.

For the new UI driven tests for Firefox Desktop we created the firefox-ui-tests repository at Github. We decided on that name to make it clear to which product the tests belong to, and also to get rid of any relationship to the underling test framework name. This repository contains the harness extensions around Marionette, a separate puppeteer library for back-end and UI modules, and last but not least the tests themselves. As goal for Q1 we had to get the basics working including the full set of remote security tests, and most important the update tests. A lot of help on the security tests we got from Barbara Miller our intern from December to March. She did great amount of work here, and also assisted other community members in getting their code done. Finally we got all the security tests converted.

My own focus beside the harness pieces were the update tests. Given the complete refactoring of those Mozmill tests we were able to easily port them over to Marionette. We tried to keep the class structure as is, and only did enhancements where necessary. Here Bob Silverberg helped with two big chunks of work which I’m gladly thankful about! Thanks a lot! With all modules in-place I finally converted the update tests and got them running for each version of Firefox down to 38.0, which will be the next ESR release and kinda important to be tested with Marionette. For stability and ease of contribution we added support for Travis CI to our new repository. It helps us a lot with reviews of patches from community members, and they also can see immediately if changes they have done are working as expected.

The next big chunk of work will be to get those tests running in Mozmill CI (to be renamed) and the test reporting to use Treeherder. Also we want to get our update tests for Firefox releases executed by the RelEng system, to further reduce the amount of time for signoffs from QE. About this work I will talk more in my next blog post. So please stay tuned.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agendas, the video recordings, and notes from the Firefox Automation meetings. Please note that since end of February we no longer host a meeting due to the low attendance and other meetings like the A-team ones, where I have to report my status.

May 12, 2015 04:03 PM

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

May 12, 2015 01:26 PM

Joel Maher

Re-Triggering for a [root] cause – some notes from doing it

With all this talk of intermittent failures and folks coming up with ideas on how to solve them, I figured I should dive head first into looking at failures.  I have been working with a few folks on this, specifically :parkouss and :vaibhav1994.  This experiment (actually the second time doing so) is where I take a given intermittent failure bug and retrigger it.  If it reproduces, then I go back in history looking for where it becomes intermittent.  This weekend I wrote up some notes as I was trying to define what an intermittent is.

Lets outline the parameters first for this experiment:

Here are what comes out of this:

The next step was to look at each of the 25 bugs and see if it makes sense to do this.  In fact 13 of the bugs I decided not to take action on (remember this is an experiment, so my reasoning for ignoring these 13 could be biased):

This leaves us with 12 bugs to investigate.  The premise here is easy, find the first occurrence of the intermittent (branch, platform, testjob, revision) and re-trigger it 20 times (picked for simplicity).  When the results are in, see if we have reproduced it.  In fact, only 5 bugs reproduced the exact error in the bug when re-triggered 20 times on a specific job that showed the error.

Moving on, I started re-triggering jobs back in the pushlog to see where it was introduced.  I started off with going back 2/4/6 revisions, but got more aggressive as I didn’t see patterns.  Here is a summary of what the 5 bugs turned out like:

In summary, out of 356 bugs 2 root causes were found by re-triggering.  In terms of time invested into this, I have put about 6 hours of time to fine the root cause of the 5 bugs.

May 12, 2015 01:00 PM

May 11, 2015

Julien Pagès

mozregression-gui first release!

A year from now, I started to contribute to Mozilla. Mozregression (a regression range finder for Mozilla nightly and inbound builds) was one of the first projects that catched my attention: I made one patch, then another, and one again… I was mentored by the awesome William Lachance. Since then, I became one of the core contributors and a packager for mozregression and am now myself mentoring people for new contributions on this project!

At the time of the Google Summer of Code, William and I thought about adding a graphical interface for mozregression. Unfortunately, Mozilla was not accepted for gsoc 2015 – but still the idea was interesting – and we worked on it anyway.

And here we are! Ready for a first release of mozregression-gui! Please give it a try, and help us by reporting bugs and giving feedback. :) For now we provide binaries for Windows and Linux 64 (a Mac port is planned) but it is still possible to build from sources for each platform.

Much work remains to be done on the GUI, and this is a good place to start contributing to Mozilla – so if you’d like to hack with Python and Qt don’t wait and get involved!mozregui

May 11, 2015 06:44 PM

mozregression updates

Mozregression GUI is out!

I am proud to announce the first release of mozregression-gui, a graphical interface for mozregression!

This is a pre-release alpha version, provided so you can help us to find bugs and give us some feedback.

Give it a try!

May 11, 2015 12:00 AM

May 10, 2015

Joel Maher

intermittent oranges- missing a real definition

There are all kinds of great ideas folks have for fixing intermittent issues.  In fact each idea in and of itself is a worthwhile endeavor.   I have spent some time over the last couple of months fixing them, filing bugs on them, and really discussing them.  One question that remains- what is the definition of an intermittent.

I don’t plan to lay out a definition, instead I plan to ask some questions and lay out some parameters.  According to orange factor, there are 4640 failures in the last week (May 3 -> May 10) all within 514 unique bugs.  These are all failures that the sheriffs have done some kind of manual work on to star on treeherder.  I am not sure anybody can find a way to paint a pretty picture to make it appear we don’t have intermittent failures.

Looking at a few bugs, there are many reasons for intermittent failures:

There are a lot of reasons, many of these have nothing to do with poor test cases or bad code in Firefox.  But many of these are showing up many times a day and as a developer who wants to fix a bad test, many are not really actionable.  Do we need to have some part of a definition to include something that is actionable?

Looking at the history of ‘intermittent-failure’ bugs in Bugzilla, many occur once and never occur again.  In fact this is the case for over half of the bugs filed (we file upwards of 100 new bugs/week).  While there are probably reasons for a given test case to fail, if it failed in August 2014 and has never failed again, is that test case intermittent?  As a developer could you really do anything about this given the fact that reproducing it is virtually impossible?

This is where I start to realize we need to find a way to identify real intermittent bugs/tests and not clutter the statistics with tests which are virtually impossible to reproduce.  Thinking back to what is actionable- I have found that while filing bugs for Talos regressions the closer the bug is filed to the original patch landing, the better the chance it will get fixed.  Adding to that point, we only keep 30 days of builds/test packages around for our CI automation.  I really think a definition of an intermittent needs to have some kind of concept of time.  Should we ignore intermittent failures which occur only once in 90 days?  Maybe ignore ones that don’t reproduce after 1000 iterations?  Some could argue that we look in a smaller or larger window of time/iterations.

Lastly, when looking into specific bugs, I find many times they are already fixed.  Many of the intermittent failures are actually fixed!  Do we track how many get fixed?  How many have patches and have debugging already taking place?  For example in the last 28 days, we have filed 417 intermittents, of which 55 are already resolved and of the remaining 362 only 25 have occurred >=20 times.  Of these 25 bugs, 4 already have patches.  It appears a lot of work is done to fix intermittent failures which are actionable.  Are the ones which are not being fixed not actionable?  Are they in a component where all the developers are busy and heads down?

In a perfect world a failure would never occur, all tests would be green, and all users would use Firefox.  In reality we have to deal with thousands of failures every week, most of which never happen again.  This quarter I would like to see many folks get involved in discussions and determine:

Thanks for reading, I look forward to hearing from many who have ideas on this subject.  Stay tuned for an upcoming blog post about re-trigging intermittent failures to find the root cause.

May 10, 2015 09:49 PM

May 08, 2015

Armen Zambrano G. (@armenzg)

"Thank you!"

This week I had a co-worker thank me again for a project I worked on at the end of 2014.
It touched my heart to hear it so I encourage you to do the same if someone has enabled you recently.

I have failed again and again to use the power of these two powerful words and I hope to use them more often.

Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

May 08, 2015 10:20 PM

mozci 0.5.0 released - Store password in keyring, prevent corrupted data, progress bar and many small improvements

In this release we have many small improvements that help with issues we have found.

The main improvement is that we now don't store credentials in plain-text (sorry!) but use keyring to store it encrypted.

We also prevent partially downloading any data (corrupted data) and added progress bar to downloads.

Congrats to @chmanchester as our latest contributor!
Our usual and very appreciated contributions are by @adusca @jmaher and @vaibhavmagarwal

Minor improvements:
  • Lots of test changes and increased coverage
  • Do not use the root logger but a mozci logger
  • Allow passing custom files to a triggered job
  • Work around buildbot status corruptions (Issue 167)
  • Allow passing buildernames with lower case and removing trailing spaces (since we sometimes copy/paste from TH)
  • Added support to use build a buildername based on trychooser syntax
  • Allow passing extra properties when scheduling a job on Buildbot
You can see all changes in here:

Link to official release notes.

Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

May 08, 2015 10:15 PM