Planet Mozilla Automation

July 02, 2015

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

July 02, 2015 05:59 AM

July 01, 2015

David Burns

Who wants to be an alpha tester for Marionette?

Are you an early adopter type? Are you an avid user of WebDriver and want to use the latest and great technology? Then you are most definitely in luck.

Marionette, the Mozilla implementation of the FirefoxDriver, is ready for a very limited outing. There is a lot of things that have not been implemented or, since we are implementing things agains the WebDriver Specification they might not have enough prose to implement (This has been a great way to iron out spec bugs).

Getting Started

At the moment, since things are still being developed and we are trying to do things with new technologies (like writing part this project using Rust) we are starting out with supporting Linux and OS X first. Windows support will be coming in the future!

Getting the driver

We have binaries that you can download. For Linux and for OS X . The only bindings currently updated to work are the python bindings that are available in a branch on my fork of the Selenium project. Do the following to get it into a virtualenv:
  1. Create a virtualenv
  2. activate your virtualenv
  3. cd to where you have cloned my repository
  4. In a terminal type the following: ./go py_install

Running tests

Running tests against marionette requires that you do the following changes (which hopefully remains small)

Update the desired capability to have marionette:true and add binary:/path/to/Firefox/DeveloperEdition/or/Nightly. We are only supporting those two versions of at the moment because we have had a couple in compatible issues that we have fixed which means speaking to Marionette in the browser in the beta or release versions quite difficult.

Since you are awesome early adopters it would be great if we could raise bugs. I am not expecting everything to work but below is a quick list that I know doesn't work.

Thanks for being an early adopter and thanks for raising bugs as you find them!

July 01, 2015 07:57 PM

June 30, 2015

Geoff Brown

Firefox for Android Performance Measures – Q2 Check-up

This review of Android performance measurements covers the period April 1 – June 30: the second quarter of 2015. I will write these summary posts on a quarterly basis from now on.


– Most tests fairly steady over the quarter.


This section tracks Perfomatic graphs from for mozilla-central builds of Firefox for Android, for Talos tests run on Android 4.0 Opt. The test names shown are those used on treeherder. See for background on Talos.


Measure of “checkerboarding” during simulation of real user interaction with page. Lower values are better.


19 (start of period) – 20 (end of period)

Small regression on June 10 (no bug?). This test is exhibiting some inconsistent behavior, as noted in bug 1149567.


An svg-only number that measures SVG rendering performance. About half of the tests are animations or iterations of rendering. This ASAP test (tsvgx) iterates in unlimited frame-rate mode thus reflecting the maximum rendering throughput of each test. The reported value is the page load time, or, for animations/iterations – overall duration the sequence/animation took to complete. Lower values are better.


720 (start of period) – 680 (end of period).

Small improvement on May 27.


Generic page load test. Lower values are better.


680 (start of period) – 670 (end of period).

Small improvement on May 18.

Throbber Start / Throbber Stop

These graphs are taken from  Browser startup performance is measured on real phones (a variety of popular devices).

Unfortunately, I could not find any devices for mozilla-central which reported consistently throughout this period.




These graphs are taken from Eideticker is a performance harness that measures user perceived performance of web browsers by video capturing them in action and subsequently running image analysis on the raw result.

More info at:




Most other tests have incomplete data for this time period.


These graphs are taken from the mozbench dashboard at which includes some comparisons involving Firefox for Android. More info at





June 30, 2015 10:41 PM

June 29, 2015

Alice Scarpa

Changing goals

I started my Outreachy internship with Mozilla on May 25th, working with mozci. A lot happened in the first five weeks:

Part of the application process was writing a project proposal. Mozci was about 2 months old at the time and so much changed since then that strictly following the original proposal would have been a bad idea.

Our current plan for the rest of my internship is to focus on using Pulse Actions to add new functionality to Treeherder, but a lot can happen until then (for example, we depend a lot on feedback from the sheriffs to guide us). I’m not nervous because I know I can count with the help of people from Outreachy, the Recurse Center and the A-team, especially my mentor Armen who has been awesome since I started contributing to mozci a long long time ago (four whole months!).

June 29, 2015 12:00 AM

June 16, 2015

Armen Zambrano G. (@armenzg)

mozci 0.8.0 - New feature -- Trigger coalesced jobs + improved performance

Beware! This release is full awesome!! However, at the same time new bugs might pop up so please let us know :)
You can trigger now all coalesced jobs on a revision:
mozci-trigger --coalesced --repo-name REPO -r REVISION


Thanks to @adusca @glandium @vaibhavmagarwal @chmanchester for their contributions on this release.

How to update

Run "pip install -U mozci" to update

Major highlights

  • #259 - New feature - Allow triggering
  • #227 - Cache files as gzip files instead of uncompressed
    • Less disk print
  • #227 - Allow using multiple builders
  • 1e591bf - Make sure that we do not fetch files if they are not newer
    • We were failing to double check that the last modification date of a file was the same as the one in the server
    • Hence, we were downloading files more often than needed
  • Caching builds-4hr on memory for improved performance

Minor improvements

  • f72135d - Backfilling did not accept dry_run or to trigger more than once
  • More tests and documents
  • Support for split tests (test packages json file)
  • Some OOP refactoring

All changes

You can see all changes in here:

Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

June 16, 2015 05:38 PM

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

June 16, 2015 06:42 AM

June 13, 2015

Syd Polk

WWDC Day 4 - ...and that's a wrap

And the last day...

I had lunch with Emily Toop on the Firefox iOS team. She is new to Mozilla, but an experienced iOS developer out of the UK. Lovely venue at the Samovar Tea Lounge in Yerba Buena Gardens.

Overall impressions of WWDC:
I would be happy to come back next year, but I am not going to push again unless programming the Mac or iOS is my primary job. Lots of good stuff. Also, there are some sessions from last year I need to watch.

June 13, 2015 12:35 AM

June 12, 2015

Syd Polk

WWDC Day 3 - Sessions are starting to run together...

We are in the thick of things now; next to last day. Sessions I attended:

I had lunch with my good buddy Jim Ingham. We worked together at Sun, Cygnus, Red Hat and Apple, off and on from 1997-2006. It was good to see him. I wish I could remember the last name of the other Apple engineer who joined us, Greg. He had been there a few months when I left, and he had moved from Austin (!). It was good to eat a good lunch away from the show.

I also went to the Apple Bash.  They had the band Walk the Moon, and I was pleasantly surprised. There was actually musicianship on display. I am not going to buy anything, but for once, I wasn't counting time until the band stopped.

It was not a dinner outing. There were small entree-like snacks, but it's not the same. So Paul and I found a late-night place near Union Square.

June 12, 2015 04:56 PM

June 11, 2015

Syd Polk

WWDC Day 2 - more sessions and a party

Lots of sessions today. Let's tear right in:

After the main conference, I went to see James Dempsey and the Breakpoints. Uneven songs musically, but the words are all about programming in Cocoa, and it's pretty funny. Check it out on iTunes if you are into that sort of thing.

And I closed the night having an excellent cocktail with my best friend, Paul Tien-Shih Lee, whose birthday it was. The View bar in the Marriot is wonderful.

June 11, 2015 03:56 PM

June 10, 2015

Syd Polk

WWDC Day 1 - Sessions, Sessions, and some old friends

Tuesday was filled with sessions, with breaks in the Developer Tools lab. Sessions I attended:

I also spent some time in the Developer Tools Lab catching up with former coworkers. Plus, I discovered an Xcode 7 bug launching the simulator, and I talked to them about that.

After the show, I met some friends from college and/or my very first projects for dinner in San Mateo. I was truly great to see them.

June 10, 2015 04:21 PM

Alice Scarpa

Tracking down a non-deterministic coverage failure

At Mozilla CI Tools, we use Coveralls to track our test coverage progress. In the past month we noticed that sometimes the coverage changes even in PRs that only changed files that are not tracked by coveralls. How come?

Comparing coverage reports for builds in which this problem happens, I noticed that this line would sometimes be marked as covered, sometimes as not covered. The line belongs to a function that builds a graph from a list. An interesting thing about this function is that the order of the list does not affect the final graph, but it does affect the code path taken inside the function. In the test for this function, the input comes from a call to a dict.keys() method. Also, an interesting thing about the .keys() method is that it always produces a list containing the same elements, but the order varies. Mystery solved!

So a single sort() command on the test input could fix those annoying random coverage failures!

June 10, 2015 12:00 AM

June 09, 2015

Joel Maher

Please welcome the Dashboard Hacker team

A few weeks ago we announced that we would be looking for committed contributors for 8+ weeks on Perfherder.  We have found a few great individuals, all of whom show a lot of potential and make a great team.

They are becoming familiar with the code base and are already making a dent in the initial list of work set aside.  Let me introduce them (alphabetical order by nicks):

akhileshpillai – Akhilesh has jumped right in and started fixing bugs in Perfherder.  He is new to Mozilla and will fit right in.  With how fast he has come up to speed, we are looking forward to what he will be delivering in the coming weeks.  We have a lot of UI workflow as well as backend data refactoring work on our list, all of which he will be a great contributor towards.

mikeling – mikeling has been around Mozilla for roughly two years, recently he started out helping with a few projects on the A*Team.  He is very detailed oriented, easy to work with and is willing to tackle big things.

theterabyte – Tushar is excited about this program as an opportunity to grow his skills as a python developer and experiencing how software is built outside of a classroom.  Tushar will get a chance to grow his UI skills on Perfherder by making the graphs and compare view more polished and complete, while helping out with an interface for the alerts.

Perfherder will soon become the primary dashboard for all our performance needs.

I am looking forward to the ideas and solutions these new team members bring to the table.  Please join me in welcoming them!

June 09, 2015 04:55 PM

Please join me in welcoming the DX Team

A few weeks ago, I posted a call out for people to reach out and commit to participate for 8+ weeks.  There were two projects and one of them was Developer Experience.  Since then we have had some great interest, there are 5 awesome contributors participating (sorted by irc nicks).

BYK – I met BYK 3+ years ago on IRC- he is a great person and very ambitious.  As a more senior developer he will be focused primarily on improving interactions with mach.  While there are a lot of little things to make mach better, BYK proposed a system to collect information about how mach is used.

gma_fav – I met gma_fav on IRC when she heard about the program.  She has a lot of energy, seems very detail oriented, asks good questions, and brings fresh ideas to the team!  She is a graduate of the Ascend project and is looking to continue her growth in development and open source.  Her primary focus will be on the interface to try server (think the try chooser page, extension, and taking other experiments further).

kaustabh93 – I met Kaustabh on IRC about a year ago and since then he has been a consistent friend and hacker.  He attends university.  In fact I do owe him credit for large portions of alert manager.  While working on this team, he will be focused on making run-by-dir a reality.  There are two parts: getting the tests to run green, and reducing the overhead of the harness.

sehgalvibhor – I met Vibhor on IRC about 2 weeks ago.  He was excited about the possibility of working on this project and jumped right in.  Like Kaustabh, he is a student who is just finishing up exams this week.  His primary focus this summer will be working in a similar role to Stanley in making our test harnesses act the same and more useful.

stanley – When this program was announced Stanley was the first person to ping me on IRC.  I have found him to be very organized, a pleasure to chat with and he understands code quite well.  Coding and open source are both new things to Stanley and we have the opportunity to give him a great view of it.  Stanley will be focusing on making the commands we have for running tests via mach easier to use and more unified between harnesses.

Personally I am looking forward to seeing the ambition folks have translate into great solutions, learning more about each person, and sharing with Mozilla as a whole the great work they are doing.

Take a few moments to say hi to them online.

June 09, 2015 04:32 PM

Syd Polk

WWDC Day 0, Part 2 - Platforms State of the Union

The Platforms State of the Union session was much more interesting than the keynote. Highlights applicable to what I work on:
You should do those things anyway.
Great session.

June 09, 2015 04:14 PM

David Burns

WebDriver Specification - Have you read it lately?

A lot of work has gone into the WebDriver Specification this year. The methods in there have had a major make over to make them more specific in the steps that are required as well as having the relevant links. Go have a read of it and feel free to raise bugs against it, we will be updating it quite regularly. You can see all the work that is happening on Github. We do everything via pull requests so you can read things before they land.

My team have also been working hard at making sure that our implementation is following the specification and are making some great leaps with it. I will be releasing a development version of the python bindings soon that use the httpd, like InternetExplorerDriver and ChromeDriver, to drive the browser. Currently our httpd only works against Nightly but there is a merge to Aurora happening soon when we will be sending out links for you to start playing with it all. I am actually looking forward to some of the feedback that we get about it.

June 09, 2015 02:42 PM

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

June 09, 2015 06:37 AM

June 08, 2015

Syd Polk

WWDC Day 0, Part 1 - Keynote

First of all, there are a lot of people here. Last night, my buddy told me that I might want to line up early; otherwise, I might have ended up in the overflow room.

And my Mozilla colleagues, Emily Toop and Darrin Henin, wanted to hook up early. So I set out to Moscone West and arrived at 6:30. I initially went to the back of the line. It went north on 4th, all of the way west on Minna, back South on 5th, and then almost completing the circle halfway back up Howard. 

Fortunately, the other Mozillians found some of their friends who work at Facebook, and got us a place at the front of the line. Hooray.

They let us into the building at 8. We went up the escalators, and then walked all of the way around the second floor, where we waited in the (hot, crowded) hallway for another hour.

At nine, they let us crowd around the base of the escalator to the 3rd floor. We were there for about half an hour when we were finally let into the main hall.

The keynote itself was a little disappointing. New versions of OS X and iOS with some whizzy features. My favorite was the multiapp features in iPad; looking forward to playing with those. My single favorite announcement was that Swift is now going to be Open Source and run on Linux. They then talked about watchOS, which I currently have no interest in.

And then they wasted 45 minutes with the Apple Music announcement. OK, I can stream all of Apple's catalog. But I have to either stream or play my library? Wasn't clear. And the performance did nothing for me.

So. OS X El Capitain and iOS 9: Yay. Looks good. Carry on. 
watchOS: Cool, I guess. 
MUSIC: meh.

June 08, 2015 08:45 PM

Dave Hunt

Joining Web QA

Dylan with origami foxI’m excited to announce that as of last week I am officially on Mozilla’s Web QA team! Despite working closely with the team since I started at Mozilla over four years ago, I’ve always reported to another team. Originally I had a hybrid role, where I reported to the Director of QA and assisted with automation of both Web and Desktop products. Later, we formed the QA Automation Services team, which existed to provide automation services to any QA team. This was mostly absorbed into the A-Team, which shares a lot of the same goals but is not just focused on QA. During my time with the A-Team a lot of my work started to shift towards Firefox OS, so it made sense during a organisational shift towards vertical stacks for me to officially join the Firefox OS automation team.

Many times since I started at Mozilla I’ve felt that I had more to offer the Web QA team, and I’ve always been a keen contributor to the team. I can’t say it was an easy decision to move away from Firefox OS, as it’s a terrifically exciting project, but the thought of joining Web QA just had me bursting with enthusiasm! In a time where there’s been a number of departures, I’m pleased to say that I feel like I’m coming home. Look out web – you’re about to get even more automated!

Last week Stephen Donner interviewed me on being a Mozilla Web QA contributor. You can read the blog post over on the Web QA blog.

I’d like to take this opportunity to thank Stephen Donner, Matt Evans, Clint Talbert, Jonathan Griffin, and James Lal for their support and leadership. I’ve made so many great friendships at Mozilla, and with our Whistler work week just around the corner I’m so looking forward to catching up with many of them in person!

June 08, 2015 09:42 AM

Syd Polk

WWDC - D -1

When WWDC signups went up, I decided to take a chance and see if I could score a ticket. Much to my surprise, I did get one. So, here I am, in San Francisco, the night before it starts, and I am about to go for the first time since 2005.

In that WWDC, I was an Apple employee. I worked on the Developer Tools team. We were shuffled into the Apple Employee room for the Keynote.  At the end of the keynote, Steve Jobs announced that Apple was moving the Mac product line to Intel. I was on the team that was "in the know" about the transition. SJ asked us all to stand up, and the rest of the employees gave us a standing O. Was one of the proudest moments of my career.

Since then, I have moved to Austin and have worked several jobs. Currently, I work in automation for Mozilla. I hope to do some development work someday, and Mozilla has an effort underway for iOS, so I am going to be attending sessions and talking with people here about that effort. I am also going to sessions for my personal projects.

I checked in today, and they did something really smart. They gave us a windbreaker. Anybody who has been to San Francisco in June knows that this is really smart. There are probably many attendees who have never been here and don't know that June and July are cold, foggy and windy here.

This is not the first year they have done this (I saw somebody with a "14" on his back), but it is really brilliant.

Tomorrow is dominated by the Keynote, and in the afternoon, the only things to see are Platforms State of the Union and the Apple Design awards.

I'm stoked!

June 08, 2015 01:44 AM

June 06, 2015

Julien Pagès

Python code cleanup – vulture

I’m pretty sure you all already know about flake8, a great tool that combine other tools (PyFlakes, pep8 and Ned Batchelder’s McCabe script) to check style issues and code errors.

Working on Talos, I needed to find a way to find dead code on the codebase (the less code, the better!). I found a pretty good tool that helped me: vulture. This is a one shot tool – it can not be automated like flake8 as there is some false positives, but still running it once on an old codebase may be a big win to find and remove useless stuff.

Obviously it is easier to use on python programs – you will have to whitelist a lot of things if you run this on a python library that offers a public API.

If you want to seek and destoy dead code on your python program, give it a try!

June 06, 2015 09:46 AM

June 05, 2015

Alice Scarpa


Recently we had an ‘UnboundLocalError’ in mozci. A local variable was being referenced before assignment. Turns out we were only defining the variable in one branch of an ‘if’ statement. It was very straightforward to fix. What bothered me was that this is the exact sort of error that would have been caught at compile time in other languages.

Is there a Python static analysis tool that would have caught this? I wrote a very simple module with the same bug and tried with different tools1:

"""This module bakes pizzas."""

def bake_pizza():
    """Bake a pizza."""
    number = int(raw_input())

    if number > 3:
        additional = 2
        number += 1
    print number + additional

if __name__ == '__main__':

Neither Pylint, Pyflakes nor PyChecker were able to notify me about the possibility of an ‘UnboundLocalError’.

ekyle suggested that PyCharm might be able to solve this problem, and indeed, PyCharm code inspection shows the message I wanted:

>  Local variable 'additional' might be referenced before assignment

I spent a couple of hours today going through every PyCharm warning in mozci to see if there were any other bugs that it could catch. It didn’t find other serious issues (I don’t know if that’s fortunate or not).

  1. It ended up being the only module I’ve ever written that scored 10.0/10 in pylint.

June 05, 2015 12:00 AM

June 03, 2015

Armen Zambrano G. (@armenzg)

mozci 0.7.2 - Support b2g jobs that still run on Buildbot

There are a lot of b2g (aka Firefox OS) jobs that still run on Buildbot .
Interestingly enough we had not tried before to trigger one with mozci.
This release adds support for it.
This should have been a minor release (0.8.0) rather than a security release (0.7.2). My apologies!
All jobs that start with "b2g_" in all_builders.txt are b2g jobs that still run on Buildbot instead of TaskCluster (docs - TC jobs on treeherder).

Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

June 03, 2015 07:23 PM

mozci 0.7.1 - regression fix - do not look for files for running jobs

This release mainly fixes a regression we introduced in the release 0.7.0.
The change (#220) we introduced checked completed and running jobs for files that have been uploaded in order to trigger tests.
The problem is that running jobs do not have any metadata until they actually complete.
We fixed this on #234.


Thanks to @adusca and @glandium for their contributions on this release.

How to update

Run "pip install -U mozci" to update

Major highlights

  • #234 - (bug fix) - Do not try to find files for running jobs
  • #228 - For try, only trigger talos jobs on existing build jobs ** rather than triggering builds for platforms that were not requested
  • #238 - Read credentials through environment variables

Minor improvements

  • #226 - (bug fix) Properly cache downloaded files
  • #228 - (refactor) Move SCHEDULING_MANAGER
  • #231 - Doc fixes

All changes

You can see all changes in here:

Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

June 03, 2015 07:14 PM

Joel Maher

re-triggering for a [root] cause- version 2.0 – a single bug!

Today the orange factor email came out- the top 10 bugs with stars on them :)  Last week we had no bugs that we could take action on, and the week before we had a few bugs to take action on.

This week I looked at each bug again, annotated them with some notes as to what I did or why I didn’t do anything, here are the bugs:

It is nice to find a bug that we can take action on.  What is interesting is the bug has been around for a while, but we noticed about May 21 that the rate of failures went up from a couple a day to >5/day.  Details:

I might try a full experiment soon blindly looking at bugs instead of using orange factor.

June 03, 2015 10:46 AM

June 02, 2015

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

June 02, 2015 08:03 AM

May 28, 2015

Joel Maher

the orange factor – no need to retrigger this week

last week I did another round of re-triggering for a root cause and found some root causes!  This week I got an email from orange factor outlining the top 10 failures on the trees (as we do every week).

Unfortunately as of this morning there is no work for me to do- maybe next week I can hunt.

Here is the breakdown of bugs:

As you can see there isn’t much to do here.  Maybe next week we will have some actions we can take.  Once I have about 10 bugs investigated I will summarize the bugs, related dates, and status, etc.

May 28, 2015 04:15 PM

Armen Zambrano G. (@armenzg)

mozci 0.7.0 - Less network fetches - great speed improvements!

This release is not large in scope but it has many performance improvements.
The main improvement is to have reduced the number of times that we fetch for information and use a cache where possible. The network cost was very high.
You can read more about in here:


Thanks to @adusca @parkouss @vaibhavmagarwal for their contributions on this release.

How to update

Run "pip install -U mozci" to update

Major highlights

  • Reduce drastically the number of requests by caching where possible
  • If a failed build has uploaded good files let's use them
  • Added support for retriggering and cancelling jobs
  • Retrigger a job once with a count of N instead of triggering individually N times

Minor improvements

  • Documenation updates
  • Add badge

All changes

You can see all changes in here:

Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

May 28, 2015 02:01 PM

Byron Jones

happy bmo push day!

it appears to be “bmo push week“.  today’s push addresses some issues caused by yesterday’s push.

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

May 28, 2015 07:06 AM

Alice Scarpa

Acting upon Pulse messages

Pulse Actions is a service that listens to retrigger/cancel messages on exchange/treeherder/v1/job-actions and acts upon them using Mozilla CI Tools.

The project was deployed to Heroku today in dry-run mode, i.e., instead of actually making retrigger/cancel requests, Pulse Actions is just logging the requests it would have made.

If this experiment works, it might be possible for Treeherder to use the service for retrigger/cancel requests. That would eliminate the need for users signed in with Persona to enter LDAP credentials to make TH requests (bug 1077053) and would also open a lot of new possibilities (e.g. adding a backfill button to TH using mozci).

This project’s progress is being tracked in bug 1168148.

May 28, 2015 12:00 AM

May 27, 2015

Armen Zambrano G. (@armenzg)

Welcome adusca!

It is my privilege to announce that adusca (blog) joined Mozilla (since Monday) as an Outreachy intern for the next 4 months.

adusca has an outstanding number of contributions over the last few months including Mozilla CI Tools (which we're working on together).

Here's a bit about herself from her blog:
Hi! I’m Alice. I studied Mathematics in college. I was doing a Master’s degree in Mathematical Economics before getting serious about programming.
She is also a graduate from Hacker's School.

Even though Alice has not been a programmer for many years, she has shown already lots of potential. For instance, she wrote a script to generate scheduling relations for buildbot; for this and many other reasons I tip my hat.

adusca will initially help me out with creating a generic pulse listener to handle job cancellations and retriggers for Treeheder. The intent is to create a way for Mozilla CI tools to manage scheduling on behalf of TH, make the way for more sophisticated Mozilla CI actions and allow other people to piggy back to this pulse service and trigger their own actions.

If you have not yet had a chance to welcome her and getting to know her, I highly encourage you to do so.

Welcome Alice!

Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

May 27, 2015 05:10 PM

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

May 27, 2015 07:11 AM

May 26, 2015

Geoff Brown

Handling intermittent test timeouts in long-running tests

Tests running on our new-ish Android 4.3 Opt emulator platform have recently been plagued by intermittent timeouts and I have been having a closer look at some of them (like bug 919246 and bug 1154505) .

A few of these tests normally run “quickly”. Think of a test that runs to completion in under 10 seconds most of the time but times out after 300+ seconds intermittently. In a case like this, it seems likely that there is an intermittent hang and the test needs debugging to determine the underlying cause.

But most of the recent Android 4.3 Opt test timeouts seem to be affecting what I classify as “long-running” tests. Think of a test that normally runs to completion in 250 to 299 seconds, but intermittently times out after 300 seconds. It seems likely that normal variations in test duration are intermittently pushing past the timeout threshold; if we can tolerate a longer time-out, or make the test run faster in general, we can probably eliminate the intermittent test failure.

We have a lot of options for dealing with long-running tests that sometimes timeout.

Option: Simplify or optimize the test

Long-running tests are usually doing a lot of work. A lot of assertions can be run in 300 seconds, even on a slow platform! Do we need to test all of those cases, or could some be eliminated? Is there some setup or tear down code being run repeatedly that could be run just once, or even just less often?

We usually don’t worry about optimizing tests but sometimes a little effort can help a test run a lot more efficiently, saving test time, money (think aws costs), and aggravation like intermittent time-outs.

Option: Split the test into 2 or more smaller tests

Some tests can be split into 2 or more smaller tests with minimal effort. Instead of testing 100 different cases in one test, we may be able to test 50 in each. There may be some loss of efficiency: Maybe some setup code will need to be run twice, and copied and pasted to the second test. But now each half runs faster, reducing the chance of a timeout. And when one test fails, the cause is – at least slightly – more isolated.

Option: Request a longer timeout for the test

Mochitests can call SimpleTest.requestLongerTimeout(2) to double the length of the timeout applied to the test. We currently have about 100 mochitests that use this feature.

For xpcshell tests, the same thing can be accomplished with a manifest annotation:

requesttimeoutfactor = 2

That’s a really simple “fix” and an effective way of declaring that a test is known to be long-running.

On the other hand, it is avoiding the problem and potentially covering up an issue that could be solved more effectively by splitting, optimizing, or simplifying. Also, long-running tests make our test job “chunking” less effective: It’s harder to split load evenly amongst jobs when some tests run 100 times longer than others.

Option: Skip the test on slow platforms

Sometimes it’s not worth the effort. Do we really need to run this test on Android as well as on all the desktop platforms? Do we get value from running this test on both Android 2.3 and Android 4.3? We may “disable our way to victory” too often, but this is a simple strategy, doesn’t affect other platforms and sometimes it feels like the right thing to do.

Option: Run on faster hardware

This usually isn’t practical, but in special circumstances it seems like the best way forward.

If you have a lot of timeouts from long-running tests on one platform and those tests don’t timeout on other platforms, it may be time to take a closer look at the platform.

Our Android arm emulator test platforms are infamous for slowness. In fairness, the emulator has a lot of work to do, Firefox is complex, our tests are often relentless (compared to human-driven browsing), and we normally run the emulator on the remarkably slow (and cheap!) m1.medium AWS instances.

If we are willing to pay for better cpu, memory, and I/O capabilities, we can easily speed up the emulator by running on a faster AWS instance type — but the cost must be justified.

I recently tried running Android 4.3 Debug mochitests on m1.medium and found that many tests timed out. Also, since all tests were taking longer, each test job (each “chunk”) needed 2 to 3 hours to complete — much longer than we can wait. Increasing chunks seemed impractical (we would need 50 or so) and we would still have all those individual timeouts to deal with. In this case, running the emulator on c3.xlarge instances for Debug mochitests made a big difference, allowing them to run in the same number of chunks as Opt on m1.medium and eliminating nearly all timeouts.

I’ve enjoyed investigating mochitest timeouts and found most of them to be easy to resolve. I’ll try to investigate more timeouts as I see them. Won’t you join me?

May 26, 2015 11:34 PM

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

May 26, 2015 04:49 AM

May 25, 2015

Alice Scarpa

Using cProfile with gprof2dot to identify bottlenecks

At Mozilla CI Tools we have a script that can sometimes be annoyingly slow. I used cProfile with gprof2dot to better understand how to improve it.

To get a pretty graph of the script’s behaviour I ran:

python -m cProfile -o timing
gprof2dot -f pstats timing -o
dot -Tsvg -o graph_master.svg

This gave me a very useful graph.

Looking at the graph I was able to identify two bottlenecks that were low-hanging fruit: query_jobs and valid_revision. These two functions are called a lot of times in the script with the same arguments. This means that by adding some simple caching I could improve the script’s speed. Preliminary results show a 2x speed-up. There is still a lot of room for improvement, but it’s a nice start.

May 25, 2015 12:00 AM

May 20, 2015

Joel Maher

re-triggering for a [root] cause – version 1.57

Last week I wrote some notes about re-triggering jobs to find a root cause.  This week I decided to look at the orange factor email of the top 10 bugs and see how I could help.  Looking at each of the 10 bugs, I had 3 worth investigating and 7 I ignored.



Looking at the bugs of interest, I jumped right in in retriggering.  This time around I did 20 retriggers for the original changeset, then went back to 30 revisions (every 5th) doing the same thing.  Effectively this was doing 20 retriggers for the 0, 5th, 10th, 15th, 20th, 25th, and 30th revisions in the history list (140 retriggers).

I ran into issues doing this, specifically on Bug 1073761.  The reason why is that for about 7 revisions in history the windows 8 builds failed!  Luckily the builds finished enough to get a binary+tests package so we could run tests, but mozci didn’t understand that the build was available.  That required some manual retriggering.  Actually a few cases on both retriggers were actual build failures which resulted in having to manually pick a different revision to retrigger on.  This was fairly easy to then run my tool again and fill in the 4 missing revisions using slightly different mozci parameters.

This was a bit frustrating as there was a lot of manual digging and retriggering due to build failures.  Luckily 2 of the top 10 bugs are the same root cause and we figured it out.  Including irc chatter and this blog post, I have roughly 3 hours invested into this experiment.

May 20, 2015 01:41 PM

May 19, 2015

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

May 19, 2015 06:19 AM

May 18, 2015

Joel Maher

A-Team contribution opportunity – Dashboard Hacker

I am excited to announce a new focused project for contribution – Dashboard Hacker.  Last week we gave a preview that today we would be announcing 2 contribution projects.  This is an unpaid program where we are looking for 1-2 contributors who will dedicate between 5-10 hours/week for at least 8 weeks.  More time is welcome, but not required.

What is a dashboard hacker?

When a developer is ready to land code, they want to test it. Getting the results and understanding the results is made a lot easier by good dashboards and tools. For this project, we have a starting point with our performance data view to fix up a series of nice to have polish features and then ensure that it is easy to use with a normal developer workflow. Part of the developer work flow is the regular job view, If time permits there are some fun experiments we would like to implement in the job view.  These bugs, features, projects are all smaller and self contained which make great projects for someone looking to contribute.

What is required of you to participate?

What we will guarantee from our end:

How do you apply?

Get in touch with us either by replying to the post, commenting in the bug or just contacting us on IRC (I am :jmaher in #ateam on, wlach on IRC will be the primary mentor).  We will point you at a starter bug and introduce you to the bugs and problems to solve.  If you have prior work (links to bugzilla, github, blogs, etc.) that would be useful to learn more about you that would be a plus.

How will you select the candidates?

There is no real criteria here.  One factor will be if you can meet the criteria outlined above and how well you do at picking up the problem space.  Ultimately it will be up to the mentor (for this project, it will be :wlach).  If you do apply and we already have a candidate picked or don’t choose you for other reasons, we do plan to repeat this every few months.

Looking forward to building great things!

May 18, 2015 07:43 PM

A-Team contribution opportunity – DX (Developer Ergonomics)

I am excited to announce a new focused project for contribution – Developer Ergonomics/Experience, otherwise known as DX.  Last week we gave a preview that today we would be announcing 2 contribution projects.  This is an unpaid program where we are looking for 1-2 contributors who will dedicate between 5-10 hours/week for at least 8 weeks.  More time is welcome, but not required.

What does DX mean?

We chose this project as we continue to experience frustration while fixing bugs and debugging test failures.  Many people suggest great ideas, in this case we have set aside a few ideas (look at the dependent bugs to clean up argument parsers, help our tests run in smarter chunks, make it easier to run tests locally or on server, etc.) which would clean up stuff and be harder than a good first bug, yet each issue by itself would be too easy for an internship.  Our goal is to clean up our test harnesses and tools and if time permits, add stuff to the workflow which makes it easier for developers to do their job!

What is required of you to participate?

What we will guarantee from our end:

How do you apply?

Get in touch with us either by replying to the post, commenting in the bug or just contacting us on IRC (I am :jmaher in #ateam on  We will point you at a starter bug and introduce you to the bugs and problems to solve.  If you have prior work (links to bugzilla, github, blogs, etc.) that would be useful to learn more about you that would be a plus.

How will you select the candidates?

There is no real criteria here.  One factor will be if you can meet the criteria outlined above and how well you do at picking up the problem space.  Ultimately it will be up to the mentor (for this project, it will be me).  If you do apply and we already have a candidate picked or don’t choose you for other reasons, we do plan to repeat this every few months.

Looking forward to building great things!

May 18, 2015 07:42 PM

Mark Côté

Project Isolation

The other day I read about another new Mozilla project that decided to go with GitHub issues instead of our Bugzilla installation (BMO). The author’s arguments make a lot of sense: GitHub issues are much simpler and faster, and if you keep your code in GitHub, you get tighter integration. The author notes that a downside is the inability to file security or confidential bugs, for which Bugzilla has a fine-grained permission system, and that he’d just put those (rare) issues on BMO.

The one downside he doesn’t mention is interdependencies with other Mozilla projects, e.g. the Depends On/Blocks fields. This is where Bugzilla gets into project, product, and perhaps even program management by allowing people to easily track dependency chains, which is invaluable in planning. Many people actually file bugs solely as trackers for a particular feature or project, hanging all the work items and bugs off of it, and sometimes that work crosses product boundaries. There are also a number of tracking flags and fields that managers use to prioritize work and decide which releases to target.

If I had to rebut my own point, I would argue that the projects that use GitHub issues are relatively isolated, and so dependency tracking is not particularly important. Why clutter up and slow down the UI with lots of features that I don’t need for my project? In particular, most of the tracking features are currently used only by, and thus designed for, the Firefox products (aside: this is one reason the new modal UI hides most of these fields by default if they have never been set).

This seems hard to refute, and I certainly wouldn’t want to force an admittedly complex tool on anyone who had much simpler needs. But something still wasn’t sitting right with me, and it took a while to figure out what it was. As usual, it was that a different question was going unasked, leading to unspoken assumptions: why do we have so many isolated projects, and what are we giving up by having such loose (or even no) integration amongst all our work?

Working on projects in isolation is comforting because you don’t have to think about all the other things going on in your organization—in other words, you don’t have to communicate with very many people. A lack of communication, however, leads to several problems:

By working in isolation, we can’t leverage each other’s strengths and accomplishments. We waste effort and lose great opportunities to deliver amazing things. We know that places like Twitter use monorepos to get some of these benefits, like a single build/test/deploy toolchain and coordination of breaking changes. This is what facilitates architectures like microservices and SOAs. Even if we don’t want to go down those paths, there is still a clear benefit to program management by at least integrating the tracking and planning of all of our various endeavours and directions. We need better organization-wide coordination.

We’re already taking some steps in this direction, like moving Firefox and Cloud Services to one division. But there are many other teams that could benefit from better integration, many teams that are duplicating effort and missing out on chances to work together. It’s a huge effort, but maybe we need to form a team to define a strategy and process—a Strategic Integration Team perhaps?

May 18, 2015 02:37 AM

May 15, 2015

Armen Zambrano G. (@armenzg)

mozci 0.6.0 - Trigger based on Treeherder filters, Windows support, flexible and encrypted password managament

In this release of mozci we have a lot of developer facing improvements like Windows support or flexibility on password management.
We also have our latest experimental script mozci-triggerbyfilters (

How to update

Run "pip install -U mozci" to update.


We have move all scripts from scripts/ to mozci/scripts/.
Note that you can now use "pip install" and have all scripts available as mozci-name_of_script_here in your PATH.


We want to welcome @KWierso as our latest contributor!
Our gratitude @Gijs for reporting the Windows issues and for all his feedback.
Congratulations to @parkouss for making the first project using mozci as its dependency.
In this release we had @adusca and @vaibhavmagarwal as our main and very active contributors.

Major highlights

  • Added script to trigger jobs based on Treeherder filters
    • This allows using filters like --include "web-platform-tests" and that will trigger all matching builders
    • You can also use --exclude to exclude builders you don't want
  • With the new trigger by filters script you can preview what will be triggered:
233 jobs will be triggered, do you wish to continue? y/n/d (d=show details) d
05/15/2015 02:58:17 INFO: The following jobs will be triggered:
Android 4.0 armv7 API 11+ try opt test mochitest-1
Android 4.0 armv7 API 11+ try opt test mochitest-2
  • Remove storing passwords in plain-text (Sorry!)
    • We now prompt the user if he/she wants to store their password enctrypted
  • When you use "pip install" we will also install the main scripts as mozci-name_of_script_here binaries
    • This makes it easier to use the binaries in any location
  • Windows issues
    • The python module is uncapable of decompressing large binaries
    • Do not store buildjson on a temp file and then move

Minor improvements

  • Updated docs
  • Improve wording when triggering a build instead of a test job
  • Loosened up the python requirements from == to >=
  • Added filters to

All changes

You can see all changes in here:

Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

May 15, 2015 08:13 PM

Joel Maher

Watching the watcher – Some data on the Talos alerts we generate

What are the performance regressions at Mozilla- who monitors them and what kind of regressions do we see?  I want to answer this question with a few peeks at the data.  There are plenty of previous blog posts I have done outlining stats, trends, and the process.  Lets recap what we do briefly, then look at the breakdown of alerts (not necessarily bugs).

When Talos uploads numbers to graph server they get stored and eventually run through a calculation loop to find regressions and improvements.  As of Jan 1, 2015, we upload these to as well as email to the offending patch author (if they can easily be identified).  There are a couple folks (performance sheriffs) who look at the alerts and triage them.  If necessary a bug is filed for further investigation.  Reading this brief recap of what happens to our performance numbers probably doesn’t inspire folks, what is interesting is looking at the actual data we have.

Lets start with some basic facts about alerts in the last 12 months:

As you can see this is not a casual hobby, it is a real system helping out in fixing and understanding hundreds of performance issues.

We generate alerts on a variety of branches, here is the breakdown of branches and alerts/branch;

number of regression alerts we have received per branch

number of regression alerts we have received per branch

There are a few things to keep in mind here, mobile/mozilla-central/Firefox are the same branch, and for non-pgo branches that is only linux/windows/android, not osx. 

Looking at that graph is sort of non inspiring, most of the alerts will land on fx-team and mozilla-inbound, then show up on the other branches as we merge code.  We run more tests/platforms and land/backout stuff more frequently on mozilla-inbound and fx-team, this is why we have a larger number of alerts.

Given the fact we have so many alerts and have manually triaged them, what state the the alerts end up in?

Current state of alerts

Current state of alerts

The interesting data point here is that 43% of our alerts are duplicates.  A few reasons for this:

The last piece of information that I would like to share is the break down of alerts per test:

Alerts per test

number of alerts per test (some excluded)

There are a few outliers, but we need to keep in mind that active work was being done in certain areas which would explain a lot of alerts for a given test.  There are 35 different test types which wouldn’t look good in an image, so I have excluded retired tests, counters, startup tests, and android tests.

Personally, I am looking forward to the next year as we transition some tools and do some hacking on the reporting, alert generation and overall process.  Thanks for reading!

May 15, 2015 12:40 PM

May 12, 2015

Joel Maher

community hacking – thoughts on what works for the automation and tools team

Community is a word that means a lot of things to different people.  When there is talk of community at an A*Team meeting, some people perk up and others tune out.  Taking a voluntary role in leading many community efforts on the A*Team over the last year, here are some thoughts I have towards accepting contributions, growing community, and making it work within the team.


Historically on the A*Team we would file bugs which are mentored (and discoverable via bugsahoy) and blog/advertise help wanted.  This is always met with great enthusiasm from a lot of contributors.  What does this mean for the mentor?  There are a few common axis here:

We need to appreciate all types of contributions and ensure we do our best to encourage folks to participate.  As a mentor if you have a lot of high-touch, low-reward, short-term contributors, it is exhausting and de-moralizing.  No wonder a lot of people don’t want to participate in mentoring folks as they contribute.  It is also unrealistic to expect a bunch of seasoned coders to show up and implement all the great features, then repeat for years on end.

The question remains, how do you find low-touch contributors or identify ones that are high-touch at the start and end up learning fast (some of the best contributors fall into this latter category).

Growing Community:

The simple answer here is file a bunch of bugs.  In fact whenever we do this they get fixed real fast.  This turns into a problem when you have 8 new contributors, 2 mentors, and 10 good first bugs.  Of course it is possible to find more mentors, and it is possible to file more bugs.  In reality this doesn’t work well for most people and projects.

The real question to ask is what kind of growth are you looking for?  To answer this is different for many people.  What we find of value is slowly growing our long-term/low-touch contributors by giving them more responsibility (i.e. ownership) and really depending on them for input on projects.  There is also a need to grow mentors and mentors can be contributors as well!  Lastly it is great to have a larger pool of short-term contributors who have ramped up on a few projects and enjoy pitching in once in a while.

How can we foster a better environment for both mentors and contributors?  Here are a few key areas:

Just focusing on the relationships and what comes after the good first bugs will go a long way in retaining new contributors and reducing the time spent helping out.

How we make it work in the A*Team:

The A*Team is not perfect.  We have few mentors and community is not built into the way we work.  Some of this is circumstantial, but a lot of it is within our control.  What do we do and what does and does not work for us.

Once a month we meet to discuss what is going on within the community on our team.  We have tackled topics such as project documentation, bootcamp tools/docs, discoverability, good next bugs, good first projects, and prioritizing our projects for encouraging new contributors.

While that sounds good, it is the work of a few people.  There is a lot of negative history of contributors fixing one bug and taking off.  Much frustration is expressed around helping someone with basic pull requests and patch management, over and over again.  While we can document stuff all day long, the reality is new contributors won’t read the docs and still ask questions.

The good news is in the last year we have seen a much larger impact of contributors to our respective projects.  Many great ideas were introduced, problems were solved, and experiments were conducted- all by the growing pool of contributors who associate themselves with the A*Team!

Recently, we discussed the most desirable attributes of contributors in trying to think about the problem in a different way.  It boiled down to a couple things, willingness to learn, and sticking around at least for the medium term.

Going forward we are working on growing our mentor pool, and focusing on key projects so the high-touch and timely learning curve only happens in areas where we can spread the love between domain experts and folks just getting started.

Keep an eye out for most posts in the coming week(s) outlining some new projects and opportunities to get involved.

May 12, 2015 07:41 PM

Henrik Skupin

Firefox Automation report – Q1 2015

As you may have noticed I was not able to come up with status reports of the Firefox Automation team during the whole last quarter. I feel sad about it, but there was simply no time to keep up with those blog posts. Even now I’m not sure how often I will be able to blog. So maybe I will aim to do it at least once a quarter or if possible once a month.

You may ask how it comes? The answer is simple. Our team faced some changes and finally a massive loss of core members. Which means from the former 6 people only myself are remaining. Since end of February all 5 former team members from Softvision are no longer participating in any of the maintained projects. Thanks to all of them for the great help over all the last months and years! But every project we own is now on my own shoulders. And this is kinda hell of work with downsides like not being able to do as many reviews as I want for side projects. One positive thing at least was that I got pulled back into the A-Team at the same time. With that move I’m once more closer again to all the people who care about the basics of all test infrastructure at Mozilla. I feel back home.

So what have I done the whole last quarter… First, it was always the ongoing daily work for maintaining our Mozmill CI system. This was usually a job for a dedicated person all the last months. The amount of work can sometimes eat up a whole day. Especially if several regressions have been found or incompatible changes in Firefox have been landed. Seeing my deliverables for Q1 it was clear that we have to cut down the time to spent on those failures. As result we started to partially skip tests which were failing. There was no time to get any of those fixed. Happily the latest version of Mozmill is still working kinda nicely so no other work had to be dedicated for this project.

Most of my time during the last quarter I actually had to spent on Marionette, especially building up wrapper scripts for being able to use Marionette as test framework for Firefox Desktop. This was a kinda large change for us but totally important in terms of maintenance burden and sustainability. The code base of Mozmill is kinda outdated and features like Electrolysis (e10s) will totally break it. Given that a rewrite of the test framework is too cost-intensive the decision has been made to transition our Mozmill tests over to Marionette. Side-effect was that a lot of missing features had to be implemented in Marionette to bring it at a level as what Mozmill offers. Thanks for the amazing work goes to Andrew Halberstadt, David Burns, Jonathan Griffin, and especially Chris Manchester.

For the new UI driven tests for Firefox Desktop we created the firefox-ui-tests repository at Github. We decided on that name to make it clear to which product the tests belong to, and also to get rid of any relationship to the underling test framework name. This repository contains the harness extensions around Marionette, a separate puppeteer library for back-end and UI modules, and last but not least the tests themselves. As goal for Q1 we had to get the basics working including the full set of remote security tests, and most important the update tests. A lot of help on the security tests we got from Barbara Miller our intern from December to March. She did great amount of work here, and also assisted other community members in getting their code done. Finally we got all the security tests converted.

My own focus beside the harness pieces were the update tests. Given the complete refactoring of those Mozmill tests we were able to easily port them over to Marionette. We tried to keep the class structure as is, and only did enhancements where necessary. Here Bob Silverberg helped with two big chunks of work which I’m gladly thankful about! Thanks a lot! With all modules in-place I finally converted the update tests and got them running for each version of Firefox down to 38.0, which will be the next ESR release and kinda important to be tested with Marionette. For stability and ease of contribution we added support for Travis CI to our new repository. It helps us a lot with reviews of patches from community members, and they also can see immediately if changes they have done are working as expected.

The next big chunk of work will be to get those tests running in Mozmill CI (to be renamed) and the test reporting to use Treeherder. Also we want to get our update tests for Firefox releases executed by the RelEng system, to further reduce the amount of time for signoffs from QE. About this work I will talk more in my next blog post. So please stay tuned.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agendas, the video recordings, and notes from the Firefox Automation meetings. Please note that since end of February we no longer host a meeting due to the low attendance and other meetings like the A-team ones, where I have to report my status.

May 12, 2015 04:03 PM

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

May 12, 2015 01:26 PM

Joel Maher

Re-Triggering for a [root] cause – some notes from doing it

With all this talk of intermittent failures and folks coming up with ideas on how to solve them, I figured I should dive head first into looking at failures.  I have been working with a few folks on this, specifically :parkouss and :vaibhav1994.  This experiment (actually the second time doing so) is where I take a given intermittent failure bug and retrigger it.  If it reproduces, then I go back in history looking for where it becomes intermittent.  This weekend I wrote up some notes as I was trying to define what an intermittent is.

Lets outline the parameters first for this experiment:

Here are what comes out of this:

The next step was to look at each of the 25 bugs and see if it makes sense to do this.  In fact 13 of the bugs I decided not to take action on (remember this is an experiment, so my reasoning for ignoring these 13 could be biased):

This leaves us with 12 bugs to investigate.  The premise here is easy, find the first occurrence of the intermittent (branch, platform, testjob, revision) and re-trigger it 20 times (picked for simplicity).  When the results are in, see if we have reproduced it.  In fact, only 5 bugs reproduced the exact error in the bug when re-triggered 20 times on a specific job that showed the error.

Moving on, I started re-triggering jobs back in the pushlog to see where it was introduced.  I started off with going back 2/4/6 revisions, but got more aggressive as I didn’t see patterns.  Here is a summary of what the 5 bugs turned out like:

In summary, out of 356 bugs 2 root causes were found by re-triggering.  In terms of time invested into this, I have put about 6 hours of time to fine the root cause of the 5 bugs.

May 12, 2015 01:00 PM

May 11, 2015

Julien Pagès

mozregression-gui first release!

A year from now, I started to contribute to Mozilla. Mozregression (a regression range finder for Mozilla nightly and inbound builds) was one of the first projects that catched my attention: I made one patch, then another, and one again… I was mentored by the awesome William Lachance. Since then, I became one of the core contributors and a packager for mozregression and am now myself mentoring people for new contributions on this project!

At the time of the Google Summer of Code, William and I thought about adding a graphical interface for mozregression. Unfortunately, Mozilla was not accepted for gsoc 2015 – but still the idea was interesting – and we worked on it anyway.

And here we are! Ready for a first release of mozregression-gui! Please give it a try, and help us by reporting bugs and giving feedback. :) For now we provide binaries for Windows and Linux 64 (a Mac port is planned) but it is still possible to build from sources for each platform.

Much work remains to be done on the GUI, and this is a good place to start contributing to Mozilla – so if you’d like to hack with Python and Qt don’t wait and get involved!mozregui

May 11, 2015 06:44 PM

mozregression updates

Mozregression GUI is out!

I am proud to announce the first release of mozregression-gui, a graphical interface for mozregression!

This is a pre-release alpha version, provided so you can help us to find bugs and give us some feedback.

Give it a try!

May 11, 2015 12:00 AM

May 10, 2015

Joel Maher

intermittent oranges- missing a real definition

There are all kinds of great ideas folks have for fixing intermittent issues.  In fact each idea in and of itself is a worthwhile endeavor.   I have spent some time over the last couple of months fixing them, filing bugs on them, and really discussing them.  One question that remains- what is the definition of an intermittent.

I don’t plan to lay out a definition, instead I plan to ask some questions and lay out some parameters.  According to orange factor, there are 4640 failures in the last week (May 3 -> May 10) all within 514 unique bugs.  These are all failures that the sheriffs have done some kind of manual work on to star on treeherder.  I am not sure anybody can find a way to paint a pretty picture to make it appear we don’t have intermittent failures.

Looking at a few bugs, there are many reasons for intermittent failures:

There are a lot of reasons, many of these have nothing to do with poor test cases or bad code in Firefox.  But many of these are showing up many times a day and as a developer who wants to fix a bad test, many are not really actionable.  Do we need to have some part of a definition to include something that is actionable?

Looking at the history of ‘intermittent-failure’ bugs in Bugzilla, many occur once and never occur again.  In fact this is the case for over half of the bugs filed (we file upwards of 100 new bugs/week).  While there are probably reasons for a given test case to fail, if it failed in August 2014 and has never failed again, is that test case intermittent?  As a developer could you really do anything about this given the fact that reproducing it is virtually impossible?

This is where I start to realize we need to find a way to identify real intermittent bugs/tests and not clutter the statistics with tests which are virtually impossible to reproduce.  Thinking back to what is actionable- I have found that while filing bugs for Talos regressions the closer the bug is filed to the original patch landing, the better the chance it will get fixed.  Adding to that point, we only keep 30 days of builds/test packages around for our CI automation.  I really think a definition of an intermittent needs to have some kind of concept of time.  Should we ignore intermittent failures which occur only once in 90 days?  Maybe ignore ones that don’t reproduce after 1000 iterations?  Some could argue that we look in a smaller or larger window of time/iterations.

Lastly, when looking into specific bugs, I find many times they are already fixed.  Many of the intermittent failures are actually fixed!  Do we track how many get fixed?  How many have patches and have debugging already taking place?  For example in the last 28 days, we have filed 417 intermittents, of which 55 are already resolved and of the remaining 362 only 25 have occurred >=20 times.  Of these 25 bugs, 4 already have patches.  It appears a lot of work is done to fix intermittent failures which are actionable.  Are the ones which are not being fixed not actionable?  Are they in a component where all the developers are busy and heads down?

In a perfect world a failure would never occur, all tests would be green, and all users would use Firefox.  In reality we have to deal with thousands of failures every week, most of which never happen again.  This quarter I would like to see many folks get involved in discussions and determine:

Thanks for reading, I look forward to hearing from many who have ideas on this subject.  Stay tuned for an upcoming blog post about re-trigging intermittent failures to find the root cause.

May 10, 2015 09:49 PM

May 08, 2015

Armen Zambrano G. (@armenzg)

"Thank you!"

This week I had a co-worker thank me again for a project I worked on at the end of 2014.
It touched my heart to hear it so I encourage you to do the same if someone has enabled you recently.

I have failed again and again to use the power of these two powerful words and I hope to use them more often.

Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

May 08, 2015 10:20 PM

mozci 0.5.0 released - Store password in keyring, prevent corrupted data, progress bar and many small improvements

In this release we have many small improvements that help with issues we have found.

The main improvement is that we now don't store credentials in plain-text (sorry!) but use keyring to store it encrypted.

We also prevent partially downloading any data (corrupted data) and added progress bar to downloads.

Congrats to @chmanchester as our latest contributor!
Our usual and very appreciated contributions are by @adusca @jmaher and @vaibhavmagarwal

Minor improvements:
  • Lots of test changes and increased coverage
  • Do not use the root logger but a mozci logger
  • Allow passing custom files to a triggered job
  • Work around buildbot status corruptions (Issue 167)
  • Allow passing buildernames with lower case and removing trailing spaces (since we sometimes copy/paste from TH)
  • Added support to use build a buildername based on trychooser syntax
  • Allow passing extra properties when scheduling a job on Buildbot
You can see all changes in here:

Link to official release notes.

Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

May 08, 2015 10:15 PM

May 05, 2015

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

May 05, 2015 05:52 AM

April 28, 2015

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

April 28, 2015 06:21 AM

April 27, 2015

Armen Zambrano G. (@armenzg)

mozci hackday - Friday May 1st, 2015

I recently blogged about mozci and I was gladly surprised that people have curiosity about it.

I want to spend Friday fixing some issues on the tool and I wonder if you would like to join me to learn more about it and help me fix some of them.

I will be available as armenzg_mozci from 9 to 5pm EDT on IRC (#ateam channel).
I'm happy to jump on Vidyo to give you a hand understanding mozci.

I hand picked some issues that I could get a hand with.
Documentation and definition of the project in readthedocs.

Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

April 27, 2015 05:30 PM

April 24, 2015

Armen Zambrano G. (@armenzg)

What Mozilla CI tools is and what it can do for you (aka mozci)

Mozci (Mozilla CI tools) is a python library, scripts and package which allows you to trigger jobs on
Not all jobs can be triggered but those that are run on Release Engineering's Buildbot setup. Most (if not all) Firefox desktop and Firefox for Android jobs can be triggered. I believe some B2G jobs can still be triggered.

NOTE: Most B2G jobs are not supported yet since they run on TaskCluster. Support for it will be given on this quarter.

Using it

Once you check out the code:
git clone
python develop
you can run scripts like this one (click here for other scripts):
python scripts/ \
  --buildername "Rev5 MacOSX Yosemite 10.10 fx-team talos dromaeojs" \
  --rev e16054134e12 --times 10
which would trigger a specific job 10 times.

NOTE: This is independent if a build job exist to trigger the test job. mozci will trigger everything which is required to get you what you need.

One of the many other options is if you want to trigger the same job for the last X revisions, this would require you to use --back-revisions X.

There are many use cases and options listed in here.

A use case for developers

One use case which could be useful to developers (thanks @mike_conley!) is if you pushed to try and used this try syntax: "try: -b o -p win32 -u mochitests -t none". Unfortunately, you later determine that you really need this one: "try: -b o -p linux64,macosx64,win32 -u reftest,mochitests -t none".

In normal circumstances you would go and push again to the try server, however, with mozci (once someone implements this), we could simply pass the new syntax to a script (or with ./mach) and trigger everything that you need rather than having to push again and waster resources and your time!

If you have other use cases, please file an issue in here.

If you want to read about the definition of the project, vision, use cases or FAQ please visit the documentation.

Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

April 24, 2015 08:23 PM

Syd Polk


In order to run our multi-machine tests, we are going to need some kind of process runner which can execute remote-control commands on the test machines. Our solution uses a technology called Steeplechase (, which has a simple command language. It uses another technology called signalingserver ( This in turns relies on node.js (, which allows a server to run javascript without having to have a browser.

So, first created another Linux VM to run these pieces in. It won't need as many cores nor as much RAM as the Jenkins machine, but it will potentially need some more disk space for logs or binaries. So, the specs on this machine:
 Now installed signalling server and its dependencies. I ran the following:

sudo apt-get install git nodejs
npm install

Now, I ran this to get the signaling server source:

git clone

Finally, I started it up:

cd simplesignalling
nodejs ./server.js

Now, for steeplechase, we need the following dependencies:

sudo apt-get install python-setuptools

Clone the steeplechase repo:
git clone

We need to bootstrap the python environment for steeplechase.
cd steeplechase
sudo python install

The machine is now setup and ready to run steeplechase, but without test clients, there is nothing for steeplechase to talk to. Next installment, setting up the test clients.

April 24, 2015 07:31 PM


So, we are hiring in my group. Here is the link to the job posting:

We are looking for people with automation skills. What does that mean, you might ask? Well, here is what I think.

First of all, automation engineers are developers. The have good skills at developing automation software. At Mozilla, that means that they know Python, JavaScript, C, or C++. They know how to code and debug.

However, they are also QA engineers. They have a desire to break software to make it better. They have a desire to have some level of comfort in measurable quality. They are the kind of person who breaks websites trying to do normal things like buy a shirt or login to their bank. And they always want to know why it breaks, and may try to figure it out because either they are angry, they are curious, or they enjoy the thrill of a problem solved.

These kinds of people are very hard to find. Most people with coding skills want to work on some other kind of software than automation. Most really good testers don't necessarily have the technical skills to write automation.

I am plowing through a bunch of resumes this afternoon. Most of them are from candidates who are submitting their resume to every position that they can find regardless of qualifications. While it is true that there are many fewer jobs in tech than there are people who want them, it is also true that the number of qualified candidates is very small. There is no such thing as a candidate that matches all of the job requirements perfectly. My philosophy is looking for somebody who is smart, has a track record of success, knows some of your skills, has knowledge in similar skills, and has a track record of learning. Add on top of that some requirement of social skills, and you have a good candidate.

It's a large job standing in front of the firehose and screening resumes. It's fun, but at the end of the day, I am glad it is my bosses position and I am only helping out.

April 24, 2015 07:31 PM

Resume screening burnout

While taking care of getting cars fixed in anticipation of summer road trips, I spent two days screening resumes. I am burned out!

So, going back to building my lab tomorrow.

April 24, 2015 07:29 PM

Back to lab building

Doing two things at once:

- Starting to pay with an ESX server a coworker set up.
- Continuing building out a home lab.

So, ESX requires Virtual Center to actually build machines. You download Virtual Center with your browser and install it.

Oh, yeah. It requires Windows. Sigh.

So I decided to make a VM for it. Making a Windows 8 VM is a pain because you have to do registry tricks so as not to be presented with swipe panels you can't dismiss. So I decided to make a Windows 7 VM. Straightforward. Except that now I am waiting for 145 Windows Updates to downloads....

Meanwhile, back to building my lab. So, I need to build clients now. Time for two more Ubuntu VMs. Running out of memory, so these will be 2 core/2 GB machines (we'll try that).

And, of course, Linux has its own set of updates....

And the Macs are now wanting to update...

Lots of waiting at times when building labs. More later.

April 24, 2015 07:29 PM

Networking home lab setup

Things work much better when you have static IP addresses. This means that your IP addresses won't change when you restart your VMs, which means any command lines you use will remain valid. VMWare Fusion does not make assigning IP addresses easy, but it can certainly be done.

When you created your VMs, they were assigned generated MAC addresses by Fusion. Need to retrieve those here:

Virutal Machine -> Network Adapter -> Network Adapter Settings... Turn down the "Advanced Options" disclosure triangle...

Once you have the Mac addresses for your VMs, you can change your config. One source I drew heavily on is here; you have to restart your network services if VMWare is running. And you have to be careful; Fusion loves to blow away your changes.

On my machine, here is the mapping:

# Configuration file for ISC 2.0 vmnet-dhcpd operating on vmnet8.
# This file was automatically generated by the VMware configuration program.
# See Instructions below if you want to modify it.
# We set domain-name-servers to make some DHCP clients happy
# (dhclient as configured in SuSE, TurboLinux, etc.).
# We also supply a domain name to make pump (Red Hat 6.x) happy.

###### VMNET DHCP Configuration. Start of "DO NOT MODIFY SECTION" #####
# Modification Instructions: This section of the configuration file contains
# information generated by the configuration program. Do not modify this
# section.
# You are free to modify everything else. Also, this section must start
# on a new line
# This file will get backed up with a different name in the same directory
# if this section is edited and you try to configure DHCP again.

# Written at: 05/14/2014 16:13:27
allow unknown-clients;
default-lease-time 1800; # default is 30 minutes
max-lease-time 7200; # default is 2 hours

subnet netmask {
option broadcast-address;
option domain-name-servers;
option domain-name localdomain;
default-lease-time 1800; # default is 30 minutes
max-lease-time 7200; # default is 2 hours
option netbios-name-servers;
option routers;
host vmnet8 {
hardware ethernet 00:50:56:C0:00:08;
option domain-name-servers;
option domain-name "";
option routers;
####### VMNET DHCP Configuration. End of "DO NOT MODIFY SECTION" #######

host jenkins {
hardware ethernet 00:0c:29:ff:39:df;

host steeplechase {
hardware ethernet 00:0c:29:5a:8e:75;

host linux64-negatus-01 {
hardware ethernet 00:0c:29:b7:fe:99;

host linux64-negatus-02 {
hardware ethernet 00:0c:29:69:b0:0f;

Restart the vmware network.

sudo /Applications/VMware\ --configure
sudo /Applications/VMware\ --stop
sudo /Applications/VMware\ --start

This article also discusses this issue.

And we can see that all four machines are up and running by pinging them from Terminal on the same machine (they are not visible outside of the Mac):

sydpolkzillambp:~ spolk$ ping -c 1
PING ( 56 data bytes
64 bytes from icmp_seq=0 ttl=64 time=0.355 ms

--- ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.355/0.355/0.355/0.000 ms
sydpolkzillambp:~ spolk$ ping -c 1
PING ( 56 data bytes
64 bytes from icmp_seq=0 ttl=64 time=0.263 ms

--- ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.263/0.263/0.263/0.000 ms
sydpolkzillambp:~ spolk$ ping -c 1
PING ( 56 data bytes
64 bytes from icmp_seq=0 ttl=64 time=0.338 ms

--- ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.338/0.338/0.338/0.000 ms
sydpolkzillambp:~ spolk$ ping -c 1
PING ( 56 data bytes
64 bytes from icmp_seq=0 ttl=64 time=0.331 ms

--- ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.331/0.331/0.331/0.000 ms
sydpolkzillambp:~ spolk$

OK, so next we'll put it all together.

April 24, 2015 07:28 PM

Virtual Center

The ESX server I am installing onto does not have any OS IOS images to use to make VMs. So I need to have them locally. However, I'm in Austin, and the ESX server is in Mt. View. This is not going to work. Somebody at Moz HQ is going to get up a machine I can login via Remote Desktop Connection so I can continue.

April 24, 2015 07:28 PM

Home lab

Now, it's time to try everything out. The steeplechase machine needs to have firefox binaries and test artifacts. So, let's go get them.

The nightly build artifacts are here. On the steeplechase machine, we need to download both firefox-33.0a1.en-US.linux-x86_64.tar.bz2 and We will then unpack them appropriately:

mkdir firefox-releases
cd firefox-releases
mv ~/Downloads/firefox* .
tar xvfz firefox*.tab.bz2
mkdir tests
cd tests
unzip ../firefox*.zip

We have to have node running on this machine:

mozilla@jenkins-steeplechase:~$ cd simplesignalling/
mozilla@jenkins-steeplechase:~/simplesignalling$ ls
package.json server.js
mozilla@jenkins-steeplechase:~/simplesignalling$ nodejs server.js

We need to start the agent on the two Negatus machines:

mozilla@ubuntu:~/src$ cd Negatus/
mozilla@ubuntu:~/src/Negatus$ git pull
Already up-to-date.
mozilla@ubuntu:~/src/Negatus$ ./agent
Command handler listening on
Heartbeat handler listening on
Query url: IPADDR=
No SUTAgent.ini data.
No reboot callback data.

Mmm. It looks like I forgot a step. Running server.js should have output something. Looking back on our internal notes, I needed to run this:
npm install

If you do that from the simplesignalling directory, it will fail kind of like:
npm ERR! Error: Invalid version: "0.1"
npm ERR! at Object.module.exports.fixVersionField (/usr/lib/nodejs/normalize-package-data/lib/fixer.js:178:13)
npm ERR! at /usr/lib/nodejs/normalize-package-data/lib/normalize.js:29:38
npm ERR! at Array.forEach (native)
npm ERR! at normalize (/usr/lib/nodejs/normalize-package-data/lib/normalize.js:28:15)

Once you install this correctly, then server.js will output something correctly:
mozilla@jenkins-steeplechase:~/simplesignalling$ nodejs server.js 
info - started

Now, we are ready to try to run steeplechase.
mozilla@jenkins-steeplechase:~/steeplechase$ python `pwd`/steeplechase/ --binary /home/mozilla/firefox-releases/firefox/firefox --specialpowers-path /home/mozilla/firefox-releases/tests/steeplechase/specialpowers --prefs-file /home/mozilla/firefox-releases/tests/steeplechase/prefs_general.js --signalling-server '' --html-manifest /home/mozilla/firefox-releases/tests/steeplechase/tests/steeplechase.ini --host1 --host2
steeplechase INFO | Pushing app to Client 1...
steeplechase INFO | Pushing app to Client 2...
Writing profile for Client 1...
Pushing profile to Client 1...
cmd: ['/tmp/tests/steeplechase/app/firefox', '-no-remote', '-profile', '/tmp/tests/steeplechase/profile', '']
Writing profile for Client 2...
Pushing profile to Client 2...
cmd: ['/tmp/tests/steeplechase/app/firefox', '-no-remote', '-profile', '/tmp/tests/steeplechase/profile', '']
steeplechase INFO | Waiting for results...
steeplechase INFO | All clients finished
steeplechase INFO | Result summary:
steeplechase INFO | Passed: 112
steeplechase INFO | Failed: 0

It worked! We have a running lab on Linux now.

Next step: Get Jenkins to invoke this.

April 24, 2015 07:28 PM

ESX, remotely

OK, somebody gave me VNC to a linux box in the office over VPN, so I can download ISOs. MSDN's sight has a really really bad captcha on it, and using a password manager was a pain, and it did 2-level, but I'm safe now, right?

Anyway, I can now download ISOs onto a machine on the same network as ESX box. Need to figure out how to get ESX to mount the newly created NFS export where the ISO lives.

OK, I figured it out. You create the VM in vSphereClient. Let the boot fail. Then, connect the .iso stored on your data store to the DVD drive. Finally, press Ctrl-Alt-Ins, and it should boot to your ISO.

More later.

April 24, 2015 07:28 PM

Bootstrapping a data center remotely

Using a VM running vSphere connecting to an ESX server 2000 miles away over VPN, and using VNC to connect to a known Linux box on the same network as the ESX box, I was able to download enough ISOs to create a Windows 7 VM. The idea is to put vSphere Client on it, and use Microsoft Remote Desktop to connect to it. I can then download ISOs to its hard drive and create VMs with "ISO on local disk" option.

Had to download MS RDC. It's free now. This is great.

I am also starting to build up the first linux box. I used VNC on the other machine to download the ISO of Ubuntu 14.10 Desktop. Used the VSphere Client to install the machine. Of course, I had to use the vSphere Client to access the Desktop. Doing this from RDC is painful; typing often results in duplicated characters. So I installed openssh-server:

sudo apt-get install openssh-server
sudo /etc/init.d/ssh restart

And then I could ssh in from a Terminal on my machine. Ah, much better.

I still want desktop access, though. I enabled Desktop Sharing on the Linux VM. Alas, the Mac Screen Sharing could not connect to it, although it could connect to an Ubuntu 12 machine. My coworker point me to this article, and now the Linux machine is good to go.

April 24, 2015 07:28 PM

Running the tests

One of my coworkers is going to post a public document on how to get steeplechase and Negatus running. Once he posts that, I will repost the link here. Basically, it looks like this:

- One machine needs to run simplesignaling. This is a nodejs-based server to facilitate Firefox communication.
- A machine needs to run steelpechase. This can be the same machine as the one that runs simplesignalling, but not required.
- Each of the client machines runs Negatus, which is a test agent.

The steeplechase machine needs to download the firefox binaries and tests from The binaries and test files have to be de-archived. Steeplechase can then be run:

% tar xvfj firefox-33.0a1.en-US.linux-x86_64.tar.bz2 
% mkdir tests
% cd tests/
% unzip ../
% mkdir ~/logs
% python ~/src/steeplechase/steeplechase/ --binary /home/mozilla/firefox-releases/firefox/firefox --specialpowers-path /home/mozilla/firefox-releases/tests/steeplechase/specialpowers --prefs-file /home/mozilla/firefox-releases/tests/steeplechase/prefs_general.js --signalling-server '' --html-manifest /home/mozilla/firefox-releases/tests/steeplechase/tests/steeplechase.ini --save-logs-to ~/logs/ --host1 --host2
steeplechase INFO | Pushing app to Client 1...
steeplechase INFO | Pushing app to Client 2...
Writing profile for Client 1...
Pushing profile to Client 1...
cmd: ['/tmp/tests/steeplechase/app/firefox', '-no-remote', '-profile', '/tmp/tests/steeplechase/profile', '']
Writing profile for Client 2...
Pushing profile to Client 2...
cmd: ['/tmp/tests/steeplechase/app/firefox', '-no-remote', '-profile', '/tmp/tests/steeplechase/profile', '']
steeplechase INFO | Waiting for results...
steeplechase INFO | All clients finished
steeplechase INFO | Result summary:
steeplechase INFO | Passed: 118
steeplechase INFO | Failed: 0

I now have this working on both my lab at home and the ESX lab in Mountain View. Next: making it work autonomously.

April 24, 2015 07:27 PM

OK, time to step back

Occasionally, you reach some little milestone, and it helps to make a To Do list. So, here is what is left to have this lab up and running now that I have 3 machines that can run tests.

1. Get Negatus to run on boot for the two client machines.
2. On my home lab, develop the Jenkins scripts and job templates that will do the work.
3. Install the Jenkins instance in the ESX lab.
4. Port the Jenkins work from the home lab to the ESX lab.
5. Start adding platforms.

That's a good general list.

Tomorrow is a test day, so I probably won't be doing my with this stuff. Let's what I can get done today.

April 24, 2015 07:27 PM

New task

I was given a new area to investigate on top of building out a lab. Basically, we have some media streaming tests in our tree, in <mozilla-central>/content/media/test, and they are flaky. I have been asked to investigate a couple of things, using test_seek.html as an example:

- The tests actually get run in parallel. I have been asked to see if running them singly will affect the intermediate failure rate.
- I have been asked to split this file up. It currently calls 13 sub-files. I have already generated the first of those 13 files and it works fine.

All of this requires checking out the Firefox source (instructions here), and then running Mochitest (instructions here). I can run the individual test.

If do add a test file in a directory, such as <mozilla-central>/content/media/test, you have to add it to the mochitest.ini file in that directory to be picked up by the system.

I have two build trees (my Mercurial foo is low, and I am lazy), one to generate patches for the split out of the tests, and one to generate patches for our try system running only one test at a time.

Alas, I am not at my house; I am with family in another state, and the internet is slow here. It will be upgraded soon, but this is taking a while...

April 24, 2015 07:27 PM

[LAB] Setting up for jenkins

Packages required on Jenkins machine:

Packages required on Steeplechase machine:
  • openjdk-7-jre-headless
  • curl
Jenkins plugins:
  • Git plugin
  • Mercurial plugin
I am afraid I finished the buildout of this without good notes. Sorry!

April 24, 2015 07:27 PM

Three things I am working on

The first one is this bugzilla. Basically, the audio/video tests sometimes time out in Mozilla's build/test environment, and we are trying to track down which tests are sensitive. We run a lot of tests in Amazon's S3 cloud, and disk and network access are not predictable in that environment.

My boss and I have submitted a few patches to unparallelize some of the tests to see if it helps. Latest patch is run by our environment in tbpl, here. I need to run quite a few more tests and analyze them today. One thing I had to learn was Mercurial queues; using this is the best way to work with patches with Firefox and Bugzilla. More info here and here. You have to be really careful with them, as it is easy to blow away work. Still, it's a really nice system for managing patches. In git, you would do the same with local branches, but it's not quite as easy.

The second is a test time out running mochitest on a subdirectory on a Mac OS X VM. Mochitest is the oldest Firefox test suite. More info on it can be found here. I chose to run it from a build tree, so I had to go build Firefox on Mac. Info on that here. I then ran the following:

./mach mochitest-plain content/media/test

And then watched the magic! Note that this is the same test suite as we are watching with the first problem I am working on. Sometimes, I get a test failure on my VM that nobody else seems to be running into, so I am working on trying to get enough data to file a bug. A coworker directed me to try out setting MINIDUMP_STACKWALK before running the test. Once I dug enough for somebody to tell me that this tools was part of a another mercurial repo (, I tried it. No dice. Need to submit that bug report today.

And last, I am having trouble getting steeplechase to run in my Jenkins lab. The latest:

cmd: ['/tmp/tests/steeplechase/app/firefox', '-no-remote', '-profile', '/tmp/tests/steeplechase/profile', '']
Traceback (most recent call last):
File "/home/mozilla/jenkins/workspace/linux64-linux64/steeplechase/steeplechase/", line 311, in
sys.exit(0 if main(sys.argv[1:]) else 1)
File "/home/mozilla/jenkins/workspace/linux64-linux64/steeplechase/steeplechase/", line 301, in main
html_pass_count, html_fail_count =
File "/home/mozilla/jenkins/workspace/linux64-linux64/steeplechase/steeplechase/", line 187, in run
passes, failures = result
TypeError: 'NoneType' object is not iterable
Exception in thread Client 1:
Traceback (most recent call last):
File "/usr/lib/python2.7/", line 810, in __bootstrap_inner
File "/home/mozilla/jenkins/workspace/linux64-linux64/steeplechase/steeplechase/", line 100, in run
output = dm.shellCheckOutput(cmd, env=env)
File "/usr/local/lib/python2.7/dist-packages/mozdevice-0.37-py2.7.egg/mozdevice/", line 395, in shellCheckOutput
raise DMError("Non-zero return code for command: %s (output: '%s', retval: '%s')" % (cmd, output, retval))
DMError: Non-zero return code for command: ['/tmp/tests/steeplechase/app/firefox', '-no-remote', '-profile', '/tmp/tests/steeplechase/profile', ''] (output: 'r', retval: '256')

Exception in thread Client 2:
Traceback (most recent call last):
File "/usr/lib/python2.7/", line 810, in __bootstrap_inner
File "/home/mozilla/jenkins/workspace/linux64-linux64/steeplechase/steeplechase/", line 100, in run
output = dm.shellCheckOutput(cmd, env=env)
File "/usr/local/lib/python2.7/dist-packages/mozdevice-0.37-py2.7.egg/mozdevice/", line 395, in shellCheckOutput
raise DMError("Non-zero return code for command: %s (output: '%s', retval: '%s')" % (cmd, output, retval))
DMError: Non-zero return code for command: ['/tmp/tests/steeplechase/app/firefox', '-no-remote', '-profile', '/tmp/tests/steeplechase/profile', ''] (output: 'r', retval: '256')

Need to see what that is about...

April 24, 2015 07:26 PM

Three things I am working on - follow-up

Thing 1 - Bugzilla 1036992 - I have a patch now which completes the split up of test_seek.html, including refactoring of common javascript in test_seek-split*.html files. Figuring out how to add the javascript so that it was actually available was fun. Basically, everything goes in mochitest.ini. Who new?

Thing 2 - mochitest dying on my VM. Well, it still happens, and it often does not leave a stack dump. Still working on this one.

Thing 3 - Jenkins and nodejs. I restarted everything and it started working. Huh. Weird.

So, now I am setting up Jenkins in Mountain View to reproduce the result. Unfortunately, when the physical box was moved into the lab, the networking broke. All of the existing VMs have IPv6 addresses rather than the static addresses I thought were assigned.

Coworker in charge of box said he would look at that.

So, now I am building linux32 builders for this test on my home VM. I am working out of one of my relative's houses in the Midwest right now, and her internet connection was SLOW. Well, the upgrade came through yesterday, and now it is as fast as my home network in Texas.

It helps.

April 24, 2015 07:26 PM


You actually have to do work to run 32-bit binaries on 64-bit linux. I knew this, but I rediscovered this fact this morning.

Of course, this got much harder in modern Ubuntu:

sudo apt-get install libxtst6:i386 libXext6:i386 libxi6:i386 libncurses5:i386 libxt6:i386 libxpm4:i386 libxmu6:i386 libxp6:i386
sudo apt-get install libstdc++6-4.8-dbg:i386


April 24, 2015 07:26 PM

OK, it's just too hard to nail down all of the libraries to run firefox32 on linux64. I give up. Average users won't do this.

April 24, 2015 07:26 PM

Another fun fact: For Negatus to be able to run firefox for the tests, it has to run it in a display. Without setting up a fake X session, the Negatus client has to be logged in, at least on Linux. I am sure that this will be true on other platforms as well. For Linux, we could run in a virtual frame buffer, but I am not sure that this is necessary. Just set up the account the test will run in to auto login. Make sure it is on a private network, though...

April 24, 2015 07:25 PM


I have a Jenkins instance running the regression suite on Firefox nightly builds in all platform combinations of linux64, linux32, and macosx. The front page looks like this:

So, I have proof-of-concept. I am demoing this at our QA Work Week QA Fair later today. Should be fun!

Lots of work to do, however.

  1. I need to investigate using parameterized builds. Having to create and maintain a separate Jenkins job for each combination is painful (I have a lot of experience with this from Coverity, alas).
  2. Henrick Skupin has a Jenkins instance, and he has solved the problem of the nightly version numbers changing every month. Need to implement that.
  3. Need to add jobs for nightly versions connecting to Aurora, Beta, Release and ERS versions of Firefox.
  4. The test is not incredibly valid. I need to take the test additions done by Geo Mealer in our Sunny Day Environment and run the connections for 1 minute each. I really want to do the parameterization first so I don't have to update dozens of jobs.
  5. I need to put this in our ESX farm. However, I only want to do this when most of the above is done. I am going to set up Jenkins on the ESX farm, running linux64 regressions for now.
  6. I have got to get Jenkins and Steeplechase working for Windows. While I am in Mountain View this week, I do not have access to my Windows VMs. Will have to wait until I get home next week.
  7. Sunny Day Environment does most of its job maintenance and triggering via cron jobs. It would be nice to move this into Jenkins.
  8. Need to report results of test runs into Treeherder. This is a pretty big effort.
  9. And then, there is B2G and Android.
  10. Steeplechase changes
    1. Run with existing binaries but new profile. (or whatever; independent control)
    2. Send binary archives down instead of directories, and tell clients how to unpack them.
    3. Steeplechase should talk to treeherder.
Ought to keep me busy.

April 24, 2015 07:25 PM

Which Firefox build to download?

So, eventually this system is going to have to download all Firefox releases and test them. All of the releases are available here:

So, there are a LOT of releases on this server. So which ones to we want? Mozilla has five (or more) active release trains at any one point:

There is a utility called mozdownload which is here. Once you build and install it, mozdownload is a big help. It deals with changing file names and dates and the like.

So where is all of this stuff on And what is the mozdownload command-line for it?

I hope that this helps. It sure helps me.

April 24, 2015 07:25 PM

Back to Jenkins fun

So, now, I know how to download binaries and get the correct versions independent of what their names actually are.

Now, I need my jobs to trigger each other. The scheme I have is:

  1. Run download. If the binary is newer, overwrite the canonically named version, i.e., firefox-latest-nightly.en-US.linux-x86_64.tar.bz2.
  2. Another job triggers when firefox-latest-nightly.en-US.linux-x86_64.tar.bz2. It runs on the Steeplechase machine, and it expands the archive into a known directory location.
  3. It then triggers jobs for all of the steeplechase runs that depend on it.
At Coverity I used the URLSCM plugin to do the triggering. Basically it used the SCM polling mechanism builtin to Jenkins to see if the local copy of a file is newer than the version at a URL. The problem is, this mechanism broke a few years ago, and to this day nobody has fixed the bug.

Today I found out about another Jenkins plugin, URLTrigger Plugin. This allows you to trigger builds of a variety of things, but most germain to me is that it can trigger if md5 checksums are different. I am trying this out overnight; we'll see what happens.

April 24, 2015 07:25 PM