Planet Mozilla Automation

October 06, 2015

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

October 06, 2015 07:28 AM

October 02, 2015

Joel Maher

Hacking on a defined length contribution program

Contribution takes many forms where each person has different reasons to contribute or help people contribute.  One problem we saw a need to fix was when a new contributor came to Mozilla and picked up a “good first bug”, then upon completion was left not knowing what to do next and picking up other random bugs.  The essential problem is that we had no clear path defined for someone to start making more substantial improvements to our projects.   This can easily lead toward no clear mentorship as well as a lot of wasted time setting up and learning new things.  In response to this, we decided to launch the Summer of Contribution program.

Back in May we announced two projects to pilot this new program: perfherder and developer experience.  In the announcement we asked that interested hackers commit to dedicating 5-10 hours/week for 8 weeks to one of these projects. In return, we would act as a dedicated mentor and do our best to ensure success.

I want to outline how the program was structured, what worked well, and what we want to do differently next time.

Program Structure

The program worked well enough, with some improvising, here is what we started with:

That was it, we improvised a little by doing:

What worked well

A lot worked very well, specifically advertising by blog post and newsgroup post and then setting the expectation of a longer contribution cycle rather than a couple weeks.  Both :wlach and myself have had a good history of onboarding contributors, and feel that being patient, responding quickly, communicating effectively and regularly, and treating contributors as team members goes a long way.  Onboarding is easier if you spend the time to create docs for setup (we have the ateam bootcamp).  Without mentors being ready to onboard, there is no chance for making a program like this work.

Setting aside a pile of bugs to work on was successful.  The first contribution is hard as there is so much time required for setup, so many tools and terms to get familiar with, and a lot of process to learn.  After the first bug is completed, what comes next?  Assuming it was enjoyable, one of two paths usually take place:

Both of these are OK models, but there is a trap where you could end up with a bug that is hard to fix, not well defined, outdated/irrelevant, or requires a lot of new learning/setup.  This trap is something to avoid where we can build on the experience of the first bug and work on the same feature but on a bug that is a bit more challenging.

A few more thoughts on the predefined set of bugs to get started:

  • These should not be easily discoverable as “good first bugs“, because we want people who are committed to this program to work on them, rather than people just looking for an easy way to get involved.
  • They should all have a tracking bug, tag, or other method for easily seeing the entire pool of bugs
  • All bugs should be important to have fixed, but they are not urgent- think about “we would like to fix this later this quarter or next quarter”.  If we do not have some form of urgency around getting the bugs fixed, our eagerness to help out in mentoring and reviewing will be low.  A lot of  times while working on a feature there are followup bugs, those are good candidates!
  • There should be an equal amount (5-10) of starter bugs, next bugs, and other bugs
  • Keep in mind this is a starter list, imagine 2-3 contributors hacking on this for a month, they will be able to complete them all.
  • This list can grow as the project continues

Another thing that worked is we tried to work in public channels (irc, bugzilla) as much as possible, instead of always private messaging or communicating by email. Also communicating to other team members and users of the tools that there are new team members for the next few months. This really helped the contributors see the value of the work they are doing while introducing them to a larger software team.

Blog posts were successful at communicating and helping keep things public while giving more exposure to the newer members on the team.  One thing I like to do is ensure a contributor has a Mozillians profile as well as links to other discoverable things (bugzilla id, irc nick, github id, twitter, etc.) and some information about why they are participating.  In addition to this, we also highlighted achievements in the fortnightly Engineering Productivity meeting and any other newsgroup postings we were doing.

Lastly I would like to point out a dedicated mentor was successful.  As a contributor it is not always comfortable to ask questions, or deal with reviews from a lot of new people.  Having someone to chat with every day you are hacking on the project is nice.  Being a mentor doesn’t mean reviewing every line of code, but it does mean checking in on contributors regularly, ensuring bugs are not stuck waiting for needinfo/reviews, and helping set expectations of how work is to be done.  In an ideal world after working on a project like this a contributor would continue on and try to work with a new mentor to grow their skills in working with others as well as different code bases.

What we can do differently next time?

A few small things are worth improving on for our next cycle, here is a few things we will plan on doing differently:

  • Advertising 4-5 weeks prior and having a defined start/end date (e.g. November 20th – January 15th)
  • Really limiting this to a specific number of contributors, ideally 2-3 per mentor.
  • Setting acceptance criteria up front.  This could be solving 2 easy bugs prior to the start date.
  • Posting an announcement welcoming the new team members, posting another announcement at the halfway mark, and posting a completion announcement highlighting the great work.
  • Setting up a weekly meeting schedule that includes status per person, great achievements, problems, and some kind of learning (guest speaker, Q&A, etc.).  This meeting should be unique per project.
  • Have a simple process for helping folks transition out of they have less time than they thought- this will happen, we need to account for it so the remaining contributors get the most out of the program.

In summary we found this to be a great experience and are looking to do another program in the near future.  We named this Summer of Contribution for our first time around, but that is limiting to when it can take place and doesn’t respect the fact that the southern hemisphere is experiencing Winter during that time.  With that :maja_zf suggested calling it Quarter of Contribution which we plan to announce our next iteration in the coming weeks!

October 02, 2015 06:18 PM

October 01, 2015

Jonathan Griffin

Engineering Productivity Update, Oct 1, 2015

We’ve said good-bye to Q3, and are moving on to Q4. Planning for Q4 goals and deliverables is well underway; I’ll post a link to the final versions next update.

Last week, a group of 8-10 people from Engineering Productivity gathered in Toronto to discuss approaches to several aspects of developer workflow. You can look at the notes we took; next up is articulating a formal Vision and Roadmap for 2016, which incorporates both this work as well as other planning which is ongoing separately for things like MozReview and Treeherder.


Bugzilla: Support for 2FA has been enhanced.


Perfherder and Performance Automation:

MozReview and Autoland: mcote posted a blog post detailing some of the rough edges in MozReview, and explaining how the team intends on tackling these. dminor blogged about the state of autoland; in short, we’re getting close to rolling out an initial implementation which will work similarly to the current “checkin-needed” mechanism, except, of course, it will be entirely automated. May you never have to worry about closed trees again!

Mobile Automation: gbrown made some additional improvements to mach commands on Android; bc has been busy with a lot of Autophone fixes and enhancements.

Firefox Automation: maja_zf has enabled MSE playback tests on trunk, running per-commit. They will go live at the next buildbot reconfig.

Developer Workflow: numerous enhancements have been made to |mach try|; see list below in the Details section.  run-by-dir has been applied to mochitest-plain on most platforms, and to mochitest-chrome-opt, by kaustabh93, one of team’s contributors. This reduces test bleedthrough, a source of intermittent failures, as well as improves our ability to change job chunking without breaking tests.

Build System: gps has improved test package generation, which results in significantly faster builds – a savings of about 5 minutes per build on OSX and Windows in automation; about 90s on linux.

TaskCluster Migration: linux64 debug builds are now running, so ahal is unblocked on getting linux64 debug tests running in TaskCluster.  armenzg has landed mozharness code to support running buildbot jobs via TaskCluster scheduling, via buildbot bridge.

The Details


Perfherder/Performance Testing

TaskCluster Support

Mobile Automation

Firefox and Media Automation

General Automation


WebDriver (highlights)




Firefox build system

October 01, 2015 10:50 PM

Dan Minor

Autoland to Inbound

We’re currently putting the finishing touches on Autoland to Inbound from MozReview. We have a relatively humble goal for the initial version of this: we’d like to replace the current manual “checkin-needed” process done by a sheriff. Basically, if a patch is reviewed and a try run has been done, the patch author can flag “checkin-needed” in Bugzilla and at some point in the future, a sheriff will land the changes.

In the near future, if a “ship-it” has been granted and a try run has been made, a button to land to Inbound will appear in the Automation menu in MozReview. At the moment, the main benefit of using Autoland is that it will retry landings in the event that the tree is closed. Enforcing a try run should cut down on tree closures, analysis of closure data in the past has shown a not insignificant number of patches which end up breaking the tree have had no try run performed.

A number of workflow improvements should be easy to implement once this is in place, for instance automatically closing MozReview requests to keep people’s incoming review requests sane and rewriting any r? in the commit summary to be r=.

October 01, 2015 07:15 AM

September 30, 2015

Mark Côté

Fixing MozReview's sore spots

MozReview was intentionally released early, with a fairly minimal feature set, and some ugly things bolted onto a packaged code-review tool. The code-review process at Mozilla hasn’t changed much since the project began—Splinter, a slightly fancier UI than dealing with raw diffs, notwithstanding. We knew this would be a controversial subject, with a variety of (invariably strong) opinions. But we also knew that we couldn’t make meaningful progress on a number of long-desired features, like autolanding commits and automatic code analysis, without moving to a modern repository-based review system. We also knew that, with the incredible popularity of GitHub, many developers expect a workflow that involves them pushing up commits for review in a rich web UI, not exporting discrete patches and looking at almost raw diffs.

Rather than spending a couple years off in isolation developing a polished system that might not hit our goals of both appealing to future Mozillians and increasing productivity overall, we released a very early product—the basic set of features required by a push-based, repository-centric code-review system. Ironically, perhaps, this has decreased the productivity of some people, since the new system is somewhat raw and most definitely a big change from the old. It’s our sincere belief that the pain currently experienced by some people, while definitely regrettable and in some cases unexpected, will be balanced, in the long run, by the opportunities to regularly correct our course and reach the goal of a world-class code review-and-submission system that much faster.

And so, as expected, we’ve received quite a bit of feedback. I’ve been noticing a pattern, which is great, because it gives us insight into classes of problems and needs. I’ve identified four categories, which interestingly correspond to levels of usage, from basic to advanced.

Understanding MozReview’s (and Review Board’s) models

Some users find MozReview very opaque. They aren’t sure what many of the buttons and widgets do, and, in general, are confused by the interface. This caught us a little off-guard but, in retrospect, is understandable. Review Board is a big change from Splinter and much more complex. I believe one of the sources of most confusion is the overall review model, with its various states, views, entry points, and exit points. Splinter has the concept of a review in progress, but it is a lot simpler.

We also had to add the concept of a series of related commits to Review Board, which on its own has essentially a patch-based model, similar to Splinter’s, that’s too limited to build on. The relationship between a parent review request and the individual “child” commits is the source of a lot of bewilderment.

Improving the overall user experience of performing a review is a top priority for the next quarter. I’ll explore the combination of the complexity of Review Board and the commit-series model we added in a follow-up post.

Inconveniences and lack of clarity around some features

For users who are generally satisfied by MozReview, at least, enough to use it without getting too frustrated, there are a number of paper cuts and limitations that can be worked around but generate some annoyance. This is an area we knew we were going to have to improve. We don’t yet have parity with Splinter/Bugzilla attachments, e.g. reviewers can’t delegate review requests, nor can they mark specific files as reviewed. There are other areas that we can go beyond Bugzilla, such as being able to land parts of a commit series (this is technically possible in Bugzilla by having separate patches, but it’s difficult to track). And there are specific things that Review Board has that aren’t as useful for us as they could be, like the dashboard.

This will also be a big part of the work in the next quarter (at least).

Inability to use MozReview at all due to technological limitations

The single biggest item here is lack of support for git, particularly a git interface for hg repos like mozilla-central. There are many people interested in using MozReview, but their work flows are based around git using git-cinnabar. gps and kanru did some initial work around this in bug 1153053; fleshing this out into a proper solution isn’t a small task, but it seems clear that we’ll have to finish it regardless before too long, if we want MozReview to be the central code-review tool at Mozilla. We’re still trying to decide how this fits into the above priorities; more users is good, but making existing users happier is as well.

Big-ticket items

As mentioned at the beginning of this post, the main reason we’re building a new review tool is to make it repository-centric, that is, based around commits, not isolated patches. This makes a lot of long-desired tools and features much more feasible, including autoland, automatic static analysis, commit rewriting to automatically include metadata like reviewers, and a bunch of other things.

This has been a big focus for the last few months. We’ve had autoland-to-try for a little while now, and autoland-to-inbound is nearly complete. We have a generic library for static analysis with which we’ll be able to build various review bots. And, of course, the one big feature we started with, the ability push commits to MozReview instead of exporting standalone patches, which by itself is both more convenient and preserves more information.

After autoland-to-inbound we’ll be putting aside other big features for a little while to concentrate on general user experience so that people enjoy using MozReview, but rest assured we’ll be back here to build more powerful workflows for everyone.

September 30, 2015 06:15 PM

Cameron Dawson

Treeherder loading data from Pulse

As our infrastructure is moving away from BuildBot and toward Task Cluster, we need to update the way in which job and build data gets to Treeherder.  This is an upcoming feature that will roll out over the next few weeks.  As with any feature that requires synchronicity between teams, I expect we will have to iterate on this as we move forward.  But this is the basic layout of how it will work.

As of now, build and test data gets to Treeherder 1 of 2 ways:

  1. BuildBot: We Extract, Translate and Load the JSON files pending.js, running.js and builds4hr.js and persist that information in the Treeherder database.
  2. Non-BuildBot projects (Task Cluster, Autophone, etc): those projects must post their data via our REST web API interfact.  This means, to get them into Production and Stage, they must be posted to each, independently.

What this also means is that, for method #2, on my local machine, I can’t ingest any data from Task Cluster or the other projects that post to Treeherder Stage/Production.

The Treeherder team has worked with Jonas Jensen and Greg Arndt from Task Cluster to come up with a better solution using Pulse Exchanges.  Below is a diagram of how this will work:

Treeherder Pulse-4

In a nutshell:

  1. Treeherder provides a JSON Schema defining exactly what a job can/must look like.
  2. An external application (I’m looking at you, Task Cluster) will create a Pulse Exchange via Pulse Guardian.
  3. The external application will write a bug or create a pull request against Treeherder to update the “Exchange Config” in ““.  This config specifies their exchange name(s) and which repositories they will post jobs for.
  4. Treeherder creates Pulse Queues to listen to all registered Pulse Exchanges.
  5. The external application (I’m looking at you, fictional Oscillation Overthruster) will start posting jobs to their exchange.
  6. Treeherder reads jobs with matching routing keys from Pulse, queues them asynchronously and writes them to the Treeherder database.

Et Voila!  Your dish is served piping hot.

The Exchange Config is specified as JSON like this in Treeherder’s ““:

        "name": "exchange/taskcluster-treeherder/jobs",
        "projects": [
            # other repos TC can submit to
        "destinations": [
        "name": "exchange/treeherder-test/jobs",
        "projects": [
        "destinations": [
    ... other CI systems

Once the feature has landed on our “master“ branch, you can read more about it in the Treeherder Documentation.  There you will find more information about testing this in a local Treeherder Vagrant environment.

September 30, 2015 05:36 PM

Joel Maher

Say hi to Gabriel Machado- a newer contributor on the Perfherder project

Earlier this summer, I got an email from Gabriel asking how he could get involved in Automation and Tools projects at Mozilla.  This was really cool and I was excited to see Gabriel join the Mozilla Community.  Gabriel is knows as :goma on IRC and based on his interests and projects with mentors  available, hacking on TreeHerder was a good fit.  Gabriel also worked on some test cases for the Firefox-UI-Tests.  You can see the list of bugs he has been involved with and check him out on Github.

While it is great to see a contributor become more comfortable in their programming skills, it is even better to get to know the people you are working with.  As I have done before, I would like to take a moment to introduce Gabriel and let him do the talking:

Tell us about where you live –

I lived in Durham-UK since last year, where I was doing an exchange program. About Durham I can say that is a lovely place, it has a nice weather, kind people, beautiful castles and a stunning “medieval ambient”.  Besides it is a student city, with several parties and cultural activities.

I moved back to Brazil 3 days ago, and next week I’ll move to the city of Ouro Preto to finish my undergrad course. Ouro Preto is another beautiful historical city, very similar to Durham in some sense. It is a small town with a good university and stunning landmarks. It’s a really great place, designated a world heritage site by UNESCO.

Tell us about your school –

In 2 weeks I’ll begin my third year in Computer Science at UFOP(Federal University of Ouro Preto). It is a really good place to study computer science, with several different research groups. In my second year I earned a scholarship from the Brazilian Government to study in the UK. So, I studied my second year at Durham University. Durham is a really great university, very traditional and it has a great infra-structure. Besides, they filmed several Harry Potter scenes there :P

Tell us about getting involved with Mozilla –

In 2014 I was looking for some open source project to contribute with when I found the Mozilla Contributing Guide. It is a really nice guide and helped me a lot. I worked on some minors bugs during the year. In July of 2015, as part of my scholarship to study in the UK, I was supposed to do a small final project and I decided to work with some open source project, instead of an academic research. I contacted jmaher by email and asked him about it. He answered me really kindly and guided me to contribute with the Treeherder. Since then, I’ve been working with the A-Team folks, working with Treeherder and Firefox-Ui-Tests.

I think Mozilla does a really nice job helping new contributors, even the new ones without experience like me. I used to think that I should be a great hacker, with tons of programming knowledge to contribute with an open source project. Now, I think that contributing with an open source project is a nice way to become a great hacker with tons of programming knowledge

Tell us what you enjoy doing –

I really enjoy computers. Usually I spent my spare time testing new operating systems, window managers or improving my Vim. Apart from that, I love music. Specially instrumental.  I play guitar, bass, harmonica and drums and I really love composing songs. You can listen some of my instrumental musics here:

Besides, I love travelling and meeting people from different cultures. I really like talking about small particularities of different languages.

Where do you see yourself in 5 years?

I hope be a software engineer, working with great and interesting problems and contributing for a better (and free) Internet.

If somebody asked you for advice about life, what would you say?

Peace and happiness comes from within, do not seek it without.

Please say hi to :goma on irc in #treeherder or #ateam.

September 30, 2015 09:52 AM

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

September 30, 2015 07:42 AM

September 29, 2015

William Lachance

Perfherder summer of contribution thoughts

A few months ago, Joel Maher announced the Perfherder summer of contribution. We wrapped things up there a few weeks ago, so I guess it’s about time I wrote up a bit about how things went.

As a reminder, the idea of summer of contribution was to give a set of contributors the opportunity to make a substantial contribution to a project we were working on (in this case, the Perfherder performance sheriffing system). We would ask that they sign up to do 5-10 hours of work a week for at least 8 weeks. In return, Joel and myself would make ourselves available as mentors to answer questions about the project whenever they ran into trouble.

To get things rolling, I split off a bunch of work that we felt would be reasonable to do by a contributor into bugs of varying difficulty levels (assigning them the bugzilla whiteboard tag ateam-summer-of-contribution). When someone first expressed interest in working on the project, I’d assign them a relatively easy front end one, just to cover the basics of working with the project (checking out code, making a change, submitting a PR to github). If they made it through that, I’d go on to assign them slightly harder or more complex tasks which dealt with other parts of the codebase, the nature of which depended on what they wanted to learn more about. Perfherder essentially has two components: a data storage and analysis backend written in Python and Django, and a web-based frontend written in JS and Angular. There was (still is) lots to do on both, which gave contributors lots of choice.

This system worked pretty well for attracting people. I think we got at least 5 people interested and contributing useful patches within the first couple of weeks. In general I think onboarding went well. Having good documentation for Perfherder / Treeherder on the wiki certainly helped. We had lots of the usual problems getting people familiar with git and submitting proper pull requests: we use a somewhat clumsy combination of bugzilla and github to manage treeherder issues (we “attach” PRs to bugs as plaintext), which can be a bit offputting to newcomers. But once they got past these issues, things went relatively smoothly.

A few weeks in, I set up a fortnightly skype call for people to join and update status and ask questions. This proved to be quite useful: it let me and Joel articulate the higher-level vision for the project to people (which can be difficult to summarize in text) but more importantly it was also a great opportunity for people to ask questions and raise concerns about the project in a free-form, high-bandwidth environment. In general I’m not a big fan of meetings (especially status report meetings) but I think these were pretty useful. Being able to hear someone else’s voice definitely goes a long way to establishing trust that you just can’t get in the same way over email and irc.

I think our biggest challenge was retention. Due to (understandable) time commitments and constraints only one person (Mike Ling) was really able to stick with it until the end. Still, I’m pretty happy with that success rate: if you stop and think about it, even a 10-hour a week time investment is a fair bit to ask. Some of the people who didn’t quite make it were quite awesome, I hope they come back some day. :)

On that note, a special thanks to Mike Ling for sticking with us this long (he’s still around and doing useful things long after the program ended). He’s done some really fantastic work inside Perfherder and the project is much better for it. I think my two favorite features that he wrote up are the improved test chooser which I talked about a few months ago and a get related platform / branch feature which is a big time saver when trying to determine when a performance regression was first introduced.

I took the time to do a short email interview with him last week. Here’s what he had to say:

1. Tell us a little bit about yourself. Where do you live? What is it you do when not contributing to Perfherder?

I’m a postgraduate student of NanChang HangKong university in China whose major is Internet of things. Actually,there are a lot of things I would like to do when I am AFK, play basketball, video game, reading books and listening music, just name it ; )

2. How did you find out about the ateam summer of contribution program?

well, I remember when I still a new comer of treeherder, I totally don’t know how to start my contribution. So, I just go to treeherder irc and ask for advice. As I recall, emorley and jfrench talk with me and give me a lot of hits. Then Will (wlach) send me an Email about ateam summer of contribution and perfherder. He told me it’s a good opportunity to learn more about treeherder and how to work like a team! I almost jump out of bed (I receive that email just before get asleep) and reply with YES. Thank you Will!

3. What did you find most challenging in the summer of contribution?

I think the most challenging thing is I not only need to know how to code but also need to know how treeherder actually work. It’s a awesome project and there are a ton of things I haven’t heard before (i.e T-test, regression). So I still have a long way to go before I familiar with it.

4. What advice would give you to future ateam contributors?

The only thing you need to do is bring your question to irc and ask. Do not hesitate to ask for help if you need it! All the people in here are nice and willing to help. Enjoy it!

September 29, 2015 03:37 PM

September 23, 2015

Dan Minor

Beyond the Code 2015

Mozilla was kind enough to send me to the Beyond the Code 2015 in Ottawa today. When I went to PyCon back in April, the sessions I felt I got the most out of were the non-technical ones so this seemed like a fun and useful way to spend the day.

My favourite talk of the day was given by Kat Bautista on Drawing as a Way of Thinking which was a funny and insightful talk on using drawings as a means of visual note-taking both for yourself and in front of a whiteboard as a way of facilitating meetings. Not “draw me like one of your french girls” drawing but rather a visual, cartoony shorthand for capturing ideas.

Having suffered through overzealous project management from time to time (e.g. two project managers for five developers) I could relate to her description of Gantt Charts as a way of to avoid having meaningful discussions (“Look at the spreadsheets I can make!”) but I think sketches can also be used to obscure issues, people love pictures and can easily overlook a lack of substance behind them.

draw autoland

I had read a number of books and papers on sketching for design as part of a Software Usability course I took a few years ago so I had already been exposed to some of the ideas before but I still got a lot out of the talk. That usability course was for the most part disappointing, the focus was on doing studies to evaluate existing user interfaces with basically nothing at all on how one would go about creating such a thing in the first place.

As I mentioned, Mozilla also sent me to PyCon 2015 back in April. I was remiss in capturing my thoughts back then. After nearly six months what has stayed most with me was Pete Fein’s talk Free Software, Free People on his experiences with Telecomix, in particular, one slide, where he urged people to be active and work for causes they care about (whatever they may be.)


September 23, 2015 02:37 PM

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

September 23, 2015 05:24 AM

September 22, 2015

Joel Maher

DX Contribution Program – Wrap up

This summer went by fast.  Back in May we announced the opportunity to participate in the DX project.  In June we announced the team.  August ended and the project officially came to a halt.  First off, I would really like to thank all the participants who took interest in this program.  With 5 eager hackers, there is a lot to do!  As the summer wore on, some folks got side tracked, others worked when they could and some kept moving forward.  No matter what the outcome, I do applaud the 5 great contributors who spent their free time getting setup and started on a project.

Now that we are at the end, I would like to outline some of the work that was done and things that were learned.

A few great things got started like mach usage statistics, and Marionette harness cleanup.  These are not easy to get started on and figure out all the quirks with our full scale ecosystem.  It is wonderful to see ideas and progress being made.   Thanks BYK and sehgalvibhor.

Two other projects didn’t get much traction.  While I like to blame myself as a mentor (yes, I could have done a bit more or scoped the projects differently), we ended up not making progress on a few projects.  This is OK, and there is always stuff to learn.  On the projects that didn’t get going, I saw gma_fav struggle a bit, but persist.  I know the right project is out there and she will succeed next time, ideally at Mozilla!  I enjoyed learning about wunderlist from stanley, and will look forward to more of his enthusiasm.

The last project is run-by-dir.  kaustabh93 really worked consistently on this and learned a lot about working in the mozilla-central tree, using try, and understanding our infrastructure.  A lot of work was done for mochitest-plain and mochitest-chrome.  He is still hacking away and making fast progress.  As Kaustabh was the only contributor who was consistent throughout the entire program, I wanted to hear his perspective on the program:

How did you hear about the program, why were you interested?
I had been a contributor at Mozilla for quite sometime working mainly on AlertManager and familiarizing myself with Mochitests by means of solving bugs and reading code. It was at this time, that my curriculum needed me to do an industrial project (internships/training/other programs for which the company would acknowledge mt work by means of certificates or letters or both). I brought this up with my mentor and a few days later, he informed me about this program.
It had been an awesome experience coding for Mozilla so far & hence I got interested instantly.
What did you find challenging?  How did you overcome these challenges?
The existing volume of codes were huge and getting to know what is what was challenging. I overcame this by solving smaller bugs first and reading codes and of course a lot of help, advice & explanation from my mentor, jmaher. He is an awesome person and one of the best mentors, I’ve known.

What advice would you give for anybody looking to contribute in this program?

People looking to contribute to this program should get involved in coding for Mozilla some time prior to commencement of this program. That way they can be familiarized with the existing codes and can better understand their work and this way, they can make the maximum impact with their work. And they should continue contributing to Mozilla even after this program closes.
Other thoughts : One more thing that I should advice people regarding this program is that if they contribute to Mozilla just for this programs (contribution start and end dates coincides with that of the program), then they’re probably doing it wrong.

While I appreciate the callout as a great mentor, there are many other great Mozillian mentors, it is my pleasure to work with folks who contribute their free time to the project,

Without making this too long, I did learn a lot from this great group of participants.  A few tips for becoming a better mentor (you get better with experience and learn from mistakes) is to communicate regularly with whom you are mentoring, this should be at least twice a week.  While mentoring involves a lot of encouragement and gentle guidance, it is important to listen more to what roadblocks are in place (schedules, technical, etc.) and sometimes we need to move to a smaller or simpler project until the timing is right for the big project.  One other big callout was that gma_fav was helpful in making me really understand how our use of master/slave is offensive to many, I am glad I listened to her and worked with her for a couple months!

I look forward to watching these contributors grow outside of this program and meeting more great folks!

September 22, 2015 01:39 AM

September 21, 2015

Armen Zambrano G. (@armenzg)

Minimal job scheduling

One of the biggest benefits of moving the scheduling into the tree is that you can adjust the decisions on what to schedule from within the tree.

As chmanchester and I were recently discussing, this is very important as we believe we can do much better on deciding what to schedule on try.

Currently, too many developers push to try with  -p all -u all (which schedules all platforms and all tests). It is understandable, it is the easiest way to reduce the risk of your change being backed out when it lands on one of the integration trees (e.g. mozilla-inbound).

In-tree scheduling analysis

What if your changes would get analysed and we would determine the best educated guess set of platforms and test jobs required to test your changes in order to not be backed out on an integration tree?

For instance, when I push Mozharness changes to mozilla-inbound, I wish I could tell the system that I only need these set of platforms and not those other ones.

If everyone had the minimum amount of jobs added to their pushes, our systems would be able to return results faster (less load) and no one would need to take short cuts.

This would be the best approximation and we would need to fine tune the logic over time to get things as right as possible. We would need to find the right balance of some changes being backed out because we did not get the right scheduling on try and getting results faster for everyone.

Prioritized tests

There is already some code that chmanchester landed where we can tell the infrastructure to run a small set of tests based on files changed. In this case we hijack one of the jobs (e.g. mochitest-1) to run the most relevant tests to your changes which would can normally be tested on different chunks. Once the prioritized tests are run, we can run the remaining tests as we would normally do. Prioritized tests also applies to suites that are not chunked (run a subset of tests instead of all).

There are some UI problems in here that we would need to figure out with Treeherder and Buildbot.

Tiered testing

Soon, we will have all technological pieces to create a multi tiered job scheduling system.

For instance, we could run things in this order (just a suggestion):

This has the advantage of using prioritized tests as a canary job which would prevent running the remaining tests if we fail the canary (shorter) job.

Post minimal run (automatic) precise scheduling (manual)

This is not specifically to scheduling the right thing automatically but to extending what gets scheduled automatically.
Imagine that you're not satisfied with what gets scheduled automatically and you would like to add more jobs (e.g. missing platforms or missing suites).
You will be able to add those missing jobs later directly from Treeherder by selecting which jobs are missing.
This will be possible once bug 1194830 lands.

NOTE: Mass scheduling (e.g. all mochitests across all platforms) would be a bit of a pain to do through Treeherder. We might want to do a second version of try-extender.

Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

September 21, 2015 01:15 PM

September 18, 2015

Armen Zambrano G. (@armenzg)

Mozharness' support for Buildbot Bridge triggered test jobs

I have landed today [1] some code which allows Buildbot *test* jobs triggered through the Buildbot Bridge (BBB) to work properly.

In my previous post I explain a bit on how Mozci works with the Buildbot Bridge.
In this post I will only explain what we fixed on the Mozharness side.

Buildbot Changes

If a Buildbot test job is scheduled through TaskCluster (The Buildbot Bridge supports this), then the generated Buildbot Change associated to a test job does not have the installer and
test urls necessary for Mozharness to download for a test job.

What is a Buildbot Change? It is an object which represents the files touched associated to a code push. For the build jobs, this value gets set as part of the process of polling the Mercurial repositories, however, the test jobs are triggered via  a "buildbot sendchange" step part of the build job.
This sendchange creates the Buildbot Change for the test job which Mozharness can then use.
The BBB does not listen to sendchanges, hence, jobs triggered via the BBB have an empty changes object. Therefore, we can't download the files needed to run a test job and fail to execute.

In order to overcome this limtation, we have to detect when a Buildbot job is triggered normally or through the Buildbot Bridge.
Buildbot Bridge triggered jobs have a 'taskId' property defined (this represents the task associated to this Buildbot job). Through this 'taskId' we can determine the parent task and find a file called properties.json [2], which, it is uploaded by every BBB triggered job.
In such file, we can find both the installer and test urls.


Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

September 18, 2015 07:26 PM

Mozilla CI tools: Upcoming support for Buildbot Bridge

What is the Buildbot Bridge?

The Buildbot Bridge (BBB) allows scheduling Buildbot jobs through TaskCluster.
In other words, you can have taskcluster tasks which represent Buildbot jobs.
This allows having TaskCluster graphs composed of tasks which will be executed either on Buildbot or TaskCluster, hence, allowing for *all* relationships between tasks to happen in TaskCluster.

Read my recent post on the benefits of scheduling every job via TaskCluster.

The next Mozilla CI tools (mozci) release will have support for BBB.

Brief explanation

You can see in this try push both types of Buildbot jobs [1].
One set of jobs were triggered through Buildbot's analysis of the try syntax in the commit message while two of the jobs should not have been scheduled.

Those two jobs were triggered off-band via Mozci submitting a task graph.
You can see the TaskCluster graph representing them in here [2].

These jobs were triggered using this simple data structure:
  'Linux x86-64 try build': [
    'Ubuntu VM 12.04 x64 try opt test mochitest-1'

Mozci turns this simple graph into a TaskCluster graph.
The graph is composed of tasks which follow this structure [3]

Notes about the Buildbot Bridge

bhearsum's original post, recording and slides:

Some notes which Selena took about the topic:

The repository is in here:


Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

September 18, 2015 07:16 PM

September 17, 2015

Armen Zambrano G. (@armenzg)

Platform Operations lightning talks (Whistler 2015)

You can read and watch in here about the Platform Operations lighting talks:

Here the landing pages for the various Platform Operations teams:

PS = I don't know what composes platform operations so feel free to add your team if I'm missing it.

Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

September 17, 2015 03:57 PM

September 14, 2015

Mark Côté

MozReview Meet-Up

Two weeks ago the MozReview developers got together to do some focussed hacking. It was a great week, we got a lot of stuff done, and we clarified our priorities for the coming months. We deployed several new features and improvements during the week, and we made good progress on several other goals.

For this week, we actually opted to not go to a Mozilla office and instead rented an AirBNB in Montreal—our own little hacker house! It was a great experience. There was no commuting (except for me, since I live here and opted to sleep at my house) and no distractions. One evening when we were particularly in the zone, we ordered a bunch of pizzas instead of going out, although we made sure to take breaks and go outside regularly on all the other days, in order to refresh our minds. Five out of five participants said they’d do it again!

See also dminor’s post about the week, with a bit more detail on what he worked on.

What we pushed out

My main contribution to the week was finally switching MozReview to Bugzilla’s API-key-based authentication-delegation system. I’d been working on this for the last couple months when I found time, and it was a big relief to finally see it in action. I won’t go into detail here, since I’ve already blogged about it and announced it to dev.platform.

gps, working amazingly fast as always, got API-key support working on the command line almost immediately after UI support was deployed. No more storing credentials or login cookies!

Moving on, we know the MozReview’s UI could… stand some improvement, to say the least. So we split off some standalone work from Autoland around clarifying the status of reviews. Now in the commit table you’ll see something like this:

This warrants a bit of explanation, since we’re reusing some terms from Bugzilla but changing them a little.

r+ indicates at least one ship-it, subject to the following:

The reason for the L3 requirement is for Autoland. Since a human won’t necessarily be looking at your patch between the time that a review is granted and the commit landing in tree, we want some checks and balances. If you have L3 yourself, you’re trusted enough to not require an L3 reviewer, and vice versa. We know this is a bit different from how things work right now with regards to the checkin-needed flag, so we’re going to open a discussion on mozilla.governance about this policy.

If one or more reviewers have left any issues, the icon will be the number of open issues beside an exclamation mark on a yellow backgroud. If that or any other reviewer has also left a ship-it (the equivalent of an “r+ with minor issues”), the issue count will switch to an r+ when all the issues have been marked as fixed or dropped.

If there are no open issues nor any ship-its, a grey r? will be displayed.

We’ve also got some work in progress to make it clearer who has left what kind of review that should be live in a week or two.

We also removed the ship-it button. While convenient if you have nothing else to say in your review, it’s caused confusion for new users, who don’t understand the difference between the “Ship It!” and “Review” buttons. Instead, we now have just one button, labelled “Finalize Review”, that lets you leave general comments and, if desired, a ship-it. We plan on improving this dialog to make it easier to use if you really just want to leave just a ship-it and no other comments.

Finally, since our automation features will be growing, we moved the Autoland-to-try button to a new Automation menu.

Where we’re going

As alluded to above, we’re actively working on Autoland and trying to land supporting features as they’re ready. We’re aiming to have this out in a few weeks; more posts to come.

Much of the rest of our plan for the next quarter or two is around user experience. For starters, MozReview has to support the same feature set as Splinter/Bugzilla. This means implementing things like marking files as reviewed and passing your review onto someone else. We also need to continue to improve the user interface by further differentiating between parent review requests, which are intended only to provide an overview of the whole commit series, and child review requests, which is where the actual reviews should happen. Particularly confusing is when there’s only one child, which means the parent and the child are nearly indistinguishable (of course in this case the parent isn’t really needed, but hiding or otherwise doing away with the parent in this case could also be confusing).

And then we have some other big-ticket items like static analysis, which we started a while back; release-management tools; enabling Review Board’s rich emails; partial-series landing (being able to land earlier commits as they are reviewed without confusing MozReview in the next push); and, of course, git support, which is going to be tricky but will undoubtedly make a number of people happy.

Our priorities are currently documented on our new road map, which we’ll update at least once or twice a quarter. In particular, we’ll be going over it again soon once we have the results from our engineering-productivity survey.

September 14, 2015 07:26 PM

Henrik Skupin

mozdownload 1.18 released

Today we have released mozdownload 1.18 to PyPI. The reason why I think it’s worth a blog post is that with this version we finally added support for a sane API. With it available using the mozdownload code in your own script is getting much easier. So there is no need to instantiate a specific scraper anymore but a factory scraper is doing all the work depending on the options it gets.

Here some examples:

from mozdownload import FactoryScraper
scraper = FactoryScraper('release', version='40.0.3', locale='de')
from mozdownload import FactoryScraper
scraper = FactoryScraper('candidate', version='41.0b9', platform='win32')
from mozdownload import FactoryScraper
scraper = FactoryScraper('daily', branch='mozilla-aurora')

If you are using mozdownload via its API you can also easily get the remote URL and the local filename:

from mozdownload import FactoryScraper
scraper = FactoryScraper('daily', branch='mozilla-aurora')
print scraper.url
print scraper.filename

Hereby the factory class is smart enough to only select those passed-in options which are appropriate for the type of scraper. If you have to download different types of builds you will enjoy that feature given that only the scraper type has to be changed and all other options could be still passed-in.

We hope that this new feature will help you by integrating mozdownload into your own project. There is no need anymore by using its command line interface through a subprocess call.

The complete changelog can be found here.

September 14, 2015 03:32 PM

September 12, 2015

Julien Pagès

mozregression updates!

A lot of new things have been added in mozregression recently, and I think this deserve a blog post.

Well, big thanks for MikeLing and Jonathan Pigree for their great work on those tools! They are an important part of the team, helping with discussions and patches. Oh, and also a big thanks to users who report bugs and make great proposals: Elbart, Ray Satiro, arni2033, Jukka Jylänki, and all others!

September 12, 2015 10:23 AM

mozregression updates

GUI release 0.4.0

Sorry for the delay on GUI release and posts, but we were quite busy with mozregression command line these days!

Still we don’t forget about the GUI. And now there are a bunch of great new cool things with this 0.4.0 release!

As usual now, big thanks for MikeLing and Jonathan Pigree for their work and time! They both have a really great impact on the software, from decisions to implementations.

September 12, 2015 12:00 AM

September 11, 2015

Jonathan Griffin

Engineering Productivity Update, Sept 10, 2015


Bugzilla: The BMO has been busy implementing security enhancements, and as a result, BMO now supports two-factor authentication.  Setting this up is easy through BMO’s Preferences page.

Treeherder: The size of the Treeherder database dropped from ~700G to around ~80G thanks to a bunch of improvements in the way we store data.  Jonathan French is working on improvements to the Sheriff Panel.  And Treeherder is now ingesting data that will be used to support Automatic Starring, a feature we expect to be live in Q4.

Perfherder and Performance: Will Lachance has published a roadmap for Perfherder, and has landed some changes that should improve Perfherder’s performance.  Talos tests on OSX 10.10 have been hidden in Perfherder because the numbers are very noisy; the reason for this is not currently known.  Meanwhile, Talos has finally landed in mozilla-central, which should make it easier to iterate on.  Thanks to our contributor Julien Pagès for making this happen!  Joel Maher has posted a Talos update on dev.platform with many more details.

MozReview and Autoland: The web UI now uses BMO API keys; this should make logins smoother and eliminate random logouts.  Several UI improvements have been made; see full list in the “Details” section below.

Mobile Automation: Geoff Brown has landed the much-requested |mach android-emulator| command, which makes it much easier to run tests locally with an Android emulator.  Meanwhile, we’re getting closer to moving the last Talos Android tests (currently running on panda boards) to Autophone.

Developer Workflow: Our summer intern, Vaibhav Agrawal, landed support for an initial version of |mach find-test-chunk|, which can tell you which chunk a test gets run in.  This initial version supports desktop mochitest only.  Vaibhav gave an intern presentation this week, “Increasing Productivity with Automation”.  Check it out!

General Automation: James Graham has enabled web-platform-tests-e10s on try, but they’re hidden pending investigation of tests which are problematic with e10s enabled.  Joel Maher and Kim Moir in Release Engineering have tweaked our SETA coalescing, so that lower prioritized jobs are run at least every 7th push, or every hour; further increasing the coalescing window will wait until we have better automatic backfililng in place.  Meanwhile, the number of chunks of mochitest-browser-chrome has been increased from 3 to 7, with mochitest-devtools-chrome soon to follow.  This will make backfilling faster, as well as improving turnaround times on our infrastructure. The bzexport and bzpost extensions have been updated to support BMO API keys.

WebDriver and Marionette: Several changes were made to the WebDriver specification, including new chapters on screen capture and user prompts and modal handling.

Bughunter: Our platform coverage now includes opt and debug builds of linux32, linux64, opt-asan-linux64, OSX 10.6, 10.8, 10.9, and windows7 32- and 64-bit.

The Details
Perfherder/Performance Testing
  • Autoland VCS interaction performance improvements
  • MozReview web UI now using BMO API keys
  • Login in smoother and faster and random logouts should be a thing of the past
  • MozReview Mercurial client extension now requires BMO API keys
  • No more defining password and/or cookie in plaintext config files
  • “Ship It” is now determined by reviewer status so random people can’t grant Ship It
  • Messaging during pushing is more clear about what is happening
  • “commitid” hidden metadata is now preserved across `hg rebase`
  • Review buttons and text links in web interface are now consolidated to reduce confusion
  • Empty Try syntax is now rejected properly
  • Try trigger button has been moved to an “Automation” drop-down
  • pull and import commands are now displayed
  • Bugzilla’s commit list is now properly ordered
TaskCluster Support
  • armenzg – Work is underway to support running Buildbot test jobs through TaskCluster
  • ted – successful Mac cross build with Breakpad last week, landing patches and fixing some fallout from Linux TC build switch to CentOS 6
Mobile Automation
Dev Workflow
  • vaibhav1994 – A basic version of find-test-chunk has landed. This will help in determining on which chunk a particular test is present in production. It works for mochitest for desktop platforms, see various options with ‘./mach find-test-chunk’
  • vaibhav1994 – –rebuild-talos option now present in trigger-bot to trigger only talos jobs a certain number of times on try.
Firefox and Media Automation
  • sydpolk – Network bandwidth limiting tests have been written; working to deploy them to Jenkins.
  • sydpolk – Streamlined Jenkins project generation based on Jenkins python API (found out about this at the Jenkins conference last week)
  • sydpolk – Migration of hardware out of MTV2 QA lab won’t happen this quarter because Network Ops people are shutting down the Phoenix data center.
  • maja_zf – mozharness script for firefox-media-tests has been refactored into scripts for running the tests in buildbot and our Jenkins instance
General Automation
  • chmanchester – psutil 3.1.1 is now installed on all test slaves as a part of running desktop unit tests. This will help our test harnesses manages subprocesses of the browser, and particularly kill them to get stacks after a hang.
  • armenzg – Firefox UI tests can now be called through a python wrapper instead of only through a python binary. This is very important since it was causing Windows UAC prompts on Release Engineering’s Windows test machines. The tests now execute well on all test platforms.
  • jgraham – web-platform-tests-e10s now running on try, but still hidden pending some investigation of tests that are only unstable with e10s active
  • SETA work is ongoing to support new platforms, tests, and jobs.  
  • [ekyle] Queries into nested documents pass tests, but do not scale on the large cluster; startup time is unacceptable.  Moving work to separate thread for quick startup, with the hope a complicated query will not arrive until the metadata is finished collecting
  • [ekyle] Added auto-restart on ETL machines that simply stop working (using CloudWatch); probably caused by unexpected data, which must be looked into later.
  • [ekyle] SpotManager config change for r3.* instances
  • Add times for replication events on push
  • Reformat pushlog messages on push to be less eye bleedy
  • bzexport and bzpost extensions now support Bugzilla API Keys
  • [ato] Specified element interaction commands
  • [ato] New chapter on user prompts and modal handling
  • [ato] New chapter on screen capture
  • [ato] Cookies retrieval bug fixes
  • [ato] Review of normative dependencies
  • [ato] Security chapter review
  • Wires 0.4 has been released.
  • [ato] Deployed Marionette protocol changes, bringing Marionette closer to the WebDriver standard
  • [ato] Assorted content space commands converted to use new dispatching technique
  • [jgraham] Updated wires for protocol changes
Now running opt, debug tinderbox builds for Linux 32 bit, 64 bit; OSX 10.6, 10.8, 10.9; Windows 7 32 bit, 64 bit; opt asan tinderbox builds for Linux 64 bit.
  • bug 1180749 Sisyphus – Django 1.8.2 support
  • bug 1185497 Sisyphus – Bughunter – use ASAN builds for Linux 64 bit workers
  • bug 1192646 Sisyphus – Bughunter – use to upload urls to be resubmitted
  • bug 1194290 Sisyphus – Bughunter – install gtk3 on rhel6 vms

September 11, 2015 02:33 AM

September 10, 2015

Armen Zambrano G. (@armenzg)

The benefits of moving per-push Buildbot scheduling into the tree

Some of you may be aware of the Buildbot Bridge (aka BBB) work that bhearsum worked on during Q2 of this year. This system allows scheduling TaskCluster graphs for Buildbot builders. For every Buildbot job, there is a TaskCluster task that represents it.
This is very important as it will help to transition the release process piece by piece to TaskCluster without having to move large pieces of code at once to TaskCluster. You can have graphs of

I recently added to Mozilla CI tools the ability to schedule Buildbot jobs by submitting a TaskCluster graph (the BBB makes this possible).

Even though the initial work for the BBB is intended for Release tasks, I believe there are various benefits if we moved the scheduling into the tree (currently TaskCluster works like this; look for the gecko decision task in Treeherder).

To read another great blog post around try syntax and schedulling please visit ahal's post "Looking beyond Try Syntax".

NOTE: Try scheduling might not have try syntax in the future so I will not talk much about trychooser and try syntax. Read ahal's post to understand a bit more.

Benefits of in-tree scheduling:

There are various parts that will need to be in place before we can do this. Here's some that I can think of:
  • TaskCluster's big-graph scheduling
    • This is important since it will allow for the concept of coalescing to exist in TaskCluster
  • Task prioritization
    • This is important if we're to have different levels of priority for jobs on TaskCluster
    • On Buildbot we have release repositories with the highest priority and the try repo having the lowest
    • We also currently have the ability to raise/decrease task priorities through self-serve/buildapi. This is used by developers, specially on Try. to allow their jobs to be picked up sooner.
  • Treeherder to support LDAP authentication
    • It is a better security model to scheduling changes
    • If we want to move away from self-server/buildapi we need this
  • Allow test jobs to find installer and test packages
    • Currently test jobs scheduled through the BBB cannot find the Firefox installer and the 
Can you think of other benefits? Can you think of problems with this model? Are you aware of other pieces needed before moving forward to this model? Please let me know!

Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

September 10, 2015 02:13 PM

September 09, 2015

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

September 09, 2015 06:38 AM

September 05, 2015

mozregression updates

Release 1.0.0

We made a lot of great changes in mozregression recently, from refactoring to really cool new features. So I thought it was time to switch to a newer major version!

So here we are, the 1.0.0 release! Some great features:

This release also fixes an intermittent bug introduced with 0.42, see bug 1200270.

Last but not least, mozregression is now prepared for mach integration! Look at bug 1200293 to follow that.

September 05, 2015 12:00 AM

September 03, 2015

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

September 03, 2015 06:51 AM

September 01, 2015

Byron Jones

happy bmo push day!

the following changes have been pushed to

today’s push lands initial support for two-factor authentication on BMO. we currently support time-based one-time passwords (totp) with protection around just logging in. 2fa protection will be extended to protect other actions in the upcoming weeks.

visit the ‘two-factor authentication‘ section under your user preferences to enable 2fa.

discuss these changes on

Filed under: bmo, mozilla

September 01, 2015 01:05 PM

Alice Scarpa

Goodbye Outreachy!

Last Tuesday was the last day of my Outreachy internship :(


It all started last December. I was at the Recurse Center when Sonali told me about the project. At first I was sure I was not good enough to contribute to a real-life Open Source project, I shouldn’t even bother trying. But I was at the RC, which means I had a great support network and I was able to talk with Sumana about my doubts. I don’t remember the exact words, but it was something like:

  • What if it takes more time to mentor me to fix a bug then to just fix the bug directly?

  • Mentors know what they are signing up for and they think it is worth it. They think of it as part of their long-term investment in the project.

I decided I should at least try to contribute once, and after weeks of window shopping for bugs in Bugs Ahoy I finally got started.


I felt very welcomed on the a-team. No one asked for my credentials, no one assumed I was not able to code (even when I made mistakes or asked silly questions!).

Everyone was very nice with me and they helped me whenever I needed it: I was mentored by jmaher, ahal and armenzg on different bugs. mcote, camd, emorley, jfrench, Kwierso and mdoglio helped me with PRs. RyanVM and jgraham gave very useful feedback about mozci. ekyle helped me get started with ActiveData. vaibhav1994 and chmanchester helped with mozci. And there are a lot of people with which I didn’t work directly but that were very nice to me whenever I interacted with them.

For the past months, the #ateam IRC channel was my office and reading the conversations made me happy, whether they were about techinical issues, the weather, cricket or anything in between.


I knew the team was very nice, but I wasn’t expecting my mentor to be so awesome. armenzg helped me so much: he gave me fun projects that helped me grow, he arranged for me to go to Toronto, he guided me whenever I felt lost, he trusted I would be able to finish projects completely outside my comfort zone, he arranged for me to give a presentation (achievement unlocked: I have my own AirMozilla URL). Having a mentor I could trust made the whole internship a much better experience.

I followed maja_zf’s advice and kept a daily log of my internship. Now I can look back at what I did. It was an amazing Winter!

September 01, 2015 12:00 AM

August 29, 2015

mozregression updates

Release 0.42

0.42 is an early release because inbound bisection for firefox was broken, see bug 1199618.

Still there is some great improvements in there!

Also we did a really nice work of making the code source easier and cleaner. Big thanks to MikeLing and Jonathan Pigree for their patches, reviews and testing contributions!

August 29, 2015 12:00 AM

August 28, 2015

Dan Minor

MozReview Montréal Work Week


Under the watchful gaze (and gentle cooing) of the pigeons, the MozReview developers gathered in Montréal for a work week. My main goal for the week was to make substantial progress towards Autoland to Inbound, my primary project for this quarter, maybe even a deployment of an initial iteration to the development server.

While we didn’t get quite that far, we did make a lot of progress on a lot of fronts, including finally getting Bugzilla API key support deployed. This is the first work week I’ve done where we just stayed and worked together in an AirBNB apartment rather than get hotel rooms and make use of shared space in a Mozilla office. I really enjoyed this, it was a nice casual work environment and we got a lot of focused work done.

Some things I worked on this week, in varying degrees of completion:

August 28, 2015 02:55 PM

August 27, 2015

Andrew Halberstadt

Looking beyond Try Syntax

Today marks the 5 year anniversary of try syntax. For the uninitiated, try syntax is a string that you put into your commit message which a parser then uses to determine the set of builds and tests to run on your try push. A common try syntax might look like this:

try: -b o -p linux -u mochitest -t none

Since inception, it has been a core part of the Mozilla development workflow. For many years it has served us well, and even today it serves us passably. But it is almost time for try syntax to don the wooden overcoat, and this post will explain why.

A brief history on try syntax

In the old days, pushing to try involved a web interface called sendchange.cgi. Pushing is probably the wrong word to use, as at no point did the process involve version control. Instead, patches were uploaded to the web service, which in turn invoked a buildbot sendchange with all the required arguments. Like today try server was often overloaded, sometimes taking over 4 hours for results to come back. Unlike today there was no way to pick and choose which builds and tests you wanted, every try push ran the full set.

The obvious solution was to create a mechanism for people to do that. It was while brainstorming this problem that ted, bhearsum and jorendorff came up with the idea of encoding this information in the commit message. Try syntax was first implemented by lsblakk in bug 473184 and landed on August 27th, 2010. It was a simple time; the list of valid builders could fit into a single 30 line config file; Fennec still hadn't picked up full steam; and B2G wasn't even a figment of anyone's wildest imagination.

It's probably not a surprise to anyone that as time went on, things got more complicated. As more build types, platforms and test jobs were added, the try syntax got harder to memorize. To help deal with this, lsblakk created the trychooser syntax builder just a few months later. In 2011, pbiggar created the trychooser mercurial extension (which was later forked and improved by sfink). These tools were (and still are) the canonical way to build a try syntax string. Little has changed since then, with the exception of the mach try command that chmanchester implemented around June 2015.

One step forward, two steps back

Since around 2013, the number of platforms and test configurations have grown at an unprecendented rate. So much so, that the various trychooser tools have been perpetually out of date. Any time someone got around to adding a new job to the tools, two other jobs had already sprung up in its place. Another problem caused by this rapid growth, was that try syntax became finicky. There were a lot of edge cases, exceptions to the rule and arbitrary aliases. Often jobs would mysteriously not show up when they should, or mysteriously show up when they shouldn't.

Both of those problems were exacerbated by the fact that the actual try parser code has never had a definite owner. Since it was first created, there have never been more than 11 commits in a year. There have been only two commits to date in 2015.

Two key insights

At this point, there are two things that are worth calling out:

  1. Generating try strings from memory is getting harder and harder, and for many cases is nigh impossible. We rely more and more on tools like trychooser.
  2. Try syntax is sort of like an API on which these tools are built on top of.

What this means is that primary generators of try syntax have shifted from humans to tools. A command line encoded in a commit message is convenient if you're a human generating the syntax manually. But as far as tooling goes, try syntax is one god awful API. Not only do the tools need to figure out the magic strings, they need to interact with version control, create an empty commit and push it to a remote repository.

There is also tooling on the other side of the see saw, things that process the try syntax post push. We've already seen buildbot's try parser but taskcluster has a separate try parser as well. This means that your try push has different behaviour, depending on whether the jobs are scheduled in buildbot or taskcluster. There are other one off tools that do some try syntax parsing as well, including but not limited to try tools in mozharness, the try re-trigger bot and the AWSY dashboard. These tools are all forced to share and parse the same try syntax string, so they have to be careful not to step on each other's toes.

The takeaway here is that for tools, a string encoded as a commit message is quite limiting and a lot less convenient than say, calling a function in a library.

Despair not, young Padawan

So far we've seen how try syntax is finicky, how the tools that use it are often outdated and how it fails as an API. But what is the alternative? Fortunately, over the course of 2015 a lot of progress has been made on projects that for the first time, give us a viable alternative to try syntax.

First and foremost, is mozci. Mozci, created by armenzg and adusca, is a tool that hooks into the build api (with early support for taskcluster as well). It can do things like schedule builds and tests against any arbitrary pushes, and is being used on the backend for tools like adusca's try-extender with integration directly into treeherder planned.

Another project that improves the situation is taskcluster itself. With taskcluster, job configuration and scheduling all lives in tree. Thanks to bhearsum's buildbot bridge, we can even use taskcluster to schedule jobs that still live in buildbot. There's an opportunity here to leverage these new tools in conjunction with mozci to gain complete and total control over how jobs are scheduled on try.

Finally I'd like to call out mach try once more. It is more than a thin wrapper around try syntax that handles your push for you. It actually lets you control how the harness gets run within a job. For now this is limited to test paths and tags, but there is a lot of potential to do some cool things here. One of the current limiting factors is the unexpressiveness of the try syntax API. Hopefully this won't be a problem too much longer. Oh yeah, and mach try also works with git.

A glimpse into the crystal ball

So we have several different projects all coming together at once. The hard part is figuring out how they all tie in together. What do we want to tackle first? How might the future look? I want to be clear that none of this is imminent. This is a look into what might be, not what will be.

There are two places we mainly care about scheduling jobs on try.

First imagine you push your change to try. You open up treeherder, except no jobs are scheduled. Instead you see every possible job in a distinct greyed out colour. Scheduling what you want is as simple as clicking the desired job icons. Hold on a sec, you don't have to imagine it. Adusca already has a prototype of what this might look like. Being able to schedule your try jobs this way has a huge benefit: you don't need to mentally correlate job symbols to job names. It's as easy as point and click.

Second, is pushing a predefined set of jobs to try from the command line, similar to how things work now. It's often handy to have the try command for a specific job set in your shell history and it's a pain to open up treeherder for a simple push that you've memorized and run dozens of times. There are a few improvements we can do here:

Finally for those who are stuck in their ways, it should still be possible to have a "classic try syntax" front-end to the new mozci backend. As large as this change sounds, it could be mostly transparent to the user. While I'm certainly not a fan of the current try syntax, there's no reason to begrudge the people who are.

Closing words

Try syntax has served us well for 5 long years. But it's almost time to move on to something better. Soon a lot of new avenues will be open and tools will be created that none of us have thought of yet. I'd like to thank all of the people mentioned in this post for their contributions in this area and I'm very excited for what the future holds.

The future is bright, and change is for the better.

August 27, 2015 06:03 PM

August 26, 2015

Jonathan Griffin

Engineering Productivity Update, August 26, 2015

It’s PTO season and many people have taken a few days or week off.  While they’re away, the team continues making progress on a variety of fronts.  Planning also continues for GoFaster and addon-signing, which will both likely be significant projects for the team in Q4.


Treeherder: camd rolled out a change which collapses chunked jobs on Treeherder, reducing visual noise.  In the future, we plan on significantly increasing the number of chunks of many jobs in order to reduce runtimes, so this change makes that work more practical.  See camd’s blog post.  emorley has landed a change which allows TaskCluster job errors that occur outside of mozharness to be properly handled by Treeherder.

Automatic Starring: jgraham has developed a basic backend which supports recognizing simple intermittent failures, and is working on integrating that into Treeherder; mdoglio is landing some related database changes. ekyle has received sheriff training from RyanVM, and plans to use this to help improve the automated failure recognition algorithm.

Perfherder and Performance Testing: Datazilla has finally been decommissioned (R.I.P.), in favor of our newest performance analysis tool, Perfherder.  A lot of Talos documentation updates have been made at, including details about how we perform calculations on data produced by Talos.  wlach performed a useful post-mortem of Eideticker, with several takeaways which should be applicable to many other projects.

MozReview and Autoland: There’s a MozReview meetup underway, so expect some cool updates next time!

TaskCluster Support: ted has made a successful cross-compiled OSX build using TaskCluster!  Take it for a spin.  More work is needed before we can move OSX builds from the mac mini builders to the cloud.

Mobile Automation: gbrown continues to make improvements on the new |mach emulator| command which makes running Android tests locally on emulator very simple.

General Automation: run-by-dir is live on opt mochitest-plain; debug and ASAN coming soon.  This reduces test “bleed-through” and makes it easier to change chunking.  adusca, our Outreachy intern, is working to integrate the try extender into Treeherder.  And ahal has merged the mozharness “in-tree” configs with the regular mozharness config files, now that mozharness lives in the tree.

Firefox Automation: YouTube ad detection has been improved for firefox-media-tests by maja, which fixes the source of the top intermittent failure in this suite.

Bughunter: bc has got asan-opt builds running in production, and is working on gtk3 support. gps has enabled syntax highlighting in hgweb, and has added a new JSON API as well.  See gps’ blog post.

The Details
Perfherder/Performance Testing
TaskCluster Support
Mobile Automation
Firefox and Media Automation
General Automation
  •  run-by-dir is live for mochitest-plain (opt only); debug is coming soon, followed by ASAN.
  • Mozilla CI tools is moving from using BuildAPI as the scheduling entry point to use TaskCluster’ scheduling. This work will allow us to schedule a graph of buildbot jobs and their dependencies in one shot.
  • adusca is integrating into treeherder the ability to extend the jobs run for any push. This is based on the prototype. Follow along in
  • Git was deployed to the test machines. This is necessary to make the Firefox UI update tests work on them.
  • [ahal] merge mozharness in-tree configs with the main mozharness configs
  • Bug fixes to the ETL – fix bad lookups on hg repo, mostly l10n builds 
  • More error reporting on ETL – Structured logging has changed a little, handle the new variations, and be more elegant when it comes to unknowns, an complain when there is non-conformance.
  • Some work on adding hg `repo` table – acts as a cache for ETL, but can be used to calculate ‘per push’ statistics on OrangeFactor data.
  • Added Talos to the `perf` table – used the old Datazilla ETL code to fill the ES cluster.  This may speed up extracting the replicates, for exploring the behaviour of a test.
  • Enable deep queries – Effectively performing SQL join on Elasticsearch – first attempt did too much refactoring.  Second attempt is simpler, but still slogging through all the resulting test breakage
  • Updated 
  • [ahal] helped review and finish contributor patch for switching marionette_client from optparse to argparse
  • Corrected UUID used for session ID and element IDs
  • Updated dispatching of various marionette calls in Gecko
  • [bc] Have asan-opt builds running in production. Finalizing patch. Still need to build gtk3 for rhel6 32bit in order to stop using custom builds and support opt in addition to debug.
  • Updated the hierarchical burndowns to EPM’s important metabugs that track features 
  • More config changes

August 26, 2015 11:47 PM

August 25, 2015

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

August 25, 2015 06:25 AM

August 18, 2015

Cameron Dawson

Treeherder Job Counts and Chunks

The Treeherder production instance has a significant front-end change arriving in the coming days. We are condensing job groups and chunks (a job which, for individual team or project reasons, were separated into numerous multiple jobs) into a ‘+n’ count representation in the job group.

To see the motivation for this change, one need only look at this before/after example of the same job-sets.  The number of chunks has begun (and will continue to) grow.  As this happens, the Treeherder page is becoming more cluttered.  The job aggregation to counts will streamline the initial view, while still giving easy access to individual job chunks as you need them.


Screenshot 2015-08-18 08.17.13

Wow.  That’s a lot of chunks.


Screenshot 2015-08-18 08.17.30

Whew!  To be sure, this is an extreme example.  But even in less extreme chunk counts, the difference is significant overall.


Counts can be expanded/collapsed Globally via our new navbar button:


Counts can also be expanded Locally either by clicking the ‘+n’ job count…


…or counts can be expanded and collapsed Locally by clicking the job group:


When a job is selected and its count collapsed, the ‘+n’ count receives a selected style until either a different job is selected, or the selection is cleared.

Classification and unclassified failure and next/previous job navigation should remain unchanged.  As new jobs are created or their state changes, you will notice the counts updating accordingly.


[Glaring Omission from the original post:] I neglected to mention that this was all Ryan VanderMeulen’s idea.  He approached the Treeherder team at Whistler 2015 with this idea and after chatting about it for a few minutes, this just seemed like an “ah hah” moment.  I began working on it that afternoon…

I received opinions and feedback from most of the Treeherder team, Sheriffs and Blake Winton.  But I wanted to give a special shout-out to Jonathan French.  His testing and design help GREATLY improved the quality of this feature!  Thanks Jon!

Contact us

If you have any comments/suggestions about this change, please feel free to visit us on IRC in the #treeherder channel.


-Cam [:camd]

August 18, 2015 04:30 PM

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

August 18, 2015 07:58 AM

August 13, 2015

Jonathan Griffin

Engineering Productivity Update, August 13, 2015

From Automation and Tools to Engineering Productivity

Automation and Tools” has been our name for a long time, but it is a catch-all name which can mean anything, everything, or nothing, depending on the context. Furthermore, it’s often unclear to others which “Automation” we should own or help with.

For these reasons, we are adopting the name “Engineering Productivity”. This name embodies the diverse range of work we do, reinforces our mission (, promotes immediate recognition of the value we provide to the organization, and encourages a re-commitment to the reason this team was originally created—to help developers move faster and be more effective through automation.

The “A-Team” nickname will very much still live on, even though our official name no longer begins with an “A”; the “get it done” spirit associated with that nickname remains a core part of our identity and culture, so you’ll still find us in #ateam, brainstorming and implementing ways to make the lives of Mozilla’s developers better.


Treeherder: Most of the backend work to support automatic starring of intermittent failures has been done. On the front end, several features were added to make it easier for sheriffs and others to retrigger jobs to assist with bisection: the ability to fill in all missing jobs for a particular push, the ability to trigger Talos jobs N times, the ability to backfill all the coalesced jobs of a specific type, and the ability to retrigger all pinned jobs. These changes should make bug hunting much easier.  Several improvements were made to the Logviewer as well, which should increase its usefulness.

Perfherder and performance testing: Lots of Perfherder improvements have landed in the last couple of weeks. See details at wlach’s blog post.  Meanwhile, lots of Talos cleanup is underway in preparation for moving it into the tree.

MozReview: Some upcoming auth changes are explained in mcote’s blog post.

Mobile automation: gbrown has converted a set of robocop tests to the newly enabled mochitest-chrome on Android. This is a much more efficient harness and converting just 20 tests has resulted in a reduction of 30 minutes of machine time per push.

Developer workflow: chmanchester is working on building annotations into files that will automatically select or prioritize tests based on files changed in a commit. See his blog post for more details. Meanwhile, armenzg and adusca have implemented an initial version of a Try Extender app, which allows people to add more jobs on an existing try push. Additional improvements for this are planned.

Firefox automation: whimboo has written a Q2 Firefox Automation Report detailing recent work on Firefox Update and UI tests. Maja has improved the integration of Firefox media tests with Treeherder so that they now officially support all the Tier 2 job requirements.

WebDriver and Marionette: WebDriver is now officially a living standard. Congratulations to David Burns, Andreas Tolfsen, and James Graham who have contributed to this standard. dburns has created some documentation which describes which WebDriver endpoints are implemented in Marionette.

Version control: The ability to read and extra metadata from files has been added to This opens the door to cool future features, like the ability auto file bugs in the proper component and automatically selecting appropriate reviewers when pushing to MozReview. gps has also blogged about some operational changes to which enables easier end-to-end testing of new features, among other things.

The Details
Treeherder/Automatic Starring
Treeherder/Front End
  • Several retrigger features were added to Treeherder to make merging and bisections easier:  auto fill all missing/coalesced jobs in a push; trigger all Talos jobs N times; backfill a specific job by triggering it on all skipped commits between this commit and the commit that previously ran the job, retrigger all pinned jobs in treeherder.  This should improve bug hunting for sheriffs and developers alike.
  • [jfrench] Logviewer ‘action buttons’ are now centralized in a Treeherder style navbar
  • [jfrench] Logviewer skipped steps are now recognized as non-failures and presented as blue info steps,
  • [jfrench] Middle-mouse-clicking on a job in treeherder now launches the Logviewer
  • [vaibhav] Added the ability to retrigger all pinned jobs (bug
  • Camd’s job chunking management will likely land next week
Perfherder/Performance Testing
  • [wlach] / [jmaher] Lots of perfherder updates, details here: Highlights below
  • [wlach] The compare pushes view in Perfherder has been improved to highlight the most important information.
  • [wlach] If your try push contains Talos jobs, you’ll get a url for the Perfherder comparison view when pushing (
  • [jmaher/wlach] Talos generates suite and test level metrics and perfherder now ingests those data points. This fixes results from internal benchmarks which do their own summarization to report proper numbers.
  • [jmaher/parkouss] Big talos updates (thanks to :parkouss), major refactoring, cleanup, and preparation to move talos in tree.
Mobile Automation
Dev Workflow
  • [ahal] Created patch to clobber compiled python files in srcdir
  • [ahal] More progress on mach/mozlog patch
  • [chmanchester] Fix to allow ‘mach try’ to work without test arguments (bug 1192484)
Media Automation
  • [maja_zf] firefox-media-tests ‘log steps’ and ‘failure summaries’ are now compatible with Treeherder’s log viewer, making them much easier to browse. This means the jobs now satisfy all Tier-2 Treeherder requirements.
  • [sydpolk] Refactoring of tests after fixing stall detection is complete. I can now take my network bandwidth prototype and merge it in.
Firefox Automation
General Automation
  • Finished adapting mozregression ( and mozdownload ( to S3.
  • (Henrik) Isn’t only a temporary solution, before we move to TC?
  • (armenzg) I believe so but for the time being I believe we’re out of the woods
  • The manifestparser dependency was removed from mozprofile (bug 1189858)
  • [ahal] Fix for
  • [sydpolk] Platform Jenkins migration to the SCL data center has not yet begun in earnest due to PTO. Hope to start making that transition this week.
  • [chmanchester] work in progress to build annotations in to files to automatically select or prioritize tests based on what changed in a commit. Strawman implementation posted in , blog post about this work at
  • [adusca/armenzg] Try Extender ( ) is open for business, however, a new plan will soon be released to make a better version that integrates well with Treeherder and solves some technichal difficulties we’re facing
  • [armenzg] Code has landed on mozci to allow re-triggering tasks on TaskCluster. This allows re-triggering TaskCluster tasks on the try server when they fail.
  • [armenzg] Work to move Firefox UI tests to the test machines instead of build machines is solving some of the crash issues we were facing
  • [ahal] re-implemented test-informant to use ActiveData:
  • [ekyle] Work on stability: Monitoring added to the rest of the ActiveData machines.  
  • [ekyle] Problem:  ES was not balancing the import workload on the cluster; probably because ES assumes symmetric nodes, and we do not have that.  The architecture was changed to prefer a better distribution of work (and query load) – There now appears to be less OutOfMemoryExceptions, despite test-informant’s queries.
  • [ekyle] More Problems:  Two servers in the ActiveData complex failed: The first was the ActiveData web server; which became unresponsive, even to SSH.  The machine was terminated.  The second server was the ‘master’ node of the ES cluster: This resulted in total data loss, but it was expected to happen eventually given the cheap configuration we have.   Contingency was in place:  The master was rebooted, the  configuration was verified, and data re-indexed from S3.   More nodes would help with this, but given the rarity of the event, the contingency plan in place, and the low number of users, it is not yet worth paying for. 
WebDriver (highlights)
  • [ato] WebDriver is now officially a living standard (
  • [ato] Rewrote chapter on extending the WebDriver protocol with vendor-specific commands
  • [ato] Defined the Get Element CSS Value command in specification
  • [ato] Get Element Attribute no longer conflates DOM attributes and properties; introduces new command Get Element Property
  • [ato] Several significant infrastructural issues with the specification was fixed
  • Project managers for FxOS have a renewed interest in the project tracking, and overall status dashboards.   Talk only, no coding yet.

August 13, 2015 11:17 PM

August 12, 2015

Andreas Tolfsen

The sorry state of women in tech

I read the anonymous blog post Your Pipeline Argument is Bullshit with great interest. While it’s not very diplomatic and contains a great deal of ranting, it speaks about something very important: Women in tech. I read it more as an expression of frustration, than a mindless flamatory rant as others have indicated.

On the disproportion of women granted CS degrees in the US:

I use this data because I graduated with a CS degree in 2000. Convenient. The 2000s is also when the proportion of women getting CS degrees started falling even faster. My first job out of school had an older woman working there who let me know back in the 80s there were a lot more women. Turns out, in the early 80s, almost 40% of CS degrees in the US were granted to women.

In 2009 only about 18% women graduated with CS degrees.

Although purely anecdotal evidence, my experience after working seven years as a professional programmer is that more women than men leave the industry to pursue other careers. In fact the numbers show that the cumulative pool of talent (meaning all who hold CS degrees) is close to 25% women.

We have to recognise what a sorry state our industry is in. Callous and utilitarian focus on technology alone, as opposed to building a healthy discourse and community around computing that is inclusive to women, does more harm than we can imagine.

Because men hire men, and especially those men similar to ourselves, I’m in favour of positive action to balance the numbers.

Whoever is saying that we should hire based on talent alone, there’s little evidence to support that 25% of the available CS talent is any worse than the remaining three quarters.

August 12, 2015 06:27 PM

mozregression updates

Release 0.41

mozregression 0.41 fixes bug 1185756, which caused the following warning on Windows:

WindowsError: [Error 32] The process cannot access the file
because it is being used by another process

Also, mozregression is now more tolerant when facing failures like a build not installable or a crash from the tested application (see bug 1192485).

And finally, when using mozregression with a custom profile it won’t be modified anymore (see bug 999009). This was a longstanding bug - thanks to Jonathan Pigree for his help in fixing it!

August 12, 2015 12:00 AM

August 11, 2015

Henrik Skupin

Firefox Automation report – Q2 2015

It’s been a while since I wrote my last Firefox automation report, so lets do another one to wrap up everything happened in Q2 this year. As you may remember from my last report the team has been cut down to only myself, and beside that I was away the whole April. Means only 2 months worth of work will be covered this time.

In general it was a tough quarter for me. Working alone and having to maintain all of our infrastructure and keeping both of our tests (Mozmill and Marionette) green was more work as expected. But it still gave me enough time to finish my two deliverables:

Firefox UI Test Results on Treeherder

With the transition of Mozmill tests to Marionette we no longer needed the mozmill-dashboard. Especially not since nearly no job in our Mozmill CI is running Mozmill anymore. Given that we also weren’t happy with the dashboard solution and the usage of couchdb under the hood, I was happy that a great replacement exist and our Marionette tests can make use of.

Treeherder is the new reporting system for tests being run in buildbot continuous integration. Because of that it covers all products and their appropriate supported branches. That’s why it it’s kinda perfect for us.

Before I was able to get started a bit of investigation had to be done, and I also checked some other projects which make use of treeherder reporting. That information was kinda helpful even with lots of undocumented code. Given that our tests are located outside of the development tree in its own Github repository some integration can be handled more loosely compared to in-tree tests. With that respect we do not appear as Tier-1 or Tier-2 but results are listed under the Tier-3 level, which is hidden by default. But that’s fine given that our goal was to bring up reports to treeherder first. Going into details will be a follow-up task.

For reporting results to Treeherder the Python package treeherder-client can be used. It’s a collection of different classes which help with authentication, collecting job details, and finally uploading the data to the server. It’s documentation can be found on readthedocs and meanwhile contains a good number of example code which makes it easier to get your code implemented.

Before you can actually upload reports the names and symbols of the groups and jobs have to be defined. For our tests I have chosen the three group symbols “Fu”, “Ff”, and “Fr”. Each of those stays for “(F)irefox UI Tests” and the first letter of the testrun name. We currently have “updates”, “functional”, and “remote”. As job name the locale of Firefox will be used. That results in an output like the following:

Treeherder Results

Whether jobs are passing or failing it is recommended to always add the log files as generated by running the tests to the report. It will not happen via data URLs but as artifacts with a link to an upload location. In our case we make use of Amazon S3 as storage service.

With all the pieces implemented we can now report to treeherder and covering Nightly, Aurora (Developer Edition), Beta, and Release builds. As of now reporting only happens against the staging instance of Treeherder, but soon we will be able to report to production. If you want to have a sneak peak how it works, just follow this link for Nightly builds.

More details about Treeherder reporting I will do later this quarter when the next pieces have been implemented.

Firefox UI Update Tests under RelEng

My second deliverable was to assist Armen Zambrano in getting the Firefox UI Update tests run for beta and release builds on Release Engineering infrastructure. This was a kinda important goal for us given that until now it was a manually triggered process with lots of human errors on our own infrastructure. That means lots of failures if you do not correctly setup the configuration for the tests, and a slower processing of builds due to our limited available infrastructure. So moving this out of our area made total sense.

Given that Armen had already done a fair amount of work when I came back from my PTO, I majorly fixed issues for the tests and the libraries as pointed out by him. All that allowed us to finally run our tests on Release Engineering infrastructure even with a couple of failures at the beginning for the first beta. But those were smaller issues and got fixed quickly. Since then we seem to have good results. If you want to have a look in how that works, you should check the Marionette update tests wiki page.

Sadly some of the requirements haven’t been completely finished yet. So the Quality Engineering team cannot stop running the tests themselves. But that will happen once bug 1182796 has been fixed and deployed to production.

Oh, and if you wonder where the results are located… Those are not getting sent to Treeherder but to an internal mailing list as used for every other automation results.

Other Work

Beside the deliverables I got some more work done. Mainly for the firefox-ui-tests and mozmill-ci.

While the test coverage has not really been increased, I had a couple of regressions to fix as caused by changes in Firefox. But we also landed some new features thankfully as contributed by community members. Once all that was done and we agreed to have kinda stable tests, new branches have been created in the repository. That was necessary to add support for each supported version of Firefox down to ESR 38.0, and to be able to run the tests in our Mozmill CI via Jenkins. More about that you will find below. The only task I still haven’t had time for yet was the creation of proper documentation about our tests. I hope that I will find the time in Q3.

Mozmill CI got the most changes in Q2 compared to all the former quarters. This is related to the transition from Mozmill tests to Marionette tests. More details why we got rid of Mozmill tests can be found in this post. With that we decided to get rid of most of the tests and mainly start from scratch by only porting the security and update tests over to Marionette. The complete replacement in Mozmill and all its jobs can be seen on issue 576 on Github. In detail we have the following major changes:

All changes in Mozmill CI can be seen on Github.

Last but not least we also had two releases of mozdownload in Q2. Both had a good amount of features included. For details you can check the changelog.

I hope that gave you a good quick read on the stuff I was working on last quarter. Maybe in Q3 I will find the time to blog more often and in more detail. Lets see.

August 11, 2015 02:20 PM

Dan Minor

Importing GitHub Pull Requests in MozReview

One of the workflows we’ve been working on supporting for MozReview is importing GitHub pull requests for Gecko reviews. This will hopefully remove barriers for new contributors to Gecko by allowing them to use a familiar workflow for the initial development of their patches but still integrating their work into Bugzilla and official review channels. The screenshot below shows the initial user interface for this feature:

import pull request screenshot

A listener for GitHub requests is under development by Mauro Doglio which will look for pull requests on configured repositories and write an url into them which will allow for the pull request to be imported into MozReview.

The import itself is handled by Autoland. It retrieves the patches from GitHub, imports them into a local clone of mozilla-central, and then pushes them for review. We’ve decided to squash pull requests to a single commit prior to pushing to MozReview. This avoids dealing with history editing workflows that would potentially confuse MozReview, such as re-ordering commits. It fits with the common GitHub pattern of pushing follow on commits to address review comments. Once the commit is imported the user will be redirected to the review page and can add reviewers and publish their review as usual.

If a bug id is specified in the GitHub Issue title, we will use that when pushing to MozReview. Otherwise, we’ll automatically file a bug on the user’s behalf. At the moment the bug will have defaults for the product and the component. Once a bug is associated with a pull request, the bug will be reused for subsequent imports of the same pull request.

One complication of this work is the support for autolanding to Try. Because pushing to MozReview and Try both require L1 access we previously did not have to worry about checking permissions for Autoland requests, but this is no longer the case once people are allowed to create reviews through GitHub. Steven MacLeod has added LDAP permissions checks so we can disable the Autoland to Try functionality for unauthorized users. This work is also an important prerequisite for supporting autolanding to an inbound tree.

We’ve begun deploying this work to the development instances of MozReview and Autoland. Once all of the components are ready and we’ve shaken out a few more bugs, we’ll be enabling it on production as well.

In the future we hope to support non-squashed commits. This will involve writing metadata into the commit summary that allows for a commit to be associated with a review. We’re also planning to add more intelligent handling of automatic bug filing by utilizing metadata which already exists in the tree for associating files with a product and component.

August 11, 2015 08:00 AM

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

August 11, 2015 07:41 AM

August 10, 2015

Alice Scarpa

Extending Try pushes

Are you tired of having to do a new try push just because you forgot a test? Your problems are over! Meet Try Extender. This new web app allows you to add new jobs to existing try pushes, all you have to do is log in, choose a revision and pick the jobs you want!

OK, now I’m done with my parody informercial.

Try Extender is a web app running on Heroku that uses mozci to trigger new jobs for a try push. It is still a prototype, but if you want to extend a push you are welcomed to try it out! Currently it can only add new test jobs to completed build jobs or add new build jobs.

Right now the main goal is just to validate the idea. Depending on how this experiment goes we might work on integrating it into Treeherder, or leave it as a stand-alone web app.

August 10, 2015 12:00 AM

August 07, 2015

William Lachance

More Perfherder updates

Since my last update, we’ve been trucking along with improvements to Perfherder, the project for making Firefox performance sheriffing and analysis easier.

Compare visualization improvements

I’ve been spending quite a bit of time trying to fix up the display of information in the compare view, to address feedback from developers and hopefully generally streamline things. Vladan (from the perf team) referred me to Blake Winton, who provided tons of awesome suggestions on how to present things more concisely.

Here’s an old versus new picture:

Screen Shot 2015-07-14 at 3.53.20 PM Screen Shot 2015-08-07 at 1.57.39 PM

Summary of significant changes in this view:

The point of these changes isn’t necessarily to make everything “immediately obvious” to people. We’re not building general purpose software here: Perfherder will always be a rather specialized tool which presumes significant domain knowledge on the part of the people using it. However, even for our audience, it turns out that there’s a lot of room to improve how our presentation: reducing the amount of extraneous noise helps people zero in on the things they really need to care about.

Special thanks to everyone who took time out of their schedules to provide so much good feedback, in particular Avi Halmachi, Glandium, and Joel Maher.

Of course more suggestions are always welcome. Please give it a try and file bugs against the perfherder component if you find anything you’d like to see changed or improved.

Getting the word out

Hammersmith:mozilla-central wlach$ hg push -f try
pushing to ssh://
no revisions specified to push; using . to avoid pushing multiple heads
searching for changes
remote: waiting for lock on repository /repo/hg/mozilla/try held by ''
remote: got lock after 4 seconds
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 1 changes to 1 files
remote: Trying to insert into pushlog.
remote: Inserted into the pushlog db successfully.
remote: View your change here:
remote: Follow the progress of your build on Treeherder:
remote: It looks like this try push has talos jobs. Compare performance against a baseline revision:

Try pushes incorporating Talos jobs now automatically link to perfherder’s compare view, both in the output from mercurial and in the emails the system sends. One of the challenges we’ve been facing up to this point is just letting developers know that Perfherder exists and it can help them either avoid or resolve performance regressions. I believe this will help.

Data quality and ingestion improvements

Over the past couple weeks, we’ve been comparing our regression detection code when run against Graphserver data to Perfherder data. In doing so, we discovered that we’ve sometimes been using the wrong algorithm (geometric mean) to summarize some of our tests, leading to unexpected and less meaningful results. For example, the v8_7 benchmark uses a custom weighting algorithm for its score, to account for the fact that the things it tests have a particular range of expected values.

To hopefully prevent this from happening again in the future, we’ve decided to move the test summarization code out of Perfherder back into Talos (bug 1184966). This has the additional benefit of creating a stronger connection between the content of the Talos logs and what Perfherder displays in its comparison and graph views, which has thrown people off in the past.

Continuing data challenges

Having better tools for visualizing this stuff is great, but it also highlights some continuing problems we’ve had with data quality. It turns out that our automation setup often produces qualitatively different performance results for the exact same set of data, depending on when and how the tests are run.

A certain amount of random noise is always expected when running performance tests. As much as we might try to make them uniform, our testing machines and environments are just not 100% identical. That we expect and can deal with: our standard approach is just to retrigger runs, to make sure we get a representative sample of data from our population of machines.

The problem comes when there’s a pattern to the noise: we’ve already noticed that tests run on the weekends produce different results (see Joel’s post from a year ago, “A case of the weekends”) but it seems as if there’s other circumstances where one set of results will be different from another, depending on the time that each set of tests was run. Some tests and platforms (e.g. the a11yr suite, MacOS X 10.10) seem particularly susceptible to this issue.

We need to find better ways of dealing with this problem, as it can result in a lot of wasted time and energy, for both sheriffs and developers. See for example bug 1190877, which concerned a completely spurious regression on the tresize benchmark that was initially blamed on some changes to the media code– in this case, Joel speculates that the linux64 test machines we use might have changed from under us in some way, but we really don’t know yet.

I see two approaches possible here:

  1. Figure out what’s causing the same machines to produce qualitatively different result distributions and address that. This is of course the ideal solution, but it requires coordination with other parts of the organization who are likely quite busy and might be hard.
  2. Figure out better ways of detecting and managing these sorts of case. I have noticed that the standard deviation inside the results when we have spurious regressions/improvements tends to be higher (see for example this compare view for the aforementioned “regression”). Knowing what we do, maybe there’s some statistical methods we can use to detect bad data?

For now, I’m leaning towards (2). I don’t think we’ll ever completely solve this problem and I think coming up with better approaches to understanding and managing it will pay the largest dividends. Open to other opinions of course!

August 07, 2015 08:04 PM

Geoff Brown

mochitest-chrome tests for Android

Support for mochitest-chrome tests on Android was recently improved and mochitest-chrome tests can now be seen on treeherder (the M(c) job) for all Firefox for Android builds. Most mochitest-chrome tests are desktop-specific or cross-platform; most Firefox for Android tests have been written for Robocop, our Robotium-based Android UI test framework.

I noticed that some Robocop tests were implemented almost entirely in Javascript and could easily be converted to mochitest-chrome, where tests would run much more efficiently. Bug 1184186 converted about 20 such Robocop tests to mochitest-chrome, reducing our Robocop load by about 30 minutes while increasing mochitest-chrome load by only about 3 minutes. (We save time by not starting and stopping the browser between mochitests, and not waiting around for state changes, as frequently required in UI tests.)

The “new” mochitest-chrome tests are all located in mobile/android/tests/browser/chrome and can also be run locally from mach. Just make sure you have:

Here is a screen recording showing the new tests in action:

Want to write your own mochitest-chrome test for Firefox for Android? Be sure to see for general advice. If you are looking for an example, is one of the simplest tests — a great place to start.

As always, be sure to see for detailed information about running tests for Firefox for Android, and ping me on irc if you run into any trouble.

August 07, 2015 04:50 PM

August 06, 2015

Christopher Manchester

Defining Semi-Automatic Test Prioritization

We run a lot of tests against Firefox, and running them takes a long time. Ideally, a best effort test selection is done on try, and if nothing obvious shakes out of that a change is landed on inbound. Inbound runs a selection of test jobs pared down based on coarse historical data (SETA). Modulo a handful of periodic build types, the remaining integration branches run a full set of tests and builds.

With the exception of the (entirely optional) selection made by try syntax, none of these steps codify the idea that some tests might fail given a particular change, and others are unlikely to fail or never will.

This post outlines one approach to implementing smarter (semi-automatic) test selection for Gecko.



Here’s a sketch of how this could work. This list doesn’t have a lot of regard for what I think is feasible in the short term. In particular, code coverage data for Mozilla’s code base is not readily available.

After this process, platforms and build types are still not known. Build types are things like opt, debug, and pgo, while platforms are things like Windows or B2G. In general we can’t expect a set of changed files to suggest a test platform and build type (many changes will impact code that runs across platforms and build types), but when all changes are in a subdirectory known to impact a single platform, we can be confident selecting only that platform. I expect these classifications are coarse and stable enough to capture effectively with annotations in files.

This is the framework I’m planning to use to get this off the ground in the coming months. Initial work around fleshing out file metadata is tracked in bug 1184405.

Feedback is welcome and appreciated.

August 06, 2015 05:50 PM

August 04, 2015

Mark Côté

MozReview auth changes

MozReview will soon be using Bugzilla’s new OAuth-like1 API keys and auth delegation. This is long overdue, and, in addition to providing security benefits, will eliminate all those confusing session-expired errors (e.g. bug 1178814).

After we deploy the change, all users will need to log back into MozReview’s Review Board2 instance. This time, rather than entering your Bugzilla credentials directly into Review Board, when you go to the “Log In” page, you’ll be redirected to BMO. If you don’t have a current BMO session, you’ll have to log into BMO. After you log in, or immediately after being redirected to BMO if you do have a session, you’ll be redirected back to Review Board and logged in. This is because, unlike most third-party apps, MozReview’s Review Board is a trusted app that is tightly integrated to BMO, so you won’t be confronted with the standard “Auth Delegation Request” intermediate page.

This is the first stage of conversion to API keys. For pushing review requests with Mercurial, you will still have to have either your Bugzilla username and password or your cookies in your .hgrc file or enter them on the command line at push time. However, Review Board will no longer store cookies; the username/password or login cookies will only be passed to BMO for verification and then discarded. We’ll be moving to API key usage on the command line in a subsequent patch.

Through API keys, Review Board will only have access to the specific BMO APIs required by MozReview. Those actions are mainly restricted to creating and updating attachments and posting comments; however, it will also need access to the login API until we support API keys on the command line. As noted, this will be used solely for identification, and no login tokens will be stored in MozReview.

Another big benefit of API keys is the elimination of those annoying and confusing expired-session errors. The BMO cookies used by MozReview have a limited lifespan, but API keys are good until explicitly revoked by the user. You can see the API key that is transferred to MozReview, as well as any other API keys you’ve manually or automatically created, in the API Keys tab in your BMO preferences. Revoking the API key won’t automatically log you out of Review Board, but you won’t be able to do any actions that interact with BMO (most actions) unless you log out and back in again (thus generating a new key).

You can follow along progress in bug 993233.

  1. No, it’s not exactly OAuth, but it’s based on similar ideas. We haven’t found a good OAuth 2 library for use with BMO, but we’re looking around.

  2. A note about names: MozReview generally refers to the full code-review system, which is primarily an hg server and a Review Board installation with extensions that we’ve developed. It also includes BMO, Autoland, Pulse, LDAP, and little things like code-review bots. When we say “Review Board”, we are referring specifically to the web app, which is the primary user interface to MozReview.

August 04, 2015 05:18 PM

Andreas Tolfsen

WebDriver now a living standard

The WebDriver specification is now officially a living standard. Practically this means that all changes are automatically published to

This brings an end to the era of forever outdated (two years in our case!) technical reports. It also helps bridge the disconnect many readers were having when they looked for information on our specification.

This is made possible with the Echidna tool that has recently been developed at the W3C. It integrates with Github and Travis, and lets you trigger the publishing steps when changes land on a specific branch in your source repository.

A possible future enhancement is abandoning the now superfluous master branch in favour of making the autopublishing gh-pages the default. The two-step landing process seems more tuned towards a levelled Editor’s Draft-to-Working Draft model.

Thanks to tripu and Michael[tm] Smith for doing the legwork.

August 04, 2015 09:49 AM

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

August 04, 2015 07:41 AM

July 31, 2015

Vaibhav Agrawal

First Month of Mozilla Internship

It has been a month since I started my Mozilla internship in San Francisco, but it feels like I had just started yesterday. I have been interning with the Automation and Tools team this summer and it has been a wonderful experience. As an intern in Mozilla, one gets goodies, new laptop, free food, and there are various intern events in which one gets to take part in. My mentor @chmanchester also gave me the freedom to decide what I wanted to work on which is quite unlike some other companies.

I chose to work on tools to improve developer productivity and making libraries that I have worked on in the past, more relevant. In the past month, I have been working on getting various triggering options inside Treeherder, which is a reporting dashboard for checkins to Mozilla projects. This involved writing AngularJS code for the front-end UI and python code for the backend. The process involved publishing the “triggering actions” to pulse, listening to those actions and then use the mozci library on the backend to trigger jobs. Currently if developers, particularly the sheriffs, want to trigger a lot of jobs, they have to do it via command line, and it involves context switching plus typing out the syntax.  To improve that, this week we have deployed three new options in Treeherder that will hopefully save time and make the process easier. They are:

* Trigger_Missing_Jobs: This button is to ensure that there is one job for every build in a push. The problem is that on integration branches we coalesce jobs to minimize load during peak hours, but many times there is a regression and sheriffs need to trigger all jobs for the push. That is when this button will come in handy, and one will be able to trigger jobs easily.

* Trigger_All_Talos_Jobs: As the name suggests, this button is to trigger all the talos jobs on a particular push. Being a perf sheriff, I need to trigger all talos jobs a certain number of times for a regressing push to get more data, and this button will aid me and others in doing that.

Screen Shot 2015-07-30 at 22.30.56Fig: shows “Trigger Missing Jobs” and “Trigger All Talos Jobs” buttons in the Treeherder UI

* Backfill_Job: This button is to trigger a particular type of job till the last known push on which that job was run. Due to coalescing certain jobs are not run on a series of pushes, and when an intermittent or bustage is detected, sheriffs need to find the root cause and thus manually trigger those jobs for the pushes. This button should aid them in narrowing down the root regressing revision for a job that they are investigating.

Screen Shot 2015-07-31 at 09.39.43Fig: shows “Backfill this job” button in the Treeherder UI

All of the above features right now only work for jobs that are on buildapi, but when mozci will have the ability to trigger tasks on taskcluster, they will be able to work on those jobs too. Also, right now all these buttons trigger the jobs which have a completed build, in future I plan to make these buttons to also trigger jobs which don’t have an existing build. These features are in beta and have been released for sheriffs, I would love to hear your feedback! A big shout out to all the people who have reviewed, tested and given feedback on these features.

July 31, 2015 04:48 PM

mozregression updates

Release 0.40

This is it, now mozregression uses fully the new build storage!

mozregression now downloads builds from In many cases this should be faster than before. Good news!

Also this 0.40 release fixes an intermittent windows bug introduced with 0.37, see bug 1185756.

July 31, 2015 12:00 AM

July 30, 2015

David Burns

Another Marionette release! Now with Windows Support!

If you have been wanting to use Marionette but couldn't because you only work on Windows, now is your chance to do so! All the latest downloads are available from our development github repository releases page

There is also a new page on MDN that walks you through the process of setting up Marionette and using it. I have only updated the python bindings so I can get a fell for how people are using it

Since you are awesome early adopters it would be great if we could raise bugs.

I am not expecting everything to work but below is a quick list that I know doesn't work.

Switching of Frames needs to be done with either a WebElement or an index. Windows can only be switched by window handles. This is currently how it has been discussed in the specification.

If in doubt, raise bugs!

Thanks for being an early adopter and thanks for raising bugs as you find them!

July 30, 2015 11:45 AM

Jonathan Griffin

A-Team Update, July 29, 2015


Treeherder: We’ve added to mozlog the ability to create error summaries which will be used as the basis for automatic starring.  The Treeherder team is working on implementing database changes which will make it easier to add support for that.  On the front end, there’s now a “What’s Deployed” link in the footer of the help page, to make it easier to see what commits have been applied to staging and production.  Job details are now shown in the Logviewer, and a mockup has been created of additional Logviewer enhancements; see bug 1183872.

MozReview and Autoland: Work continues to allow autoland to work on inbound; MozReview has been changed to carry forward r+ on revised commits.

Bugzilla: The ability to search attachments by content has been turned off; BMO documentation has been started at

Perfherder/Performance Testing: We’re working towards landing Talos in-tree.  A new Talos test measuring tab-switching performance has been created (TPS, or Talos Page Switch); e10s Talos has been enabled on all platforms for PGO builds on mozilla-central.  Some usability improvements have been made to Perfherder –

TaskCluster: Successful OSX cross-compilation has been achieved; working on the ability to trigger these on Try and sorting out details related to packaging and symbols.  Work on porting Linux tests to TaskCluster is blocked due to problems with the builds.

Marionette: The Marionette-WebDriver proxy now works on Windows.  Documentation on using this has been added at

Developer Workflow: A kill_and_get_minidump method has been added to mozcrash, which allows us to get stack traces out of Windows mochitests in more situations, particularly plugin hangs.  Linux xpcshell debug tests have been split into two chunks in buildbot in order to reduce E2E times, and chunks of mochitest-browser-chrome and mochitest-devtools-chrome have been re-normalized by runtime across all platforms.  Now that mozharness lives in the tree, we’re planning on removing the “in-tree configs”, and consolidating them with the previously out-of-tree mozharness configs (bug 1181261).

Tools: We’re testing an auto-backfill tool which will automatically retrigger coalesced jobs in Treeherder that precede a failing job.  The goal is to reduce the turnaround time required for this currently manual process, which should in turn reduce tree closure times related to test failures

The Details

Treeherder/Automatic Starring

Treeherder/Front End

Perfherder/Performance Testing


TaskCluster Support

Mobile Automation

Dev Workflow

Media Automation

General Automation



July 30, 2015 12:31 AM

July 29, 2015

Joel Maher

Lost in data – episode 2 – bisecting and comparing

This week on Lost in Data, we tackle yet another pile of alerts.  This time we have a set of changes which landed together and we push to try for bisection.  In addition we have an e10s only failure which happened when we broke talos uploading to perfherder.  See how I get one step closer to figuring out the root cause of the regressions.

July 29, 2015 11:53 PM

July 28, 2015

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

July 28, 2015 05:21 AM

July 27, 2015

Armen Zambrano G. (@armenzg)

Enabling automated back-filling on mozilla-inbound for few hours

tl;dr; we're going to enable automated back-filling tomorrow Tuesday for
few hours on mozilla-inbound.

We were aiming for Monday but pushed it to Tuesday to help publicize this more.

If on Wednesday there are no fall-outs we will leave it running for m-i for a week before enabling it on other places.

Posted on various mailing lists including

> Hello all,
> We are planning to turn on a service that automatically backfills
> failed test jobs on m-i. If there are no concerns, we would like to
> turn this on experimentally for a couple of hours on [Tuesday]. We
> hope this will make it easier to identify which revision broke a
> test. Suggestions are welcome.
> The backfilling works like this: - It triggers the job that failed
> one extra time - Then it looks for a successful run of the job on the
> previous 5 revisions. If a good run is found, it triggers the job
> only on revisions up to this good run. If not, it triggers the job on
> every one of the previous 5 revisions. Previous jobs will be
> triggered one time.
> The tracking bug is:
> Best, Alice

Zambrano Gasparnian, Armen
Automation & Tools Engineer

Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

July 27, 2015 06:56 PM

Christopher Manchester

Introducing mach try

This is a short introduction to mach try, a mach command that simplifies the process of selecting tests and pushing them to the try server.

To follow along at home you’ll either need to be using git cinnabar or have a modern mercurial and the hg “push-to-try” extension (available from |mach mercurial-setup|). Append —no-push to commands to keep them from pushing to the try server, and -n to see verbose messages associated with the results of commands.

# mach try is a command that takes try syntax and automates the steps
# to push the current tree to try with that syntax.
# For instance:

$ ./mach try -p win32 -u mochitest-bc

# ... will result in pushing "try: -b do -p win32 -u mochitest-bc -t none"
# to try. This saves dealing with mq or other ways of generating the try
# message commit. (An in-memory commit is generated with the appropriate
# message -- mq is not invoked at any stage).

# The more novel feature exposed by mach try is the ability to select
# specific test directories containing xpcshell, mochitests or reftests
# to run on the try server.

# For instance, if I've just made a change to one of the python libraries
# used by our test harnesses, and I'd like to quickly check that this
# feature works on windows. I can run:

$ ./mach try -p win64 testing/xpcshell testing/mochitest/tests

# This will result in the small number of xpcshell and mochitest tests
# that live next to their harnesses being run (in a single chunk) on
# try, so I can get my results without waiting for the entire suite,
# and I don't need to sift through logs to figure out which chunk a
# test lives in when I only care about running certain tests.

For more details run ./mach help try. As the command will inform you, this feature is under development — bugs should be filed blocking bug 1149670).

July 27, 2015 01:41 AM

July 26, 2015

Alice Scarpa


Thanks to Mozilla and Outreachy, I was able to spend one week visiting Mozilla’s Toronto office and it was awesome. My mentor, armenzg, really went out of his way to make me feel welcomed. It was a great way to feel like I’m part of the Mozilla community.

armenzg introduced me to Toronto mozillians so I could learn more about what is going on in the organization: ekyle gave me an ActiveData tour (so much data to play with!), mconley explained the e10s project to me (this time it is going to happen!), secretrobotron talked with me about community building (I want to get more involved in the Brazilian Mozilla community).

Thursday I went with the other interns and their mentors to a real-life Escape-the-room game. It was pretty fun and a great chance to meet people from a lot of different teams.

The best part of the whole week was Friday. armenzg invited me, wlach, ekyle and rwood to a work-from-his-house day and it was very pleasant. It was a great ending for an amazing week!

July 26, 2015 12:00 AM

July 23, 2015

Joel Maher

lost in data – episode 1, tackling a bunch of alerts

Today I recorded a session of me investigating talos alerts, It is ~35 minutes, sort of long, but doable in the background whilst grabbing breakfast or lunch!

I am looking forward to more sessions and answering questions that come up.

July 23, 2015 11:22 PM

David Burns

Microsoft ship a WebDriver implementation

Microsoft, the people still claim to be evil (who are actually big proponents of the the The Open Web), have... (wait for it...) SHIPPED. AN. IMPLEMENTATION. OF. WEBDRIVER!

At GTAC in California in 2011, Simon Stewart and I discussed that the Selenium project was at a crossroads (I am pretty sure beer was involved). We could ,and should, move this to the browser vendors. We had seen how in April the Chrome team had shipped their implementation and the Selenium project then deleted its implementation. I am pretty sure Daniel (Wagner-Hall) was glad to see it go.

In that time to now we have seen Mozilla get Marionette into Firefox and into release branches since Firefox 24 while slowly working on it (as well as Firefox OS support). We have seen Blackberry ship a version for the browser on their devices. We have seen mobile implementation with iOS-Driver, Selendroid and Appium.

The Spec is on track to be put forward for Recommendation by the end of the year. All the dreams that we (the Selenium Development team (my BFFs)) had are slowly coming true. This ship might be slow moving but it's mostly because some companies haven't always seen the value.


P.s. There is an open bug on the WebKit tracker for Safari support (and it is getting some internal push so I am hopeful!)

July 23, 2015 08:19 PM

Andreas Tolfsen

Optimised Rust code in Gecko

Rust Rust wave by Glen Scott (CC BY-NC 2.0)

Following bug 1177608, Rust code in Gecko is now compiled with optimisation by default.

Compilation of Rust components can be enabled by adding ac_add_options --enable-rust to your mozconfig file. For now there’s only limited usage of Rust in Gecko, but you can take a look at the bindings for the MP4 encoder if you’re interested. This is much thanks to the work of Ralph Giles.

Since optimised compiles are the default, you will now also get optimised output from rustc. You can disable this by setting ac_add_options --disable-optimize. This disables compilation for all cc, c++, and rustc compilers.

Next up is adding debug symbols, assertions, and adding some Rust autoconf macros.

July 23, 2015 06:10 PM

Alice Scarpa

Ijson is magic

I love Python, but it can be a little resource-hungry at times.

For example, allthethings.json is a 12MB json file. How much memory does a Python script take to load it? Let’s profile the memory usage of a script that just loads it.

import json

def load_json(filename):
    with open(filename, 'r') as f:
        return json.load(f)

if __name__ == "__main__":

Running memory_profiler on the script above:

$ python -m memory_profiler

Line #    Mem usage    Increment   Line Contents
     3   13.430 MiB    0.000 MiB   @profile
     4                             def load_json(filename):
     5   13.434 MiB    0.004 MiB       with open(filename, 'r') as f:
     6   86.773 MiB   73.340 MiB           return json.load(f)

73MB, more then 6 times the size of the original file.

This is not a big problem when loading a 12MB json on my desktop. But we needed to load a 200MB file on a Heroku standard-2x dyno, so we ended up having problems. What can help us in this case is that we only needed a subset of the data.

To use our allthethings.json example, imagine that the data we need is just the ‘shortname’ key for builders. Searching for a way to get just the data we needed without having to load the whole json in memory, I learned about ijson.

According to it’s documentation ijson is an:

Iterative JSON parser with a standard Python iterator interface

That means that instead of loading the whole file into memory and parsing everything at once, it uses iterators to lazily load the data. In that way, when we pass by a key that we don’t need, we can just ignore it and the generated object can be removed from memory.

import ijson

def load_json(filename):
    with open(filename, 'r') as fd:
        parser = ijson.parse(fd)
        ret = {'builders': {}}
        for prefix, event, value in parser:
            if (prefix, event) == ('builders', 'map_key'):
                buildername = value
                ret['builders'][buildername] = {}
            elif prefix.endswith('.shortname'):
                ret['builders'][buildername]['shortname'] = value

        return ret

if __name__ == "__main__":

Profiling its memory usage:

(venv)$ python -m memory_profiler

Line #    Mem usage    Increment   Line Contents
     5   15.109 MiB    0.000 MiB   @profile
     6                             def load_json(filename):
     7   15.109 MiB    0.000 MiB       with open(filename, 'r') as fd:
     8   15.109 MiB    0.000 MiB           parser = ijson.parse(fd)
     9   15.113 MiB    0.004 MiB           ret = {'builders': {}}
    10   26.012 MiB   10.898 MiB           for prefix, event, value in parser:
    11   26.012 MiB    0.000 MiB               if (prefix, event) == ('builders', 'map_key'):
    12   25.824 MiB   -0.188 MiB                   buildername = value
    13   25.824 MiB    0.000 MiB                   ret['builders'][buildername] = {}
    14   26.012 MiB    0.188 MiB               elif prefix.endswith('.shortname'):
    15   25.824 MiB   -0.188 MiB                   ret['builders'][buildername]['shortname'] = value
    17   26.012 MiB    0.188 MiB           return ret

Now loading the file takes 11MB and the script total memory usage went from 86MB to 26MB. On our real production example, total memory usage went from 4.5GB to 300MB.


Ijson gave us a magic level of memory-saving, but there was a speed trade-off. Using the yalj2 backend helped to reduce our loss in speed. To use the yajl backend all we had to do was:

import ijson.backends.yajl2 as ijson


$ time python

real    0m2.966s
user    0m2.944s
sys    0m0.020s


$ time python

real    0m1.626s
user    0m1.613s
sys    0m0.013s

The downside of using the yajl2 backend is that it requires users to have a C library (libyajl2) installed on their systems. Luckily, it is possible to install it on Heroku.

July 23, 2015 12:00 AM

July 22, 2015

Armen Zambrano G. (@armenzg)

Few mozci releases + reduced memory usage

While I was away adusca released few releases of mozci.

From the latest release I want to highlight that we're replacing the standard json library for ijson since it solves some memory leak issues we were facing for pulse_actions (bug 1186232).

This was important to fix since our Heroku instance for pulse_actions has an upper limit of 1GB of RAM.

Here are the release notes and the highlights of them:

Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

July 22, 2015 02:21 PM

July 21, 2015

Byron Jones

happy bmo push day!

the following changes have been pushed to

to improve security now serves attachments from a different domain – – instead of from a subdomain of  all existing links should continue to work, and will redirect to a url.

discuss these changes on

Filed under: bmo, mozilla

July 21, 2015 08:13 AM

July 17, 2015

mozregression updates

Release 0.38 and 0.39

The 0.38 release fixes a minor regression introduced with the 0.37 release (bug 1184915).

Just after that mozlog 3.0 was released, breaking compatibility with older mozbase packages, so we decided to ensure fully working dependencies and released mozregression 0.39! :)

July 17, 2015 12:00 AM

July 16, 2015

Jonathan Griffin

Automation and Tools Team Update, July 16 2015

The Automation and Tools Team (the A-Team, for short) is a large team that oversees a diverse set of services, tools and test harnesses used by nearly everyone at Mozilla.  We’re borrowing a page from Release Engineering and publishing a series of updates to inform people about what we’re up to, in the hopes of fostering better visibility and inter-team coordination.


Treeherder and Automatic Starring: Our focus for Treeherder in Q3 will be improving the signal-to-noise ratio for dealing with intermittent oranges. An overall design has been agreed to for the “automatic starring” project, and work has begun; final rollout is likely in Q4. This quarter, we’ll also stop spamming Bugzilla with comments for each intermittent, but we will put in place an alternate notification system for people who rely on Bugzilla orange comments to determine when an intermittent needs attention. We’ve also agreed on a redesign for the Logviewer that should result in a more useful and intuitive interface.
MozReview and Autoland:  MozReview now offers to publish review requests when you push, so it isn’t necessary to visit the MozReview’s UI. Work has started on adding support for autoland-to-inbound, which will allow developers to push changes to inbound directly from MozReview… no more battling tree closures!
Performance: Work continues on Perfherder’s “comparison mode”, a view that compares Talos performance data between two revisions. See wlach’s blog post for more details.
TaskCluster Support: We’re helping Release Engineering migrate from Buildbot to TaskCluster; this quarter we’re standing up Linux tests in TaskCluster and getting OS X cross-compilation to work so that we can move those builds to the cloud.
BMO now has tests running in continuous integration using TaskCluster and reporting to Treeherder.
Mobile Automation: mochitest-chrome for Android is now live! Work is also underway to enable debug reftests on Android emulators, and significant reliability improvements have been landed in Autophone.
Desktop Automation: Work is in progress to get Thread Sanitizer (TSan) builds running on try and to split gTest into its own test chunk. We’re also working towards applying –run-by-dir to mochitest-plain, in order to improve isolation and enable smarter chunking in CI.
Developer Workflow: We’re adding test-selection flexibility to the reftest harness as a prelude to making ‘mach try’ work with more test types.

The Details
Treeherder/Automatic Starring
  • Work has started on backend work needed to support automatic starring, including db simplification, and db unification (so each tree doesn’t have its own database).  Bug 1179263 tracks this work.  As a side effect of this work, Treeherder code should become less complex and easier to maintain.
  • Work has started on identifying what needs to happen in order to turn off Bugzilla comments for intermittents, and to create an alternative notification mechanism instead.  Bug 1179310 tracks this work.
Treeherder/Front End
  • New shortcuts for Logviewer, Delete Classification plus improved classification save
  • Design work is in progress for collapsing chunks in Treeherder in order to reduce visual noise in bug 1163064
Perfherder/Performance Testing
  • Evaluating alerts generated from PerfHerder
  • Improvements to compare chooser and viewer inside of PerfHerder
  • Work towards building a new tab switching test (bug 1166132)
  • Automatic publishing of reviews upon pushing
  • Known bug: people using cookie auth may experience bug 1181886
  • Better error message when MozReview’s Bugzilla session has expired (bug 1178811)
  • Pruned user database to improve user searching (bug 1171274)
  • Work is progressing on autoland-to-inbound (bug 1128039)
TaskCluster Support
  • Working on OSX cross-compilation, which will allow us to move OSX builds to the cloud; this will make OSX builds much faster in CI.
Mobile Automation
  • Autophone detects USB lock-ups and gracefully restarts. This is a huge improvement in system reliability.
  • Continued work on getting Android Talos tests ported to Autophone (bug 1170685)
  • Updated manifests and mozharness configs for mochitest-chrome (bug 1026290)
  • Determined total-chunks requirements for Android 4.3 Debug reftests (bug 1140471)
  • Re-wrote robocop harness to significantly improve run-time efficiency (bug 1179981)
Dev Workflow
  • Helped RelEng resolve some problems that were preventing them from landing mozharness in the tree.  This opens the door to a lot of future dev workflow improvements, including better unification of the ways we run automated tests in continuous integration and locally.  We’ve wanted this for years and it’s great to see it finally happen.
  • Did some work on top of jgraham’s patch to make mach use mozlog structured logging
Platform QA
  • We had to respond to the breakup of into several files to keep our Jenkins instance running.
  • Getting firefox-media-tests to satisfy Tier-2 Treeherder visibility requirements involves changing how Treeherder accommodates non-buildbot jobs (e.g bug 1182299)
General Automation
  • Working on running multiple tests/manifests through reftests harness as a prelude for supporting |mach try| for more test types.
  • Created patch to move mozlog.structured to the top level package (and what was previously there to mozlog.unstructured)
  • Figured out the series of steps needed to produce a usable Thread Sanitizer enabled linux build on our infra
  • Separating out gTest into a separate job in CI – bug 1179955.
  • More memory optimizations (motivation: releng query for Chris Atlee:  query slow tests)
    • run staging environment as stability test for production
    • change etl procedure so pushing changes to prod are easier (moving toward standard procedure)
  • import treeherder data markup to active data (motivation: characterizing test failures
    • ateam query: summary of test failures, stars and resolutions (bug 1161268bug 1172048)
    • subtests are too large for download of more than one day – working on code to only pull what’s required


July 16, 2015 05:11 PM

July 15, 2015

Christopher Manchester

Automatic triggering on try server

If you’ve pushed to Mozilla’s try server recently, you’ve probably noticed some of your failures being re-triggered automatically. This happens courtesy of the trigger bot tool implemented as a pulse listener that invokes self-serve in response to certain events.

Two trigerring rules are currently implemented:

Finally, if --rebuild is specified, nothing gets triggered when a test fails, and if --no-retry is found in try syntax, no triggering happens for the push.

This was enabled at the end of June, and I’ve gotten a bit of feedback (almost mostly positive). My own curiosity is about the overall volume of these jobs, so I pulled some stats from self-serve to see how much we’ve ended up triggering. For the first 2 weeks of July, there were 373,772 jobs on try, 13,099 of which were initiated by the trigger-bot (3.5%).

This is a larger proportion than I think it should be, but we’ve recently landed changes that mean we will trigger fewer hidden jobs. Hidden jobs are usually hidden due to failing persistently, so this should reduce overall triggering.

The rudimentary script I used to gather these stats is available on the trigger-bot repo

July 15, 2015 09:44 PM

July 14, 2015

William Lachance

Perfherder update

Haven’t been doing enough blogging about Perfherder (our project to make Talos and other per-checkin performance data more useful) recently. Let’s fix that. We’ve been making some good progress, helped in part by a group of new contributors that joined us through an experimental “summer of contribution” program.

Comparison mode

Inspired by Compare Talos, we’ve designed something similar which hooks into the perfherder backend. This has already gotten some interest: see this post on dev.tree-management and this one on dev.platform. We’re working towards building something that will be really useful both for (1) illustrating that the performance regressions we detect are real and (2) helping developers figure out the impact of their changes before they land them.

Screen Shot 2015-07-14 at 3.54.57 PM Screen Shot 2015-07-14 at 3.53.20 PM

Most of the initial work was done by Joel Maher with lots of review for aesthetics and correctness by me. Avi Halmachi from the Performance Team also helped out with the t-test model for detecting the confidence that we have that a difference in performance was real. Lately myself and Mike Ling (one of our summer of contribution members) have been working on further improving the interface for usability — I’m hopeful that we’ll soon have something implemented that’s broadly usable and comprehensible to the Mozilla Firefox and Platform developer community.

Graphs improvements

Although it’s received slightly less attention lately than the comparison view above, we’ve been making steady progress on the graphs view of performance series. Aside from demonstrations and presentations, the primary use case for this is being able to detect visually sustained changes in the result distribution for talos tests, which is often necessary to be able to confirm regressions. Notable recent changes include a much easier way of selecting tests to add to the graph from Mike Ling and more readable/parseable urls from Akhilesh Pillai (another summer of contribution participant).

Screen Shot 2015-07-14 at 4.09.45 PM

Performance alerts

I’ve also been steadily working on making Perfherder generate alerts when there is a significant discontinuity in the performance numbers, similar to what GraphServer does now. Currently we have an option to generate a static CSV file of these alerts, but the eventual plan is to insert these things into a peristent database. After that’s done, we can actually work on creating a UI inside Perfherder to replace alertmanager (which currently uses GraphServer data) and start using this thing to sheriff performance regressions — putting the herder into perfherder.

As part of this, I’ve converted the graphserver alert generation code into a standalone python library, which has already proven useful as a component in the Raptor project for FirefoxOS. Yay modularity and reusability.

Python API

I’ve also been working on creating and improving a python API to access Treeherder data, which includes Perfherder. This lets you do interesting things, like dynamically run various types of statistical analysis on the data stored in the production instance of Perfherder (no need to ask me for a database dump or other credentials). I’ve been using this to perform validation of the data we’re storing and debug various tricky problems. For example, I found out last week that we were storing up to duplicate 200 entries in each performance series due to double data ingestion — oops.

You can also use this API to dynamically create interesting graphs and visualizations using ipython notebook, here’s a simple example of me plotting the last 7 days of pageload data inline in a notebook:

Screen Shot 2015-07-14 at 4.43.55 PM


July 14, 2015 08:51 PM

Julien Pagès

mozregression 0.37 release

Yesterday we just released mozregression (command line regression range finder for Mozilla nightly and inbound builds) 0.37!

This release include some new features and fixes, and uses the new TaskCluster system from Mozilla Release Engineering to retrieve integration branch information. This considerably speeds up the bisection of inbound builds.

Try it out!

July 14, 2015 04:29 PM

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

July 14, 2015 07:17 AM

July 13, 2015

Julien Pagès

Python [aggressive] pep8 conversion using autopep8

Some time ago, I started to look at Talos, the python performance testing framework for firefox that is usable on Windows, Mac and Linux.

Talos has been around for a long time, and has seen many contributors! Unfortunately the codebase reflects that, and as an example there was simply no common code style around for all the Python files.

Sometimes a 2 spaces based indentation, sometimes 4 spaces based, sometimes something else. No limit for the lines length. This end up with something that is really hard to read, understand and maintain. So one of my first goal was to try to clean this up.

And I heard about autopep8. As the name suggests, this is a tool (command line) that will automatically do some pep8 conversion on Python source files.

Awesome! I highly recommend that to you if you are trying to adapt the coding style of some Python code to pep8 convention, this worked really well for me. This is a pain to do that by hand, and almost impracticable when the indentation needs to be fixed, but autopep8 makes that for you in a safe way.

Tip: the –aggressive flag is really nice for fixing long lines.

So, as an example this is how I used autopep8 on talos:

autopep8 --recursive --in-place --aggressive --aggressive /path/to/talos

Simple and effective. See the usage documentation to see with an example what it really does and other command line flags.

July 13, 2015 09:21 AM

mozregression updates

Release 0.37

I’m excited to announce that mozregression now supports using taskcluster for inbound bisection with this release!

This considerably speeds up the process of downloading builds from integration branches (like mozilla-inbound) and also supports work by Mozilla Release Engineering to migrate build storage from ftp to s3.


New features:

July 13, 2015 12:00 AM

July 12, 2015

Julien Pagès

RunSnakeRun – graphical visualisation of dumped python profiling data

If you are a Python developer like me, you probably know the profile and cProfile modules that provides deterministic profiling of Python programs.

These modules are awesome – however, when it comes to analysing the data to improve your program, the provided pstats module is generally not powerful enough if you have quite a large codebase.

And here graphical tools comes in handy! I tried RunSnakeRun, and this is a really great program that allows you to analyse the profiling data under multiple angles (a nice view is by file), so you can find easily the bottlenecks and fix them.

RunSnakeRun helped me to improve the “mach help” command. It is a cross platform tool. Note that if you are on GNU/Linux and that you have the KDE desktop (or don’t mind to install the required KDE dependencies), KCacheGrind can be used with Python also.

July 12, 2015 04:12 PM

July 07, 2015

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

July 07, 2015 06:11 AM

July 06, 2015

Armen Zambrano G. (@armenzg)

mozci 0.8.2 - Allow using TreeHerder as a query source

In this release we have added an experimental feature where you can use Treeherder as your source for jobs' information instead of using BuildApi/Buildjson.
My apologies as this should have been a minor release (0.9.0) instead of a security release (0.8.2).


Thanks to @adusca @vaibhavmagarwal and @chmanchester for their contributions.
Our latest new contributor is @priyanklodha - thank you!

How to update

Run "pip install -U mozci" to update

Major highlights

  • Added --query-source option to get data from Treeherder or Buildapi
  • Improved usage of OOP to allow for different data sources seamlessly

Minor improvements

  • Better documentation of --times
  • Cleaning up old builds-*.js files
  • Enforced line character limit

All changes

You can see all changes in here:

Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

July 06, 2015 02:58 PM

July 02, 2015

Byron Jones

happy bmo push day!

the following changes have been pushed to

discuss these changes on

Filed under: bmo, mozilla

July 02, 2015 05:59 AM

July 01, 2015

David Burns

Who wants to be an alpha tester for Marionette?

Are you an early adopter type? Are you an avid user of WebDriver and want to use the latest and great technology? Then you are most definitely in luck.

Marionette, the Mozilla implementation of the FirefoxDriver, is ready for a very limited outing. There is a lot of things that have not been implemented or, since we are implementing things agains the WebDriver Specification they might not have enough prose to implement (This has been a great way to iron out spec bugs).

Getting Started

At the moment, since things are still being developed and we are trying to do things with new technologies (like writing part this project using Rust) we are starting out with supporting Linux and OS X first. Windows support will be coming in the future!

Getting the driver

We have binaries that you can download. For Linux and for OS X . The only bindings currently updated to work are the python bindings that are available in a branch on my fork of the Selenium project. Do the following to get it into a virtualenv:
  1. Create a virtualenv
  2. activate your virtualenv
  3. cd to where you have cloned my repository
  4. In a terminal type the following: ./go py_install

Running tests

Running tests against marionette requires that you do the following changes (which hopefully remains small)

Update the desired capability to have marionette:true and add binary:/path/to/Firefox/DeveloperEdition/or/Nightly. We are only supporting those two versions of at the moment because we have had a couple in compatible issues that we have fixed which means speaking to Marionette in the browser in the beta or release versions quite difficult.

Since you are awesome early adopters it would be great if we could raise bugs. I am not expecting everything to work but below is a quick list that I know doesn't work.

Thanks for being an early adopter and thanks for raising bugs as you find them!

July 01, 2015 07:57 PM