Planet Mozilla Automation

September 16, 2014

Armen Zambrano G. (@armenzg)

Which builders get added to buildbot?

To add/remove jobs on tbpl.mozilla.org, we have to modify buildbot-configs.

Making changes can be learnt by looking at previous patches, however, there's a bit of an art to it to get it right.

I just landed a script that sets up buildbot for you inside of a virtualenv and you can pass a buildbot-config patch and determine which builders get added/removed.

You can run this by checking out braindump and running something like this:
buildbot-related/list_builder_differences.sh -j path_to_patch.diff

NOTE: This script does not check that the job has all the right parameters once live (e.g. you forgot to specify the mozharness config for it).

Happy hacking!


Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

September 16, 2014 03:26 PM

Byron Jones

happy bmo push day!

the following changes have been pushed to bugzilla.mozilla.org:

discuss these changes on mozilla.tools.bmo.


Filed under: bmo, mozilla

September 16, 2014 06:12 AM

September 15, 2014

William Lachance

mozregression 0.24

I just released mozregression 0.24. This would be a good time to note some of the user-visible fixes / additions that have gone in recently:

  1. Thanks to Sam Garrett, you can now specify a different branch other than inbound to get finer grained regression ranges from. E.g. if you’re pretty sure a regression occurred on fx-team, you can do something like:

    mozregression --inbound-branch fx-team -g 2014-09-13 -b 2014-09-14

  2. Fixed a bug where we could get an incorrect regression range (bug 1059856). Unfortunately the root cause of the bug is still open (it’s a bit tricky to match mozilla-central commits to that of other branches) but I think this most recent fix should make things work in 99.9% of cases. Let me know if I’m wrong.
  3. Thanks to Julien Pagès, we now download the inbound build metadata in parallel, which speeds up inbound bisection quite significantly

If you know a bit of python, contributing to mozregression is a great way to have a high impact on Mozilla. Many platform developers use this project in their day-to-day work, but there’s still lots of room for improvement.

September 15, 2014 10:02 PM

September 12, 2014

Geoff Brown

Running my own AutoPhone

AutoPhone is a brilliant platform for running automated tests on physical mobile devices.

:bc maintains an AutoPhone instance running startup performance tests (aka “S1/S2 tests” or “throbber start/stop tests”) on a small farm of test phones; those tests run against all of our Firefox for Android builds and results are reported to PhoneDash, available for viewing at http://phonedash.mozilla.org/.

I have used phonedash.mozilla.org for a long time now, and reported regressions in bugs and in my monthly “Performance Check-up” posts, but I have never looked under the covers or tried to use AutoPhone myself — until this week.

All things considered, it is surprisingly easy to set up your own AutoPhone instance and run your own tests. You might want to do this to reproduce phonedash.mozilla.org results on your own computer, or to check for regressions on a feature before check-in.

Here’s what I did to run my own AutoPhone instance running S1/S2 tests against mozilla-inbound builds:

Install AutoPhone:

git clone https://github.com/mozilla/autophone

cd autophone

pip install -r requirements.txt

Install PhoneDash, to store and view results:

git clone https://github.com/markrcote/phonedash

Create a phonedash settings file, phonedash/server/settings.cfg with content:

[database]
SQL_TYPE=sqlite
SQL_DB=yourdb
SQL_SERVER=localhost
SQL_USER=
SQL_PASSWD=

Start phonedash:

python server.py <ip address of your computer>

It will log status messages to the console. Watch that for any errors, and to get a better understanding of what’s happening.

Prepare your device:

Connect your Android phone or tablet to your computer by USB. Multiple devices may be connected. Each device must be rooted. Check that you can see your devices with adb devices — and note the serial number(s) (see devices.ini below).

Configure your device:

cp devices.ini.example devices.ini

Edit devices.ini, changing the serial numbers to your device serial numbers and the device names to something meaningful to you. Here’s my simple devices.ini for one device I called “gbrown”:

[gbrown]
serialno=01498B300600B008

Configure autophone:

cp autophone.ini.example autophone.ini

Edit autophone.ini to make it your own. Most of the defaults are fine; here is mine:

[settings]
#clear_cache = False
#ipaddr = …
#port = 28001
#cachefile = autophone_cache.json
#logfile = autophone.log
loglevel = DEBUG
test_path = tests/manifest.ini
#emailcfg = email.ini
enable_pulse = True
enable_unittests = False
#cache_dir = builds
#override_build_dir = None
repos = mozilla-inbound
#buildtypes = opt
#build_cache_port = 28008
verbose = True

#build_cache_size = 20
#build_cache_expires = 7
#device_ready_retry_wait = 20
#device_ready_retry_attempts = 3
#device_battery_min = 90
#device_battery_max = 95
#phone_retry_limit = 2
#phone_retry_wait = 15
#phone_max_reboots = 3
#phone_ping_interval = 15
#phone_command_queue_timeout = 10
#phone_command_queue_timeout = 1
#phone_crash_window = 30
#phone_crash_limit = 5

python autophone.py -h provides help on options, which are analogues of the autophone.ini settings.

Configure your tests:

Notice that autophone.ini has a test path of tests/manifest.ini. By default, tests/manifest.ini is configured for S1/S2 tests — it points to configs/s1s2_settings.ini. We need to set up that file:

cd configs

cp s1s2_settings.ini.example s1s2_settings.ini

Edit s1s2_settings.ini to make it your own. Here’s mine:

[paths]
#source = files/
#dest = /mnt/sdcard/tests/autophone/s1test/
#profile = /data/local/tmp/profile

[locations]
# test locations can be empty to specify a local
# path on the device or can be a url to specify
# a web server.
local =
remote = http://192.168.0.82:8080/

[tests]
blank = blank.html
twitter = twitter.html

[settings]
iterations = 2
resulturl = http://192.168.0.82:8080/api/s1s2/

[signature]
id =
key =

Be sure to set the resulturl to match your PhoneDash instance.

If running local tests, copy your test files (like blank.html above) to the files directory. If runnng remote tests, be sure that your test files are served from the resulturl (if using PhoneDash, copy to the html directory).

Start autophone:

python autophone.py –config autophone.ini

With these settings, autophone will listen for new builds on mozilla-inbound, and start tests on your device(s) for each one. You should start to see your device reboot, then Firefox will be installed and startup tests will run. As more builds complete on mozilla-inbound, more tests will run.

autophone.py will print some diagnostics to the console, but much more detail is available in autophone.log — watch that to see what’s happening.

Check your phonedash instance for results — visit http://<ip address of your computer>:8080. At first this won’t have any data, but as autophone runs tests, you’ll start to see results. Here’s my instance after a few hours:

myphonedash


September 12, 2014 08:02 PM

September 11, 2014

William Lachance

Hacking on the Treeherder front end: refreshingly easy

Over the past two weeks, I’ve been working a bit on the Treeherder front end (our interface to managing build and test jobs from mercurial changesets), trying to help get things in shape so that the sheriffs can feel comfortable transitioning to it from tbpl by the end of the quarter.

One thing that has pleasantly surprised me is just how easy it’s been to get going and be productive. The process looks like this on Linux or Mac:


git clone https://github.com/mozilla/treeherder-ui.git
cd treeherder-ui/webapp
./scripts/web-server.js

Then just load http://localhost:8000 in your favorite web browser (Firefox) and you should be good to go (it will load data from the actually treeherder site). If you want to make modifications to the HTML, Javascript, or CSS just go ahead and do so with your favorite editor and the changes will be immediately reflected.

We have a fair backlog of issues to get through, many of them related to the front end. If you’re interested in helping out, please have a look:

https://wiki.mozilla.org/Auto-tools/Projects/Treeherder#Bugs_.26_Project_Tracking

If nothing jumps out at you, please drop by irc.mozilla.org #treeherder and we can probably find something for you to work on. We’re most active during Pacific Time working hours.

September 11, 2014 08:35 PM

Armen Zambrano G. (@armenzg)

Run tbpl jobs locally with Http authentication (developer_config.py) - take 2

Back in July, we deployed the first version of Http authentication for mozharness, however, under some circumstances, the initial version could fail and affect production jobs.

This time around we have:

If you read How to run Mozharness as a developer you should see the new changes.

As quick reminder, it only takes 3 steps:

  1. Find the command from the log. Copy/paste it.
  2. Append --cfg developer_config.py
  3. Append --installer-url/--test-url with the right values
To see a real example visit this


Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

September 11, 2014 12:45 PM

September 09, 2014

Byron Jones

happy bmo push day!

the following changes have been pushed to bugzilla.mozilla.org:

discuss these changes on mozilla.tools.bmo.

 

the new bugmail filtering ability allows you to filter on specific flags:

bugmail filtering with substrings

these two rules will prevent bugzilla from emaling you the changes to the “qa whiteboard” field or the “qe-verify” flag for bugs where you aren’t the assignee.


Filed under: bmo, mozilla

September 09, 2014 06:41 AM

September 08, 2014

Henrik Skupin

Firefox Automation report – week 29/30 2014

In this post you can find an overview about the work happened in the Firefox Automation team during week 29 and 30.

Highlights

During week 29 it was time again to merge the mozmill-tests branches to support the upcoming release of Firefox 31.0. All necessary work has been handled on bug 1036881, which also included the creation of the new esr31 branch. Accordingly we also had to update our mozmill-ci system, and got the support landed on production.

The RelEng team asked us if we could help in setting up Mozmill update tests for testing the new update server aka Balrog. Henrik investigated the necessary tasks, and implemented the override-update-url feature in our tests and the mozmill-automation update script. Finally he was able to release mozmill-automation 2.6.0.2 two hours before heading out for 2 weeks of vacation. That means Mozmill CI could be used to test updates for the new update server.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 29 and week 30.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meetings of week 29 and week 30.

September 08, 2014 06:37 PM

September 05, 2014

Mark Côté

Review Board preview

I know lots of people are very anxious to see Mozilla’s new code-review tool. It’s been a harrowing journey, but we are finally knocking out our last few blockers for initial deployment (see tracking bug 1021929). While we sort those out, here’s something to whet your palate: a walk through the new review work flow.

September 05, 2014 12:02 AM

September 02, 2014

Geoff Brown

Firefox for Android Performance Measures – August check-up

My monthly review of Firefox for Android performance measurements. This month’s highlights:

 – Eideticker for Android is back!

 – small regression in ts_paint

 – small improvement in tp4m

 – small regression in time to throbber start / stop

Talos

This section tracks Perfomatic graphs from graphs.mozilla.org for mozilla-central builds of Firefox for Android, for Talos tests run on Android 4.0 Opt. The test names shown are those used on tbpl. See https://wiki.mozilla.org/Buildbot/Talos for background on Talos.

tcanvasmark

This test is not currently run on Android 4.0.

tcheck2

Measure of “checkerboarding” during simulation of real user interaction with page. Lower values are better.

12 (start of period) – 12 (end of period)

There was a temporary regression in this test for much of the month, but it seems to be resolved now.

trobopan

Panning performance test. Value is square of frame delays (ms greater than 25 ms) encountered while panning. Lower values are better.

50000 (start of period) – 50000 (end of period)

tprovider

Performance of history and bookmarks’ provider. Reports time (ms) to perform a group of database operations. Lower values are better.

520 (start of period) – 520 (end of period).

tsvgx

An svg-only number that measures SVG rendering performance. About half of the tests are animations or iterations of rendering. This ASAP test (tsvgx) iterates in unlimited frame-rate mode thus reflecting the maximum rendering throughput of each test. The reported value is the page load time, or, for animations/iterations – overall duration the sequence/animation took to complete. Lower values are better.

6300 (start of period) – 6300 (end of period).

tp4m

Generic page load test. Lower values are better.

940 (start of period) – 850 (end of period).

tp4m

Improvement noted around August 21.

ts_paint

Startup performance test. Lower values are better.

3650 (start of period) – 3850 (end of period).

tspaint

Note the slight regression around August 12, and perhaps another around August 27 – bug 1061878.

Throbber Start / Throbber Stop

These graphs are taken from http://phonedash.mozilla.org.  Browser startup performance is measured on real phones (a variety of popular devices).

throbstart

Note the regression in time to throbber start around August 14 — bug 1056176.

throbstop

The same regression, less pronounced, is seen in time to throbber stop.

Eideticker

Eideticker for Android is back after a long rest – yahoo!!

These graphs are taken from http://eideticker.mozilla.org. Eideticker is a performance harness that measures user perceived performance of web browsers by video capturing them in action and subsequently running image analysis on the raw result.

More info at: https://wiki.mozilla.org/Project_Eideticker

eide1 eide2 eide3 eide4 eide5


September 02, 2014 07:49 PM

Byron Jones

happy bmo push day!

the following changes have been pushed to bugzilla.mozilla.org:

discuss these changes on mozilla.tools.bmo.

bitly


Filed under: bmo, mozilla

September 02, 2014 06:30 AM

August 27, 2014

Byron Jones

happy bmo push day!

the following changes have been pushed to bugzilla.mozilla.org:

discuss these changes on mozilla.tools.bmo.


Filed under: bmo, mozilla

August 27, 2014 07:11 AM

August 26, 2014

Byron Jones

happy bmo push day!

the following changes have been pushed to bugzilla.mozilla.org:

discuss these changes on mozilla.tools.bmo.


Filed under: bmo, mozilla

August 26, 2014 07:49 AM

August 20, 2014

Byron Jones

happy bmo push day!

the following changes have been pushed to bugzilla.mozilla.org:

discuss these changes on mozilla.tools.bmo.


Filed under: bmo, mozilla

August 20, 2014 07:49 AM

August 15, 2014

William Lachance

A new meditation app

I had some time on my hands two weekends ago and was feeling a bit of an itch to build something, so I decided to do a project I’ve had in the back of my head for a while: a meditation timer.

If you’ve been following this log, you’d know that meditation has been a pretty major interest of mine for the past year. The foundation of my practice is a daily round of seated meditation at home, where I have been attempting to follow the breath and generally try to connect with the world for a set period every day (usually varying between 10 and 30 minutes, depending on how much of a rush I’m in).

Clock watching is rather distracting while sitting so having a tool to notify you when a certain amount of time has elapsed is quite useful. Writing a smartphone app to do this is an obvious idea, and indeed approximately a zillion of these things have been written for Android and iOS. Unfortunately, most are not very good. Really, I just want something that does this:

  1. Select a meditation length (somewhere between 10 and 40 minutes).
  2. Sound a bell after a short preparation to demarcate the beginning of meditation.
  3. While the meditation period is ongoing, do a countdown of the time remaining (not strictly required, but useful for peace of mind in case you’re wondering whether you’ve really only sat for 25 minutes).
  4. Sound a bell when the meditation ends.

Yes, meditation can get more complex than that. In Zen practice, for example, sometimes you have several periods of varying length, broken up with kinhin (walking meditation). However, that mostly happens in the context of a formal setting (e.g. a Zendo) where you leave your smartphone at the door. Trying to shoehorn all that into an app needlessly complicates what should be simple.

Even worse are the apps which “chart” your progress or have other gimmicks to connect you to a virtual “community” of meditators. I have to say I find that kind of stuff really turns me off. Meditation should be about connecting with reality in a more fundamental way, not charting gamified statistics or interacting online. We already have way too much of that going on elsewhere in our lives without adding even more to it.

So, you might ask why the alarm feature of most clock apps isn’t sufficient? Really, it is most of the time. A specialized app can make selecting the interval slightly more convenient and we can preselect an appropriate bell sound up front. It’s also nice to hear something to demarcate the start of a meditation session. But honestly I didn’t have much of a reason to write this other than the fact than I could. Outside of work, I’ve been in a bit of a creative rut lately and felt like I needed to build something, anything and put it out into the world (even if it’s tiny and only a very incremental improvement over what’s out there already). So here it is:

meditation-timer-screen

The app was written entirely in HTML5 so it should work fine on pretty much any reasonably modern device, desktop or mobile. I tested it on my Nexus 5 (Chrome, Firefox for Android)[1], FirefoxOS Flame, and on my laptop (Chrome, Firefox, Safari). It lives on a subdomain of this site or you can grab it from the Firefox Marketplace if you’re using some variant of Firefox (OS). The source, such as it is, can be found on github.

I should acknowledge taking some design inspiration from the Mind application for iOS, which has a similarly minimalistic take on things. Check that out too if you have an iPhone or iPad!

Happy meditating!

[1] Note that there isn’t a way to inhibit the screen/device from going to sleep with these browsers, which means that you might miss the ending bell. On FirefoxOS, I used the requestWakeLock API to make sure that doesn’t happen. I filed a bug to get this implemented on Firefox for Android.

August 15, 2014 02:02 AM

August 13, 2014

Henrik Skupin

Firefox Automation report – week 27/28 2014

In this post you can find an overview about the work happened in the Firefox Automation team during week 27 and 28.

Highlights

Henrik continued his work on our QA specific PuppetAgain setup. One of the blockers for us was bug 997721, which is the full proxy support on OS X and Linux. By week 27 we were finally able to get this finished. Further Henrik also got the manifest for installing Java done.

On TPS we also made progress. So Cosmin got the Pulse listener script implemented for the Coversheet CI, which triggers TPS tests whenever new nightly builds of Firefox have been made available. Further a couple of fixes for Mozrunner were necessary given that the 6.0 release caused a couple of regressions for TPS. As result we agreed on to pin Python package dependencies to specific versions of mozrunner and related packages.

One big thing for our team is also to assist people in the decision, if automated tests are possible for certain Firefox features. The questions mainly come up for tests, which cannot be implemented for any of our developer driven test frameworks due to limitations on buildbot (no network access allowed, restart of the application, and others…). To be more successful in the future, Henrik started a discussion on the dev-quality mailing list. We hope to get the proposed process established for bug verification.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 27 and week 28.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meetings of week 27 and week 28.

August 13, 2014 09:58 AM

August 12, 2014

Byron Jones

happy bmo push day!

the following changes have been pushed to bugzilla.mozilla.org:

discuss these changes on mozilla.tools.bmo.


Filed under: bmo, mozilla

August 12, 2014 07:00 AM

August 07, 2014

Armen Zambrano G. (@armenzg)

mozdevice now mozlogs thanks to mozprocess!



jgraham: armenzg_brb: This new logging in mozdevice is awesome!
armenzg: jgraham, really? why you say that?
jgraham
: armenzg: I can see what's going on!
We recently changed the way that mozdevice works. mozdevice is a python package used to talk to Firefox OS or Android devices either through ADB or SUT.

Several developers of the Auto Tools team have been working on the Firefox OS certification suite for partners to determine if they meet the expectations of the certification process for Firefox OS.
When partners have any issues with the cert suite, they can send us a zip file with their results so we can help them out. However, until recently, the output of mozdevice would go to standard out rather than being logged into the zip file they would send us.

In order to use the log manger inside the certification suite, I needed every adb command and its output to be logged rather than printed to stdout. Fortunately, mozprocess has a parameter that I can specify which allows me to manipulate any output generate by the process. To benefit from mozprocess and its logging, I needed to switch every subprocess.Popen() call to be replaced by ProcessHandler().

If you want to "see what's going on" in mozdevice, all you have to do is to request the debug level of logging like this:
DeviceManagerADB(logLevel=mozlog.DEBUG)
As part of this change, we also switched to use structured logging instead of the basic mozlog logging.

Switching to mozprocess also helped us discover an issue in mozprocess specific to Windows.

You can see the patch in here.
You can see other changes inside of the mozdevice 0.38 release in here.

At least with mozdevice you can know what is going on!


Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

August 07, 2014 03:58 PM

Byron Jones

happy bmo push day!

the following changes have been pushed to bugzilla.mozilla.org:

discuss these changes on mozilla.tools.bmo.


Filed under: bmo, mozilla

August 07, 2014 06:04 AM

August 03, 2014

Geoff Brown

Firefox for Android Performance Measures – July check-up

My monthly review of Firefox for Android performance measurements. This month’s highlights:

- No significant regressions or improvements found!

Talos

This section tracks Perfomatic graphs from graphs.mozilla.org for mozilla-central builds of Firefox for Android, for Talos tests run on Android 4.0 Opt. The test names shown are those used on tbpl. See https://wiki.mozilla.org/Buildbot/Talos for background on Talos.

tcanvasmark

This test is not currently run on Android 4.0.

tcheck2

Measure of “checkerboarding” during simulation of real user interaction with page. Lower values are better.

Screenshot from 2014-08-03 13:13:44

12 (start of period) – 12 (end of period)

The temporary regression of July 24 was caused by bug 1031107; resolved by bug 1044702.

trobopan

Panning performance test. Value is square of frame delays (ms greater than 25 ms) encountered while panning. Lower values are better.

50000 (start of period) – 50000 (end of period)

tprovider

Performance of history and bookmarks’ provider. Reports time (ms) to perform a group of database operations. Lower values are better.

520 (start of period) – 520 (end of period).

tsvgx

An svg-only number that measures SVG rendering performance. About half of the tests are animations or iterations of rendering. This ASAP test (tsvgx) iterates in unlimited frame-rate mode thus reflecting the maximum rendering throughput of each test. The reported value is the page load time, or, for animations/iterations – overall duration the sequence/animation took to complete. Lower values are better.

6300 (start of period) – 6300 (end of period).

tp4m

Generic page load test. Lower values are better.

940 (start of period) – 940 (end of period).

ts_paint

Startup performance test. Lower values are better.

3600 (start of period) – 3650 (end of period).

Throbber Start / Throbber Stop

These graphs are taken from http://phonedash.mozilla.org.  Browser startup performance is measured on real phones (a variety of popular devices).

Screenshot from 2014-08-03 13:39:26

Screenshot from 2014-08-03 13:45:17

Screenshot from 2014-08-03 13:49:07

Eideticker

These graphs are taken from http://eideticker.mozilla.org. Eideticker is a performance harness that measures user perceived performance of web browsers by video capturing them in action and subsequently running image analysis on the raw result.

More info at: https://wiki.mozilla.org/Project_Eideticker

The Eideticker dashboard is slowly coming back to life, but there are still not enough results to show graphs here. We’ll check back at the end of August.


August 03, 2014 07:51 PM

July 29, 2014

Joel Maher

Say hi to Kaustabh Datta Choudhury, a newer Mozillian

A couple months ago I ran into :kaustabh93 online as he had picked up a couple good first bugs.  Since then he has continued to work very hard and submit a lot of great pull requests to Ouija and Alert Manager (here is his github profile).  After working with him for a couple of weeks, I decided it was time to learn more about him, and I would like to share that with Mozilla as a whole:

Tell us about where you live-

I live in a town called Santragachi in West Bengal. The best thing about this place is its ambience. It is not at the heart of the city but the city is easily accessible. That keeps the maddening crowd of the city away and a calm and peaceful environment prevails here.

Tell us about your school-

 I completed my schooling from Don Bosco School, Liluah. After graduating from there, now I am pursuing an undergraduate degree in Computer Science & Engineering from MCKV Institute of Engineering.

Right from when it was introduced to me, I was in love with the subject ‘Computer Science’. And introduction to coding was one of the best things that has happened to me so far.

Tell us about getting involved with Mozilla-

I was looking for some exciting real life projects to work on during my vacation & it was then that the idea of contributing to open source projects had struck me. Now I have been using Firefox for many years now and that gave me an idea of where to start looking. Eventually I found the volunteer tab and thus started my wonderful journey on Mozilla.

Right from when I was starting out, till now, one thing that I liked very much about Mozilla was that help was always at hand when needed. On my first day , I popped a few questions in the IRC channel #introduction & after getting the basic of where to start out, I started working on Ouija under the guidance of ‘dminor’ & ‘jmaher’. After a few bug fixes there, Dan recommended me to have a look at Alert Manager & I have been working on it ever since. And the experience of working for Mozilla has been great.

Tell us what you enjoy doing-

I really love coding. But apart from it I also am an amateur photographer & enjoy playing computer games & reading books.

Where do you see yourself in 5 years?

In 5 years’ time I prefer to see myself as a successful engineer working on innovative projects & solving problems.

If somebody asked you for advice about life, what would you say?

Rather than following the crowd down the well-worn path, it is always better to explore unchartered territories with a few.

:kaustabh93 is back in school as of this week, but look for activity on bugzilla and github from him.  You will find him online once in a while in various channels, I usually find him in #ateam.


July 29, 2014 07:49 PM

Dave Hunt

Performance testing Firefox OS on reference devices

A while back I wrote about the LEGO harness I created for Eideticker to hold both the device and camera in place. Since then there has been a couple of iterations of the harness. When we started testing against our low-cost prototype device, the harness needed modifying due to the size difference and position of the USB socket. At this point I tried to create a harness that would fit all of our current devices, with the hope of avoiding another redesign.

Eideticker harness v2.0 If you’re interested in creating one of these yourself, here’s the LEGO Digital Designer file and building guide.

Unfortunately, when I first got my hands on our reference device (codenamed ‘Flame’) it didn’t fit into the harness. I had to go back to the drawing board, and needed to be a little more creative due to the width not matching up too well with the dimensions of LEGO bricks. In the end I used some slope bricks (often used for roof tiles) to hold the device securely. A timelapse video of constructing the latest harness follows.

 

 

We now are 100% focused on testing against our reference device, so in London we have two dedicated to running our Eideticker tests, as shown in the photo below.

Eideticker harness for FlameAgain, if you want to build one of these for yourself, download the LEGO Digital Designer file and building guide. If you want to learn more about the Eideticker project check out the project page, or if you want to see the dashboard with the latest results, you can find it here.

July 29, 2014 04:08 PM

July 23, 2014

Dave Hunt

A new home for the gaiatest documentation

The gaiatest python package provides a test framework and runner for testing Gaia (the user interface for Firefox OS). It also provides a handy command line tool and can be used as a dependency from other packages that need to interact with Firefox OS.

Documentation for this package has now been moved to gaiatest.readthedocs.org, which is generated directly from the source code whenever there’s an update. In order to make this more useful we will continue to add documentation to the Python source code. If you’re interested in helping us out please get in touch by leaving a comment, or joining #ateam on irc.mozilla.org and letting us know.

July 23, 2014 12:56 PM

July 22, 2014

Byron Jones

happy bmo push day!

the following changes have been pushed to bugzilla.mozilla.org:

discuss these changes on mozilla.tools.bmo.


Filed under: bmo, mozilla

July 22, 2014 02:36 PM

July 17, 2014

Henrik Skupin

Firefox Automation report – week 25/26 2014

In this post you can find an overview about the work happened in the Firefox Automation team during week 25 and 26.

Highlights

June the 11th was actually the last Automation Training day for our team in Q3. About the results you can read here. We will implement some changes for the next quarter, when we most likely want to host 2 of them.

Henrik finally got the time to upgrade our Mozmill-CI systems to the lastest LTS version of Jenkins. There were a bit of changes necessary but in general all went fine this time, and we can see some great improvements. Especially the long delays when sending out job results seem to be gone.

Further Henrik investigated the slow behavior with the mozmill-ci production master, when it is under load, e.g. QA runs ondemand update tests for releases of Firefox. The main problem stays with Java, which is taking up about 100% of the CPU. Because of this the integrated web server cannot serve pages in a timely manner. Adding a 2nd CPU to this node gave us way better response times.

Given that the new version of Ubuntu came out already in April, we want to have our Mozmill tests also run on that platform version. So we got new VM spun-up by IT, which we now have to puppetize and bring online. But this may still take a bit, given the remaining blockers for using PuppetAgain.

While talking about Puppet we got the next big change reviewed and landed. With bug 1021230 we now have our own user account, which can be customized to our needs. And that’s what we totally need, given that our infrastructure is so different from the Releng one.

Also for TPS we made progress, so the new TPS-CI production machine came online. Yet it cannot replace the current CI due to still a fair amount of open blockers, but hopefully by end of July we should be able to turn the switch.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 25 and week 26.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meetings of week 25 and week 26.

July 17, 2014 01:57 PM

July 16, 2014

Byron Jones

happy bmo push day!

switching the default monospace font on bmo yesterday highlighted a few issues with the fira-mono typeface that we’d like to see addressed before we use it.  as a result comments are now displayed using their old font.

the following changes have been pushed to bugzilla.mozilla.org:

discuss these changes on mozilla.tools.bmo.


Filed under: bmo, mozilla

July 16, 2014 06:30 AM

July 15, 2014

Armen Zambrano G. (@armenzg)

Developing with GitHub and remote branches

I have recently started contributing using Git by using GitHub for the Firefox OS certification suite.

It has been interestting switching from Mercurial to Git. I honestly believed it would be more straight forward but I have to re-read again and again until the new ways sink in with me.

jgraham shared with me some notes (Thanks!) with regards what his workflow looks like and I want to document it for my own sake and perhaps yours:
git clone git@github.com:mozilla-b2g/fxos-certsuite.git

# Time passes

# To develop something on master
# Pull in all the new commits from master

git fetch origin

# Create a new branch (this will track master from origin,
# which we don't really want, but that will be fixed later)

git checkout -b my_new_thing origin/master

# Edit some stuff

# Stage it and then commit the work

git add -p
git commit -m "New awesomeness"

# Push the work to a remote branch
git push --set-upstream origin HEAD:jgraham/my_new_thing

# Go to the GH UI and start a pull request

# Fix some review issues
git add -p
git commit -m "Fix review issues" # or use --fixup

# Push the new commits
git push

# Finally, the review is accepted
# We could rebase at this point, however,
# we tend to use the Merge button in the GH UI
# Working off a different branch is basically the same,
# but you replace "master" with the name of the branch you are working off.


Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

July 15, 2014 09:04 PM

Byron Jones

happy bmo push day!

the following changes have been pushed to bugzilla.mozilla.org:

discuss these changes on mozilla.tools.bmo.


Filed under: bmo, mozilla

July 15, 2014 06:40 AM

July 14, 2014

Mark Côté

BMO mid-2014 update

Here’s your mid-year report from the offices, basements, and caverns of BMO!

Performance

This year we’re spending a lot of time on performance. As nearly everyone knows, Bugzilla’s an old Perl app from the early days of the Web, written way before all the technologies, processes, and standards of today were even dreamt of. Furthermore, Bugzilla (including BMO) has a very flexible extension framework, which makes broad optimizations difficult, since extensions can modify data at many points during the loading and transforming of data. Finally, Bugzilla has evolved a very fine-grained security system, crucial to an open organization like Mozilla that still has to have a few secrets, at least temporarily (for security and legal reasons, largely). This means lots of security checks when loading or modifying a bug—and, tangentially, it makes the business logic behind the UI pretty complex under the hood.

That said, we’ve made some really good progress, starting with retrofitting Bugzilla to use memcached, and then instrumenting the database and templating code to give of reams of data to analyze. Glob has lots of details in his post on BMO perf work; read it if you’re interested in the challenges of optimizing a legacy web app. The tl;dr is that BMO is faster than last year; our best progress has been on the server side of show_bug (the standard Bug view), which, for authenticated users, is about 15% faster on average than last year, with far fewer spikes.

Bugs updated since last visit

Part of an effort to improve developer productivity, in June we rolled out a feature to give a new way for users to track changes to bugs. BMO now notes when you visit a bug you’re involved in (when you load it in the main Bugzilla UI or otherwise perform actions on it), and any changes to that bug which occur since you last visited it will show up in a table in My Dashboard. Read more.

Bugmail filtering

Another improvement to developer productivity centred around notifications is the new bugmail filtering feature. Bugzilla sends out quite a lot of mail, and the controls for deciding when you want to receive a notification of a change to a bug have been pretty coarse-grained. This new feature is extremely customizable, so you can get only the notifications you really care about.

BzAPI compatibility

There have been several broad posts about this recently, but it’s worth repeating. The original Bugzilla REST API, known as BzAPI, is deprecated in favor of the new native REST API (on BMO at least; it isn’t yet available in any released version of the Bugzilla product). If possible, sites currently using BzAPI should be modified to use the new API (they are largely, but not entirely, compatible), but at a minimum they should be updated to use the new BzAPI compatibility layer, which is hosted directly on BMO and sits atop the new REST API. The compatibility layer should act almost exactly the same as BzAPI (the exceptions being that a few extra fields are returned in a small number of calls). At some point in the not-too-distant future, we’ll be (transparently) redirecting all requests to BzAPI to this layer and shutting down the BzAPI server, so it’s better to try to migrate now while the original BzAPI is still around, in case there are any lingering bugs in the compatibility layer.

More stuff

As usual, you can see our current goals and high-priority items for the quarter on the BMO wiki page.

July 14, 2014 05:42 PM

David Burns

WebDriver F2F - London 2014

Last week saw the latest face to face of the WebDriver Working Group held at Facebook. This meeting was important as this is hopefully the last face to face before we go to Last call allowing us to concentrate on issues that come up during last call.

This meeting was really useful as we were a number of discussions around the prose of the spec when it comes to conformance and usability of the spec, especially when given to implementors who have never worked on WebDriver.

The Agenda from the meeting can be found here

The notable items that were discussed are:

To read what was discussed you can see the notes for Monday and Tuesday.

July 14, 2014 02:15 PM

Bugsy 0.3.0 - Comments!

I have just released the latest version of Bugsy. This allows you to get comments from Bugzilla and add new comments too. This API is still experimental so please send back some feedback since I may change it to real world usage.

I have updated the documentation to get you started.

>>> comments = bug.get_comments()
>>> comment[0].text
"I <3 Cheese"
>>> bug.add_comment("And I love bacon")

You can see the Changelog for more details.

Please raise issues on GitHub

July 14, 2014 10:07 AM

July 11, 2014

Armen Zambrano G. (@armenzg)

Introducing Http authentication for Mozharness.

A while ago, I asked a colleague (you know who you are! :P) of mine how to run a specific type of test job on tbpl on my local machine and he told me with a smirk, "With mozharness!"

I wanted to punch him (HR: nothing to see here! This is not a literal punch, a figurative one), however he was right. He had good reason to say that, and I knew why he was smiling. I had to close my mouth and take it.

Here's the explanation on why he said that: most jobs running inside of tbpl are being driven by Mozharness, however they're optimized to run within the protected network of Release Engineering. This is good. This is safe. This is sound. However, when we try to reproduce a job outside of the Releng network, it becomes problematic for various reasons.

Many times we have had to guide people who are unfamiliar with mozharness as they try to run it locally with success. (Docs: How to run Mozharness as a developer). However, on other occasions when it comes to binaries stored on private web hosts, it becomes necessary to loan a machine. A loaned machine can reach those files through internal domains since it is hosted within the Releng network.

Today, I have landed a piece of code that does two things:
This change, plus the recently-introduced developer configs for Mozharness, makes it much easier to run mozharness outside of continuous integration infrastructure.

I hope this will help developers have a better experience reproducing the environments used in the tbpl infrastructure. One less reason to loan a machine!

This makes me *very* happy (see below) since I don't have VPN access anymore.




Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

July 11, 2014 07:42 PM

Using developer configs for Mozharness

To help run mozharness by developers I have landed some configs that can be appended to the command appearing on tbpl.

All you have to do is:
  • Find the mozharness script line in a log from tbpl (search for "script/scripts")
  • Look for the --cfg parameter and add it again but it should end with "_dev.py"
    • e.g. --cfg android/androidarm.py --cfg android/androidarm_dev.py
  • Also add the --installer-url and --test-url parameters as explained in the docs
Developer configs have these things in common:
  • They have the same name as the production one but instead end in "_dev.py"
  • They overwrite the "exes" dict with an empty dict
    • This allows to use the binaries in your personal $PATH
  • They overwrite the "default_actions" list
    • The main reason is to remove the action called read-buildbot-configs
  • They fix URLs to point to the right public reachable domains 
Here are the currently available developer configs:
You can help by adding more of them!















Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

July 11, 2014 07:15 PM

Joel Maher

I invite you to join me in welcoming 3 new tests to Talos

2 months ago, we added session restore tests to Talos, today I am pleased to announce 3 new tests:

 

Stay tuned in the coming weeks as we have 2 others tests in the works:

 


July 11, 2014 02:52 PM

Henrik Skupin

Firefox Automation report – week 23/24 2014

In this post you can find an overview about the work happened in the Firefox Automation team during week 23 and 24.

Highlights

To continue the training for Mozilla related test frameworks, we had the 3rd automation training day on June 4th. This time lesser people attended, but we were still able to get a couple of tasks done on oneanddone.

Something which bothered us already for a while, is that for our mozmill-tests repository no push_printurl hook was setup. As result the landed changeset URL gets not printed to the console during its landing. Henrik fixed that on bug 1010563 now, which allows an easier copy&paste of the link to our bugs.

Our team started to work on the new continuous integration system for TPS tests. To be able to manage all the upcoming work ourselves, Henrik asked Jonathan Griffin to move the Coversheet repository from his own account to the Mozilla account. That was promptly done.

In week 24 specifically on June 11th we had our last automation training day for quarter 2 in 2014. Given the low attendance from people we might have to do some changes for future training days. One change might be to have the training on another day of the week. Andreea probably will post updates on that soon.

Henrik was also working on getting some big updates out for Mozmill-CI. One of the most important blockers for us was the upgrade of Jenkins to the latest LTS release. With that a couple of issues got fixed, including the long delays in sending out emails for failed jobs. For more details see the full list of changes.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 23 and week 24.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meetings of week 23 and week 24.

July 11, 2014 08:59 AM

July 10, 2014

Geoff Brown

New try aliases “xpcshell” and “robocop”

We now have two new try aliases which will be of interest to some mobile developers.

Android 2.3 tests run xpcshell tests in 3 chunks, which can be specified in a try push:

try: … -u xpcshell-1,xpcshell-2,xpcshell-3

but since all other test platforms run xpcshell as a single chunk, it’s easy to forget about Android 2.3’s chunks and push something like:

try: -b o -p all -u xpcshell -t none

…and then wonder why xpcshell tests didn’t run for Android 2.3!

As of today, a new try alias recognizes “xpcshell” to mean “run all the xpcshell test chunks”.

Similarly, a new try alias recognizes “robocop” to mean “run all the robocop test chunks”.

An example: https://tbpl.mozilla.org/?tree=Try&rev=e52bcf945dcd

tryaliases

How convenient!

(Of course, “-u xpcshell-1″, “-u robocop-2, robocop-3″, etc still work and you should use them if you only need to run specific chunks.)

Thanks to :Callek and :RyanVM for making this happen.


July 10, 2014 04:55 PM

Henrik Skupin

Firefox Automation report – week 21/22 2014

In this post you can find an overview about the work happened in the Firefox Automation team during week 21 and 22.

Highlights

To assist everyone from our community to learn more about test automation at Mozilla, we targeted 4 full-day automation training days from mid of May to mid of June. The first training day was planned for May 21rd and went well. Lots of [people were present and actively learning more about automation[https://quality.mozilla.org/2014/05/automation-training-day-may-21st-results/). Especially about testing with Mozmill.

To support community members to get in touch with Mozmill testing a bit easier, we also created a set of one-and-done tasks. Those start from easy tasks like running Mozmill via the Mozmill Crowd extension, and end with creating the first simple Mozmill test.

Something, which hit us by surprise was that with the release of Firefox 30.0b3 we no longer run any automatically triggered Mozmill jobs in our CI. It took a bit of investigation but finally Henrik found out that the problem has been introduced by RelEng when they renamed the product from ”’firefox”’ to ”’Firefox”’. A quick workaround fixed it temporarily, but for a long term stable solution we might need a frozen API for build notifications via Mozilla Pulse.

One of our goals in Q2 2014 is also to get our machines under the control of PuppetAgain. So Henrik started to investigate the first steps, and setup the base manifests as needed for our nodes and the appropriate administrative accounts.

The second automation training day was also planned by Andreea and took place on May 28th. Again, a couple of people were present, and given the feedback on one-and-done tasks, we fine-tuned them.

Last but not least Henrik setup the new Firefox-Automation-Contributor team, which finally allows us now to assign people to specific issues. That was necessary because Github doesn’t let you do that for anyone, but only known people.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 21 and week 22.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meetings of week 21 and week 22.

July 10, 2014 12:54 PM

July 09, 2014

Byron Jones

using “bugmail filtering” to exclude notifications you don’t want

a common problem with bugzilla emails (aka bugmail) is there’s too much of it.  if you are involved in a bug or watching a component you receive all changes made to bugs, even those you have no interest in receiving.

earlier this week we pushed a new feature to bugzilla.mozilla.org : bugmail filtering.

bugmail_filtering

this feature is available on the “bugmail filtering” tab on the “user preference” page.


 

there are many reasons why bugzilla may send you notification of a change to a bug:

dealing with all that bugmail can be time consuming.  one way address this issue is to use the x-headers present in every bugmail to categorise and separate bugmail into different folders in your inbox.  unfortunately this option isn’t available to everyone (eg. gmail users still cannot filter on any email header).

bugmail filtering allows you to tell bugzilla to notify you only if it’s a change that you’re interested in.

for example, you can say:

don’t send me an email when the qa-whiteboard field is changed unless the bug is assigned to me

  filter-qa

if multiple filters are applicable to the same bug change, include filters override exclude filters.  this interplay allows you to write filters to express “don’t send me an email unless …”

don’t send me an email for developer tools bugs that i’m CC’ed on unless the bug’s status is changed

filter-devtools-exclude

filter-devtools-include


Filed under: bmo

July 09, 2014 04:39 PM

July 08, 2014

Byron Jones

happy bmo push day!

the following changes have been pushed to bugzilla.mozilla.org:

server-side bugmail filtering

accessible via the “Bugmail Filtering” user preference tab, this feature provides fine-grained control over what changes to bugs will result in an email notification.

for example to never receive changes made to the “QA Whiteboard” field for bugs where you are not the assignee add the following filter:

Field: QA Whiteboard
Product: __Any__
Component: __Any__
Relationship: Not Assignee
Action:Exclude

discuss these changes on mozilla.tools.bmo.


Filed under: bmo, mozilla

July 08, 2014 09:14 AM

July 07, 2014

William Lachance

Measuring frames per second and animation smoothness with Eideticker

[ For more information on the Eideticker software I'm referring to, see this entry ]

Just wanted to write up a few notes on using Eideticker to measure animation smoothness, since this is a topic that comes up pretty often and I wind up explaining these things repeatedly. ;)

When rendering web content, we want the screen to update something like 60 times per second (typical refresh rate of an LCD screen) when an animation or other change is occurring. When this isn’t happening, there is often a user perception of jank (a.k.a. things not working as they should). Generally we express how well we measure up to this ideal by counting the number of “frames per second” that we’re producing. If you’re reading this, you’re probably already familiar with the concept in outline. If you want to know more, you can check out the wikipedia article which goes into more detail.

At an internal level, this concept matches up conceptually with what Gecko is doing. The graphics pipeline produces frames inside graphics memory, which is then sent to the LCD display (whether it be connected to a laptop or a mobile phone) to be viewed. By instrumenting the code, we can see how often this is happening, and whether it is occurring at the right frequency to reach 60 fps. My understanding is that we have at least some code which does exactly this, though I’m not 100% up to date on how accurate it is.

But even assuming the best internal system monitoring, Eideticker might still be useful because:

Unfortunately, deriving this sort of information from a video capture is more complicated than you’d expect.

What does frames per second even mean?

Given a set of N frames captured from the device, the immediate solution when it comes to “frames per second” is to just compare frames against each other (e.g. by comparing the value of individual pixels) and then counting the ones that are different as “unique frames”. Divide the total number of unique frames by the length of the
capture and… voila? Frames per second? Not quite.

First off, there’s the inherent problem that sometimes the expected behaviour of a test is for the screen to be unchanging for a period of time. For example, at the very beginning of a capture (when we are waiting for the input event to be acknowledged) and at the end (when we are waiting for things to settle). Second, it’s also easy to imagine the display remaining static for a period of time in the middle of a capture (say in between gestures in a multi-part capture). In these cases, there will likely be no observable change on the screen and thus the number of frames counted will be artificially low, skewing the frames per second number down.

Measurement problems

Ok, so you might not consider that class of problem that big a deal. Maybe we could just not consider the frames at the beginning or end of the capture. And for pauses in the middle… as long as we get an absolute number at the end, we’re fine right? That’s at least enough to let us know that we’re getting better or worse, assuming that whatever we’re testing is behaving the same way between runs and we’re just trying to measure how many frames hit the screen.

I might agree with you there, but there’s a further problems that are specific to measuring on-screen performance using a high-speed camera as we are currently with FirefoxOS.

An LCD updates gradually, and not all at once. Remnants of previous frames will remain on screen long past their interval. Take for example these five frames (sampled at 120fps) from a capture of a pan down in the FirefoxOS Contacts application (movie):

sidebyside

Note how if you look closely these 5 frames are actually the intersection of *three* seperate frames. One with “Adam Card” at the top, another with “Barbara Bloomquist” at the top, then another with “Barbara Bloomquist” even further up. Between each frame, artifacts of the previous one are clearly visible.

Plausible sounding solutions:

Personally the last solution appeals to me the most, although it has the obvious disadvantage of being a “homebrew” metric that no one has ever heard of before, which might make it difficult to use to prove that performance is adequate — the numbers come with a long-winded explanation instead of being something that people immediately understand.

July 07, 2014 04:13 PM

July 04, 2014

Byron Jones

bugzilla and performance, 2014

bugzilla is a large legacy perl application initially developed at the birth of the world wide web. it has been refactored significantly in the 15 years since its release; however, some design decisions made at the start of the project carry significant technical debt in today’s modern web.

while there have been a large number of micro-optimisations over the course of bugzilla’s life (generally as a result of performance profiling), there are limits to the benefits these sorts of optimisations can provide.

this year has seen a lot of focus on improving bugzilla’s performance within mozilla, centred around the time it takes to display a bug to authenticated users.

response-times-may-2013response-times-may-2014

tl;dr bugzilla is faster

memcached

the design of a modern large web application is generally centred around caches, between the business logic layer and the data sources, as well as between the users and the user interface. while bugzilla has been refactored to use objects, database abstraction, templates, etc, it had zero caching capabilities. this coupled with completely stateless processing of each request meant that every byte of html returned to the user was regenerated from scratch, starting with a new connection to the database.

towards the end of 2013 i worked on implementing a memcached framework into bugzilla [bug 237498].

retrofitting a caching mechanism into a large extendible framework proved to be a significant challenge. bugzilla provides the ability for developers to extend bugzilla’s functionality via extensions, including but not limited to adding new fields, user interface, or process. the extensions system conflicts with caching as it’s possible for an extension to conditionally alter an object in ways that would render it impossible to cache (eg. add a new field to an object only if the current user is a member of a particular group).

some compromises had to be made. instead of caching fully constructed objects, the cache sits between the object’s constructor and the database. we avoid a trip to the database, but still have to construct objects from that data (which allows extensions to modify the object during construction).

code which updated the database directly instead of using bugzilla’s objects had to be found and rewritten to use the objects or updated to manually clear the cache entries. extra care had to be taken as returning stale data could silently result in data loss. to placate concerns that these errors would be impossible to detect, the caller of an object’s constructor must pass in a parameter to “opt-in” to caching.

in 2014 i built upon the memcached framework to support most of bugzilla’s objects [bug 956233], with the “bugs” object being the only notable exception. memcached also caches bugzilla’s configuration parameters (classifications, products, components, groups, flags, …) [bug 987032]. although caching the “bugs” object would be beneficial given its central role in all things bugzilla, it is highly likely that enabling this by default would break bugzilla extensions and customisations as a high proportion of them update the database directly instead of using bugzilla’s object layer. this would manifest as data which is silently stale, making undetectable dataloss a high probability.

memcached support will be released with bugzilla 5.0, but has been live on bugzilla.mozilla.org (bmo) since february 2014.

instrumentation

while profilling tools such as nytprof have often been pointed at bugzilla, bmo’s high number of concurrent requests and usage patterns time and time again resulted in performance optimisations performing worse than expected once deployed to production.

we took the decision to deploy instrumentation code into bugzilla itself, reporting on each http request, database query, and template execution. as bugzilla is written in perl, support was absent for off-the-shelf instrumentation tools such as new relic, so we had to roll our own data collection and reporting system [bug 956230].

the collector wraps specific Bugzilla::DB, Bugzilla::Memcached and Bugzilla::Template calls via subclassing, then reports data to an elasticsearch cluster. currently all reporting solutions are ad-hoc and involve running of scripts which identify the most costly database queries and templates.

this data identified areas of bugzilla which require optimisation or caching. examples include the optimisation of a single query which shaved 200ms (~15%) off page load times for most users [bug 993894], caching of all queries which read an entire table [part of bug 987032], and caching of user groups [bugs 993939] and settings [bug 993926].

we continue to revisit the instrumentation reports in order to guide further improvements.

stylesheet concatenation and minification

improving the performance of a web applications isn’t limited to focusing on server-side execution speed.

due to bugzilla’s extensions we ended up in a situation where bugzilla was serving multiple small css files – on bmo we loaded 17 stylesheets as part of show_bug in comparison with 5 for an installation of bugzilla without any extensions installed.

similar to the issue encountered with memcached, extensions have complete control with regards to optionally loading stylesheets, which means any css concatenation and minification solution needed to be implemented at run-time.

[bug 977969] does exactly that – the template passes an array of stylesheets to load to bugzilla’s global header, where a hash of the array is used to find a unified stylesheet. simple minification is performed which dropped the stylesheet size from 54kb to 43kb on show_bug on bmo.

stylesheet concatenation and minification support will be released with bugzilla 5.0, and has been live on bugzilla.mozilla.org since may 2014.

database

in order to address performance issues caused by bugzilla’s use of the myisam table type, in march our DBAs upgraded our database cluster to mysql version 5.6. this was the result of analysis by the DBAs into replication and stability issues around our myisam table.

as mysql 5.6 adds support for fulltext indexing to its innodb table type, bugzilla was able to switch away from myisam. this immediately fixed the database replication issues, and provided a noticeable performance boost to searches involving comments.

looking forward

the next large project is to update bugzilla so the bug object can use memcached on an unmodified installation without any non-default extensions enabled. for reasons previously covered it’s unlikely we’ll ship a version of bugzilla with this enabled by default, however this will allow sites to audit their own code (if any) and enable caching of the bug object if required.

we will build on the css concatenation and minification work to provide the same benefits to javascript files.

we periodically enable instrumentation and use it to identify the next set of targets for optimisation. this will continue for the foreseeable future.


Filed under: bmo, bugzilla

July 04, 2014 07:50 AM

July 03, 2014

Joel Maher

Alert Manager has a more documentation and a roadmap

I have been using alert manager for a few months to help me track performance regressions. It is time to take it to the next level and increase productivity with it.

Yesterday I created a wiki page outlining the project. Today I filed a bug of bugs to outline my roadmap.

Basically we have:
* a bunch of little UI polish bugs
* some optimizations
* addition of reporting
* more work flow for easier management and investigations

In the near future we will work on making this work for high resolution alerts (i.e. each page that we load in talos collects numbers and we need to figure out how to track regressions on those instead of the highly summarized version of a collection).

Thanks for looking at this, improving tools always allows for higher quality work.


July 03, 2014 07:41 PM

More thoughts on Auto-land and try server

Last week I wrote a post with some thoughts on AutoLand and Try Server, this had some wonderful comments and because of that I have continued to think in the same problem space a bit more.

In chatting with Vaibhav1994 (who is really doing an awesome GSoC project this summer for Mozilla), we started brainstorming another way to resolve our intermittent orange problem.

What if we rerun the test case that caused the job to go orange (yes in a crash, leak, shutdown timeout we would rerun the entire job) and if it was green then we could deem the failure as intermittent and ignore it!

With some of the work being done in bug 1014125, we could achieve this outside of buildbot and the job could repeat itself inside the single job instance yielding a true green.

One thought- we might want to ensure that if it is a test failing that we run it 5 times and it only fails 1 time, otherwise it is too intermittent.

A second thought- we would do this by try by default for autoland, but still show the intermittents on integration branches.

I will eventually get to my thoughts on massive chunking, but for now, lets hear more pros and cons of this idea/topic.


July 03, 2014 06:40 PM

Armen Zambrano G. (@armenzg)

Tbpl's blobber uploads are now discoverable

What is blobber? Blobber is a server and client side set of tools that allow Releng's test infrastructure to upload files without requiring to deploy ssh keys on them.

This is useful since it allows uploads of screenshots, crashdumps and any other file needed to debug what failed on a test job.

Up until now, if you wanted your scripts determine the files uploaded in a job, you would have to download the log and parse it to find the TinderboxPrint lines for Blobbler uploads, e.g.
15:21:18 INFO - (blobuploader) - INFO - TinderboxPrint: Uploaded 70485077-b08a-4530-8d4b-c85b0d6f9bc7.dmp to http://mozilla-releng-blobs.s3.amazonaws.com/blobs/mozilla-inbound/sha512/5778e0be8288fe8c91ab69dd9c2b4fbcc00d0ccad4d3a8bd78d3abe681af13c664bd7c57705822a5585655e96ebd999b0649d7b5049fee1bd75a410ae6ee55af
Now, you can look for the set of files uploaded by looking at the uploaded_files.json that we upload at the end of all uploads. This can be discovered by inspecting the buildjson files or by listening to the pulse events. The key used is called "blobber_manifest_url" e.g.
"blobber_manifest_url": "http://mozilla-releng-blobs.s3.amazonaws.com/blobs/try/sha512/39e400b6b94ac838b4e271ed61a893426371990f1d0cc45a7a5312d495cfdb485a1866d7b8012266621b4ee4df0cf9aa7d0f6d0e947ff63785543d80962aaf9b",
In the future, this feature will be useful when we start uploading structured logs. It will help us not to download logs to extract meta-data about the jobs!

No, your uploads are not this ugly
This work was completed in bug 986112. Thanks to aki, catlee, mtabara and rail to help me get this out the door. You can read more about Blobber by visiting: "Blobber is live - upload ALL the things!" and "Blobber - local environment setup".


Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

July 03, 2014 12:02 PM

Geoff Brown

Firefox for Android Performance Measures – June check-up

My monthly review of Firefox for Android performance measurements. June highlights:

- Talos values tracked here switch to Android 4.0, rather than Android 2.2

- Talos regressions in tcheck2 and tsvgx

- small regression in time to throbber stop

- Eideticker still not reporting results.

Talos

This section tracks Perfomatic graphs from graphs.mozilla.org for mozilla-central builds of Native Fennec. In all of my previous posts, this section has tracked Talos for Android 2.2 Opt. This month, and going forward, I switch to Android 4.0 Opt, since the Android 2.2 Opt tests are being phased out. The test names shown are those used on tbpl. See https://wiki.mozilla.org/Buildbot/Talos for background on Talos.

tcanvasmark

This test is not currently run on Android 4.0.

tcheck2

Measure of “checkerboarding” during simulation of real user interaction with page. Lower values are better.

tcheck2

6 (start of period) – 12 (end of period)

Regression of June 17 – bug 1026742.

trobopan

Panning performance test. Value is square of frame delays (ms greater than 25 ms) encountered while panning. Lower values are better.

50000 (start of period) – 50000 (end of period)

There was a large temporary regression between June 12 and June 14 – bug 1026798.

tprovider

Performance of history and bookmarks’ provider. Reports time (ms) to perform a group of database operations. Lower values are better.

520 (start of period) – 520 (end of period).

tsvgx

An svg-only number that measures SVG rendering performance. About half of the tests are animations or iterations of rendering. This ASAP test (tsvgx) iterates in unlimited frame-rate mode thus reflecting the maximum rendering throughput of each test. The reported value is the page load time, or, for animations/iterations – overall duration the sequence/animation took to complete. Lower values are better.

6100 (start of period) – 6300 (end of period).

Regression of June 16 – bug 1026551.

tp4m

Generic page load test. Lower values are better.

940 (start of period) – 940 (end of period).

ts_paint

Startup performance test. Lower values are better.

3600 (start of period) – 3600 (end of period).

Throbber Start / Throbber Stop

These graphs are taken from http://phonedash.mozilla.org.  Browser startup performance is measured on real phones (a variety of popular devices).

throbstart

 

throbstop

“Time to throbber start” looks very flat for all devices, but “Time to throbber stop” has a slight upward trend, especially for nexus-s-2 — bug 1032249.

Eideticker

These graphs are taken from http://eideticker.mozilla.org. Eideticker is a performance harness that measures user perceived performance of web browsers by video capturing them in action and subsequently running image analysis on the raw result.

More info at: https://wiki.mozilla.org/Project_Eideticker

Eideticker results are still not available. We’ll check back at the end of July.


July 03, 2014 02:39 AM

July 01, 2014

Armen Zambrano G. (@armenzg)

Down Memory Lane

It was cool to find an article from "The Senecan" which talks about how through Seneca, Lukas and I got involved and hired by Mozilla. Here's the article.



Here's an excerpt:
From Mozilla volunteers to software developers 
It pays to volunteer for Mozilla, at least it did for a pair of Seneca Software Development students. 
Armen Zambrano and Lukas Sebastian Blakk are still months away from graduating, but that hasn't stopped the creators behind the popular web browser Firefox from hiring them. 
When they are not in class learning, the Senecans will be doing a wide range of software work on the company’s browser including quality testing and writing code. “Being able to work on real code, with real developers has been invaluable,” says Lukas. “I came here to start a new career as soon as school is done, and thanks to the College’s partnership with Mozilla I've actually started it while still in school. I feel like I have a head start on the path I've chosen.”  
Firefox is a free open source web browser that can...



Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

July 01, 2014 05:58 PM

Byron Jones

happy bmo push day!

the following changes have been pushed to bugzilla.mozilla.org:

discuss these changes on mozilla.tools.bmo.


Filed under: bmo, mozilla

July 01, 2014 06:45 AM

June 27, 2014

William Lachance

End of Q2 Eideticker update: Flame tests, future plans

[ For more information on the Eideticker software I'm referring to, see this entry ]

Just wanted to give an update on where Eideticker is at the end of Q2 2014. The big news is that we’ve started to run startup tests against the Flame, the results of which are starting to appear on the dashboard:

eideticker-contacts-flame [link]

It is expected that these tests will provide a useful complement to the existing startup tests we’re running with b2gperf, in particular answering the “is this regression real?” question.

Pending work for Q3:

The above isn’t an exhaustive list: there’s much more that we have in mind for the future that’s not yet scheduled or defined well (e.g. get Eideticker reporting to Treeherder’s new performance module). If you have any questions or feedback on anything outlined above I’d love to hear it!

June 27, 2014 09:23 PM

June 26, 2014

David Burns

Bugsy 0.2 - Now with 100% more search!

I have updated Bugsy to now have the ability to search Bugzilla in a meaningful way. I have updated the documentation to get you started.

For example to search Bugzilla you would do

>>> bugs = bugzilla.search_for\
...                .keywords("checkin-needed")\
...                .include_fields("flags")\
...                .search()

You can see the Changelog for more details..

Please raise issues on GitHub

June 26, 2014 10:36 PM

June 25, 2014

Joel Maher

Thoughts on Auto-Land, Try server, and intermittent oranges outline

This is the first post is a series where I will post some ideas.  These are ideas, not active projects (although these ideas could be implemented with many active projects).

My first idea is surrounding the concept of AutoLand.  Mozilla has talked about this for a long time.  In fact a conversation I had last week got me thinking more of the value of AutoLand vs blocking on various aspects of it.  There are a handful of issues blocking us from a system where we push to try and if it goes well we magically land it on the tip of a tree.  My vested interest comes in the part of “if it goes well”.

The argument here has been that we have so many intermittent oranges and until we fix those we cannot determine if a job is green.  A joke for many years has been that it would be easier to win the lottery than to get an all green run on tbpl.  I have seen a lot of cases where people push to Try and land on Inbound to only be backed out by a test failure- a test failure that was seen on Try (for the record I personally have done this once).  I am sure someone could write a book on human behavior, tooling, and analysis of why failures land on integration branches when we have try server.

My current thought is this-

* push to try server with a final patch, run a full set of tests and builds

* when all the jobs are done [1], we analyze the results of the jobs and look for 2 patterns

* pattern 1: for a given build, at most 1 job fails

* pattern 2: for a given job [2], at most 1 platform fails

* if pattern 1 + 2 pass, we put this patch in the queue for landing by the robots

[1] – we can determine the minimal amount of jobs or verify with more analysis (i.e. 1 mochitest can fail, 1 reftest can fail, 1 other can fail)

[2] – some jobs are run in different chunks.  on opt ‘dt’ runs all browser-chrome/devtools jobs, but this is ‘dt1′, ‘dt2′, ‘dt3′ on debug builds

 

This simple approach would give us the confidence that we need to reliably land patches on integration branches and achieve the same if not better results than humans.

For the bonus we could optimize our machine usage by not building/running all jobs on the integration commit because we have a complete set done on try server.

 


June 25, 2014 04:52 PM

June 24, 2014

Byron Jones

happy bmo push day!

the following changes have been pushed to bugzilla.mozilla.org:

discuss these changes on mozilla.tools.bmo.


Filed under: bmo, mozilla

June 24, 2014 08:25 AM

June 20, 2014

Armen Zambrano G. (@armenzg)

My first A-team project: install all the tests!


As a welcoming bug to the A-team I had to deal with changing what tests get packaged.
The goal was to include all tests on a tests.zip regardless if they are marked as disabled on the test manifests or not.

Changing it the packaging was not too difficult as I already had pointers from jgriffin, the problem came with the runners.
The B2G emulator and desktop mochitest runners did not read the manifests; what they did is to run all tests that came inside of the tests.zip (even disabled ones).

Unfortunately for me, the mochitest runners code is very very old and it was hard to figure out how to make it work as clean as possible. I did a lot of mistakes and landed it twice incorrectly (improper try landing and lost my good patch somewhere) - sorry Ryan!.

After a lot of tweaking it, reviews from jmaher and help from ted & ahal, it landed last week.

For more details you can read bug 989583.

PS = Using trigger_arbitrary_builds.py was priceless to speed up my development.


Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

June 20, 2014 08:06 PM

June 18, 2014

Joel Maher

Firefox 32 leaves the train station- what does the performance look like

Now that we have an uplift completed and enough future data has been collected to ensure sustained changes in data automatically, it is time for the triple fortnightly report of what performance looks like.  For reference there is some data in a blog post about general talos numbers.

Firefox 32 uplift, m-c -> Aurora (tracking bug 1004427):

Firefox 31 uplift, m-c -> Aurora (tracking bug 990085):

Firefox 30 uplift, m-c -> Aurora:

As you can see Firefox32 has a lot of improvements and fewer regressions (of those 20 about half are related to rebasing numbers).

Lets look at bugs:

 

After reviewing the process of investigating alerts, it makes sense that we continue forward with the same process in 6 week intervals and any changes are made on uplift day and they would apply only to trunk.  Some future changes we are considering:

Onward to Firefox 33!


June 18, 2014 10:13 PM

Byron Jones

happy bmo push day!

the following changes have been pushed to bugzilla.mozilla.org:

discuss these changes on mozilla.tools.bmo.


Filed under: bmo, mozilla

June 18, 2014 06:21 AM

June 17, 2014

Byron Jones

happy bmo push day!

the following changes have been pushed to bugzilla.mozilla.org:

after a herculean effort by dkl, this week’s update brings native bzapi support to bugzilla’s production infrastructure.  bzapi operates as a proxy server external to bugzilla, developed and operated by gerv. it is a reflection of gerv’s commitment and abilities that bzapi is by far the primary api which mozilians use to interface with bzapi. moving bzapi into bugzilla itself should improve performance and stability.

the plan with regards to migration is to continue to run the current bzapi proxy server until we’re happy with this code. at that point we’ll redirect bzapi calls made against the api-dev endpoint to the new endpoint.

please help us test this by changing your endpoints from https://api-dev.bugzilla.mozilla.org/latest/ to https://bugzilla.mozilla.org/bzapi/

eg.
old: https://api-dev.bugzilla.mozilla.org/latest/bug/880669
new: https://bugzilla.mozilla.org/bzapi/bug/880669

if you find any issues don’t hesitate to file bugs in the bzapi-compatibility component.

 

discuss these changes on mozilla.tools.bmo.


Filed under: bmo, mozilla

June 17, 2014 08:50 AM

June 16, 2014

David Burns

Introducing Bugsy - Client Library for interacting with Bugzilla

I have created a library for interacting with Bugzilla using the native REST API. Bugsy allows you to get bugs from Bugzilla, change what you need to and then post it back to Bugzilla. I have created documentation to get you started.

For example to get a bug you would do

import bugsy
bugzilla = bugsy.Bugsy()
bug = bugzilla.get(123456)

and then to put it back, or if there is no bug ID (like if you were creating it) then you would do

import bugsy
bug = bugsy.Bug()
bug.summary = "I really realy love cheese"
bug.add_comment("and I really want sausages with it!")
bugzilla = bugsy.Bugsy("username", "password")
bugzilla.put(bug)
bug.id #returns the bug id from Bugzilla

Searching Bugzilla is not currently supported but will definitely be there for the next version.

Please raise issues on GitHub

June 16, 2014 09:46 AM

June 11, 2014

Armen Zambrano G. (@armenzg)

Who doesn't like cheating on the Try server?

Have you ever forgotten about adding a platform to your Try push and had to push again?
Have you ever wished to *just* make changes to a tests.zip file without having to build it first?
Well, this is your lucky day!

In this wiki page, I describe how to trigger arbitrary jobs on you try push.
As always be gentle with how you use it as we all share the resources.

Go crazy!













Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

June 11, 2014 04:23 PM

William Lachance

Managing test manifests: ManifestDestiny -> manifestparser

Just wanted to make a quick announcement that ManifestDestiny, the python package we use internally here at Mozilla for declaratively managing lists of tests in Mochitest and other places, has been renamed to manifestparser. We kept the versioning the same (0.6), so the only thing you should need to change in your python package dependencies is a quick substitution of “ManifestDestiny” with “manifestparser”. We will keep ManifestDestiny around indefinitely on pypi, but only to make sure old stuff doesn’t break. New versions of the software will only be released under the name “manifestparser”.

Quick history lesson: “Manifest destiny” refers to a philosophy of exceptionalism and expansionism that was widely held by American settlers in the 19th century. The concept is considered offensive by some, as it was used to justify displacing and dispossessing Native Americans. Wikipedia’s article on the subject has a good summary if you want to learn more.

Here at Mozilla Tools & Automation, we’re most interested in creating software that everyone can feel good about depending on, so we agreed to rename it. When I raised this with my peers, there were no objections. I know these things are often the source of much drama in the free software world, but there’s really none to see here.

Happy manifest parsing!

June 11, 2014 02:44 PM

Henrik Skupin

Last Automation Training day for this quarter

Today we hold our last Automation Training day for this quarter. So if you want to learn something about test automation at Mozilla, feel free to join at any time during the day. We are around all day, waiting to answer your questions, and to get you started if you want to contribute to one of our projects.

Please have a look at our etherpad for the current status:
https://etherpad.mozilla.org/automation-training

I’m looking forward to see some of you!

June 11, 2014 08:11 AM

June 10, 2014

Henrik Skupin

Firefox Automation report – week 17/18 2014

In this post you can find an overview about the work happened in the Firefox Automation team during week 17 and 18.

Highlights

In one of the last automation reports I mentioned that we wanted to finish up the already in-progress Mozmill tests for Firefox Metro. With lots of work mostly done by Daniel we were finally able to close all the bugs as fixed. Now everyone is back on top of our new quarterly goals.

To allow new contributors of the mozmill-tests repository to pick appropriate reviewers for their patches more easily, Henrik requested to setup default reviewers on Bugzilla. The changes are now in place and the list points to Andreea, Andrei, and Henrik.

To follow our goals in educating more people about automation, we also have to reach out more on Github to actually find interested people. Therefore Henrik requested a Github mirror of our mozmill-tests repository. It’s available at https://github.com/mozilla/qa-mozmill-tests now.

With the last release of Mozmill 2.0.6 we got the mozcrash packages included. So far we aren’t able to process stacktraces, given that the necessary crashreporter symbols were not available for daily builds of Firefox on the FTP server. This has been fixed now for all kinds of daily builds.

As seen for the last beta and final releases of Firefox we still have a good amount of broken configuration files for ondemand update tests in Mozmill CI. Those are failures done by the person who runs those tests, but also by broken behavior in the mciconf tool. Given that, we want to put more focus on the full automation stack for update tests now. To get started we need appropriate notifications send out by Mozilla Pulse.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 17 and week 18.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meetings of week 17 and week 18.

June 10, 2014 10:09 AM

Byron Jones

happy bmo push day!

the following changes have been pushed to bugzilla.mozilla.org:

discuss these changes on mozilla.tools.bmo.


Filed under: bmo, mozilla

June 10, 2014 08:02 AM

June 05, 2014

David Burns

My Ideal build, test and land world

The other week I tweeted I was noticing for that day we had 1 revert push to Mozilla Inbound in every 10 pushes. For those that don't know, Mozilla Inbound is the most active integration repository that Firefox code lands in. A push can contain a number of commits depending on the bug or if a sheriff is handling checkin-needed bugs.

This tweet got replies like, and I am paraphrasing, "That's not too bad", "I expected it to be worse". Personally I think this is awful rate. Why? On that day, only 80% of pushes were code changes to the tree. The bad push and the revert leads to no changes to the tree but still uses our build and test infrastructure. This mean that at best we can(on that day) only be 80% efficient. So how can we fix this?

Note: A lot of this work is already in hand but I want to document where I wish them to go. A lot of the issues are really paper cuts but it can be death by 1000 paper cuts.

Building

Mach, the current CLI dispatch tool at Mozilla, passes the build detail to the build scripts. It is a great tool if you haven't been using it yet. The work the build peers have done with this is pretty amazing. However I do wish the build targets passed to Mach and then executed were aligned with the way that Chromium, Facebook, Twitter and Google build targets worked.

For example, working on Marionette, if I want run marionette tests I would do |./mach //testing/marionette:test| instead of the current |./mach marionette-test| . By passing in the directory we are declaring what we want explicitly to be built and test.. The moz.build file should have a dependency saying that we need a firefox binary (or apk or b2g). The test task in the moz.build in testing/marionette folder would pick runtests.py and then pass in the necessary arguments ideally based on items in the MozConfig. Knowing the relevant arguments based on the build is hard work involving looking at your history or at a wiki.

Working on something where it has unit tests and mochitests or xpcshell tests? It's simple to just change the task. E.g. ./mach //testing/marionette:test changes quickly to //testing/marionette:xpcshell. Again, not worrying about arguments when we can create sane defaults based on what we just built. I have used testing in my examples because it is simple to show different build targets on the same call.

The other reason declaring the path (and mentioning the dependencies in the same manner) is that if you call |./mach //testing/marionette:test| after updating your repo it will do an incremental build (or a clobber if needed) without you knowing you needed it. Manually clearing things or running builds just to run tests is just busy work again.

Reviews and Precommit builds/tests

Want a review? You currently either have to use bzexport or manually create a diff and upload it to the bug and set the reviewer. The Bugzilla team are working to stand up review board that would allow us to upload patches and has a gives us a better review tool.

The missing pieces for me are: 1)We have to manually pick a reviewer and 2) that we don't have a pre commit build and test step.

1) I have been using Opera's Critic for reviewing Web Platform Tests. Having the ability to assign people to review changes for a directory means that reviewing is everyones responsibility. Currently Bugzilla allows you to pick a reviewer based on the component that the bug is on. Sometimes a patch may span other areas and you then need to figure out who a reviewer should be. I think that we can do better here.

As for 2) don't necessarily need to do everything but the equivalent of a T-Style run would suffice I'm my opinion. We could even work to pair this down more to be literally a handful of tests that regularly catch bugs or make it run tests based on where the patch was landing.

Why does this matter, we have try that people can use and "my code works" and "it was reviewed, it will compile". Mozilla Inbound was closed for a total of 2 days (48+ hours) in April and 1 day (24hrs) in May. At the moment I only have the data on why the tree was closed, not the individual bugs that caused the failure, but a pre commit step would definitely limit the damage. The pre commit step might also catch some of the test failures (if we had test suites we could agree on for being the smoke test suite) which had Mozilla Inbound closed for over 2 1/2 (61+ Hours) days in April and over 3 days (72+ hours) in May.

Landing code

Once the review has passed we still have to manually push the code or set a keyword in the bug (checkin-needed) so that the sheriffs can land it. This manual step is just busy work really that isn't really needed. If something has a r+ then ideally we should be queueing this up to be landed. This is minor compared to manual step required to update the bug with the SHA when it has landed, when it is landed we should be updating the bug accordingly. It's not really that hard to do.

Unneeded manual steps have an impact on engineering productivity which have a financial cost that could be avoided.

I think the main reason why a lot of these issues have never been surfaced is there is not enough data to show the issues. I have created a dashboard, only has the items I care about currently but could easily be expanded if people wanted to see other bits of information. The way we can solve the issues above is being able to show the issues.

June 05, 2014 10:16 AM

June 04, 2014

Byron Jones

bugzilla can now show bugs that have been updated since you last visited them

thanks to dylan‘s work on bug 489028, bugzilla now tracks when you view a bug, allowing you to search for bugs which have been updated since you last visited them.

on bugzilla.mozilla.org this has been added to “my dashboard“:

my dashboard - last visited

bugzilla will only track a bug if you are involved in it (you have to be the reporter, assignee, qa-contact, or on the cc list), and if you have javascript enabled.

the “last visit” field can be used as part of a custom search, and is most useful when used in conjunction with the “%last_changed%” pronoun (which expands to the bug’s last-changed timestamp).

last visited - custom search


Filed under: bmo, mozilla

June 04, 2014 06:32 AM

Henrik Skupin

Automation Training day is today!

The Firefox QA Automation team is hosting another Automation Training day on June, 4th. The event is a day long, so join us whenever you are free. You will be able to learn details about our projects, how we tackle automation requests, or simply to get started with coding. We are always around to assist you in getting started.

We are looking out for you and your participation.

June 04, 2014 05:41 AM

June 03, 2014

Henrik Skupin

Firefox Automation report – week 15/16 2014

In this post you can find an overview about the work happened in the Firefox Automation team during week 15 and 16.

Highlights

To be able to support the new update path for Firefox Beta users via a final release candidate on the beta channel, we had to update our update testruns with Mozmill in the mozmill-automation repository. Further we also had to add support for this update path in Mozmill CI. Cosmin spend his time on those patches, and finally we got those out and active. That means from now on our software update tests can handle Firefox X.0b9 -> Firefox X.0RC -> Firefox %X+1%.0b1 updates. More information can be found on bug 960781.

We also pushed some other updates for Mozmill-CI to our production system, which includes timestamps for the log output of the pulse listener, and the addition of the application name for the product to test. The latter was necessary for MetroFirefox. But even after we stopped support for it, we wanted to see it landed given that we could add other products more easily in the future.

Also Henrik pushed a change for Mozmill-CI, which removed support for running localized versions of Firefox Nightly builds. This was necessary so that our systems can be better utilized for those Aurora, and Beta builds – where actually most of the localization work is happening.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 15 and week 16.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meetings of week 15 and week 16.

June 03, 2014 05:08 PM

Byron Jones

happy bmo push day!

the following changes have been pushed to bugzilla.mozilla.org:

discuss these changes on mozilla.tools.bmo.

 


Filed under: bmo, mozilla

June 03, 2014 08:57 AM

June 01, 2014

Byron Jones

firebot has a new owner

to echo wolf’s post – “The End of Firebot – A New Beginning”, i’m the new maintainer of the irc bot firebot.

firebot provided an excellent service, and i’m happy to ensure firebot’s continued existence.

i’d like to thank wolf for running firebot for a decade, and for allowing me to take over.


Filed under: mozilla

June 01, 2014 04:27 AM

May 30, 2014

Geoff Brown

Firefox for Android Performance Measures – May check-up

My monthly review of Firefox for Android performance measurements. May highlights:

- slight regressions in tcanvasmark and trobopan

- small regression in time to throbber stop

- Eideticker still not reporting results.

Talos

This section tracks Perfomatic graphs from graphs.mozilla.org for mozilla-central builds of Native Fennec (Android 2.2 opt). The test names shown are those used on tbpl. See https://wiki.mozilla.org/Buildbot/Talos for background on Talos.

tcanvasmark

This test runs the third-party CanvasMark benchmark suite, which measures the browser’s ability to render a variety of canvas animations at a smooth framerate as the scenes grow more complex. Results are a score “based on the length of time the browser was able to maintain the test scene at greater than 30 FPS, multiplied by a weighting for the complexity of each test type”. Higher values are better.

Image

6300 (start of period) – 5700 (end of period).

Regression of May 12 – bug 1009646.

tcheck2

Measure of “checkerboarding” during simulation of real user interaction with page. Lower values are better.

9 (start of period) – 9 (end of period)

trobopan

Panning performance test. Value is square of frame delays (ms greater than 25 ms) encountered while panning. Lower values are better.

Image

110000 (start of period) – 130000 (end of period)

This regression just happened today and has not triggered a Talos alert yet — I don’t have a bug number yet.

tprovider

Performance of history and bookmarks’ provider. Reports time (ms) to perform a group of database operations. Lower values are better.

425 (start of period) – 425 (end of period).

tsvgx

An svg-only number that measures SVG rendering performance. About half of the tests are animations or iterations of rendering. This ASAP test (tsvgx) iterates in unlimited frame-rate mode thus reflecting the maximum rendering throughput of each test. The reported value is the page load time, or, for animations/iterations – overall duration the sequence/animation took to complete. Lower values are better.

7300 (start of period) – 7300 (end of period).

tp4m

Generic page load test. Lower values are better.

750 (start of period) – 750 (end of period).

ts_paint

Startup performance test. Lower values are better.

3600 (start of period) – 3600 (end of period).

Throbber Start / Throbber Stop

These graphs are taken from http://phonedash.mozilla.org.  Browser startup performance is measured on real phones (a variety of popular devices).

Image

Image

The improvement on May 2 was due to a change in the test setup (sut vs adb).

The small regression of May 11 is tracked in bug 1018463.

Eideticker

These graphs are taken from http://eideticker.mozilla.org. Eideticker is a performance harness that measures user perceived performance of web browsers by video capturing them in action and subsequently running image analysis on the raw result.

More info at: https://wiki.mozilla.org/Project_Eideticker

Eideticker results are still not available. We’ll check back at the end of June.


May 30, 2014 10:31 PM

May 28, 2014

Armen Zambrano G. (@armenzg)

How to create local buildbot slaves


For the longest time I have wished for *some* documentation on how to setup a buildbot slave outside of the Release Engineering setup and not needing to go through the Puppet manifests.

On a previous post, I've documented how to setup a production buildbot master.
In this post, I'm only covering the slaves side of the setup.

Install buildslave

virtualenv ~/venvs/buildbot-slave
source ~/venvs/buildbot-slave/bin/activate
pip install zope.interface==3.6.1
pip install buildbot-slave==0.8.4-pre-moz2 --find-links http://pypi.pub.build.mozilla.org/pub
pip install Twisted==10.2.0
pip install simplejson==2.1.3
NOTE: You can figure out what to install by looking in here: http://hg.mozilla.org/build/puppet/file/ad32888ce123/modules/buildslave/manifests/install/version.pp#l19

Create the slaves

NOTE: I already have build and test master in my localhost with ports 9000 and 9001 respecively.
buildslave create-slave /builds/build_slave localhost:9000 bld-linux64-ix-060 pass
buildslave create-slave /builds/test_slave localhost:9001 tst-linux64-ec2-001 pass

Start the slaves

On a normal day, you can do this to start your slaves up:
 source ~/venvs/buildbot-slave/bin/activate
 buildslave start /builds/build_slave
 buildslave start /builds/test_slave


Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

May 28, 2014 07:05 PM

May 27, 2014

Byron Jones

happy bmo push day!

the following changes have been pushed to bugzilla.mozilla.org:

discuss these changes on mozilla.tools.bmo.


Filed under: bmo, mozilla

May 27, 2014 06:26 AM

May 23, 2014

Andrew Halberstadt

When would you use a Python mixin?

That's not a rhetorical question. I'd like to know in which scenarios a mixin in python really is the best option. I can't seem to think of any, but maybe I'm not thinking outside the box enough. The basic idea of a mixin is to create a small re-usable class that can "plug-in" to other larger classes. From the [wikipedia definition][1], a mixin is a way to compose classes together *without* using inheritance. The problem is unlike ruby, python mixins are a purely conceptual construct. Python mixins *are* inheritance (the only difference is that the class name usually contains 'Mixin'). It is up to the developer to remember this, and to manually avoid all of the common pitfalls of multiple inheritance. This kind of defeats the whole purpose of the mixin in the first place. What's more is that most people use python mixins improperly. Consider the following example: from book import Book from index import IndexMixin class TextBook(Book, IndexMixin): def __init__(self, name, contents, edition='1st'): Book.__init__(self, name, contents) self.edition = edition t = TextBook('Learn Python the Hard Way') t.search('python') >>> 0 results found Here we correctly use inheritance to denote `Book` as a base class, a textbook **is a** book. We use an `IndexMixin` because textbooks aren't indices, they **have an** index. We then instantiate it and search the book for the string 'python'. Wait a minute, how can a textbook about python not contain the word python? In python, the multiple inheritance chain is read **right to left**, this means that IndexMixin is technically the base class here. It just so happens that `IndexMixin` also has a function called `search` which overrides the one defined in `Book`. It turns out there's another way to compose classes together without using inheritance. Just *use* the class (or module) directly: from book import Book from index import Index class TextBook(Book): def __init__(self, name, contents, edition='1st'): Book.__init__(self, name, contents) self.edition = edition self.index = Index() t = TextBook('Learn Python the Hard Way') t.search('python') >>> Many results found t.index.search('python') >>> 0 results found If the relationship between objects A and B is "A is a B", then B is a base class, not a mixin. If the relationship is "A has a B", then why not use composition? What benefit does a mixin give you? [1]: http://en.wikipedia.org/wiki/Mixin

May 23, 2014 07:44 PM

Armen Zambrano G. (@armenzg)

Technical debt and getting rid of the elephants

Recently, I had to deal with code where I knew there were elephants in the code and I did not want to see them. Namely, adding a new build platform (mulet) and running a b2g desktop job through mozharness on my local machine.

As I passed by, I decided to spend some time to go and get some peanuts to get at least few of those elephants out of there:

I know I can't use "the elephant in the room" metaphor like that but I just did and you just know what I meant :)

Well, how do you deal with technical debt?
Do you take a chunk every time you pass by that code?
Do you wait for the storm to pass by (you've shipped your awesome release) before throwing the elephants off the ship?
Or else?

Let me know; I'm eager to hear about your own de-elephantization stories.





Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

May 23, 2014 03:35 AM

May 20, 2014

Byron Jones

happy bmo push day!

the following changes have been pushed to bugzilla.mozilla.org:

 

note: 1010751 changes the “from” header in email bugzilla.mozilla.org sends from:

From: bugzilla-daemon@mozilla.org

to:

From: "Bugzilla@Mozilla" <bugzilla-daemon@mozilla.org>

it’s possible this change may impact mail filters which current perform an exact match on the from field.

 

discuss these changes on mozilla.tools.bmo.

 


Filed under: bmo, mozilla

May 20, 2014 06:32 AM

May 19, 2014

Joel Maher

Looking for long term trends and patterns in how I work

Early last year (2013), I noticed I would work really productive for a couple weeks, and then get in a rut for a week here and there.  After discussing this perceived trend with Clint, I started tracking it every week (end of work day on Friday).  I have been tracking it for a year, and now I have data to examine in more detail:

Image

For the first part of last year (through September 2013), I would go in 6 week cycles which appeared to be about 1 week after the uplifts.  Oddly enough I wasn’t doing any specific work for uplifts, but I do recall a lot of odd issues that required debugging for each uplift.  Quite possible the day or two spent handling these issues resulted in me getting backlogged on emails.

Oddly enough when I transitioned from full time mobile automation -> full time performance automation, my cycles became more regular.  One exception was a focused project development week early in 2014 which had me doing other tasks and getting behind on a few other projects.

There is no direct correlation in the health data, but I have some theories.  I record my general feeling of health (for the most part physical, not emotional).  This is pure judgemental and there is no science behind it.  10 is good, 0 is bad, so when there is a dip in health on the graph, I usually see an increase in email volume the next week.  No explanation for that, just an observation.

In summary, I have enjoyed looking back on this data.  It was good to see a trend for most of 2013.  Maybe next year I will see a different trend or pattern.


May 19, 2014 04:20 PM

Henrik Skupin

Automation Training Day on May 21st

For anyone who is interested in Test Automation we offer again a full day of trainings. You will be able to learn more about Javascript and Python, which are the languages used a lot for projects in our Automation team.

All the details about this all day event you can read here.

Please spread the news around so that we will have a fruitful day with a lot of discussions, and small or larger tasks accomplished.

May 19, 2014 08:48 AM