the following changes have been pushed to bugzilla.mozilla.org:
the following changes have been pushed to bugzilla.mozilla.org:
the following changes have been pushed to bugzilla.mozilla.org:
[ For more information on the Eideticker software I'm referring to, see this entry ]
I just put up a proof of concept Eideticker dashboard for FirefoxOS here. Right now it has two days worth of data, manually sampled from an Unagi device running b2g18. Right now there are two tests: one the measures the “speed” of the contacts application scrolling, another that measures the amount of time it takes for the contacts application to be fully loaded.
For those not already familiar with it, Eideticker is a benchmarking suite which captures live video data coming from a device and analyzes it to determine performance. This lets us get data which is more representative of actual user experience (as opposed to an oft artificial benchmark). For example, Eideticker measures contacts startup as taking anywhere between 3.5 seconds and 4.5 seconds, versus than the 0.5 to 1 seconds that the existing datazilla benchmarks show. What accounts for the difference? If you step through an eideticker-captured video, you can see that even though something appears very quickly, not all the contacts are displayed until the 3.5 second mark. There is a gap between an app being reported as “loaded” and it being fully available for use, which we had not been measuring until now.
At this point, I am most interested in hearing from FirefoxOS developers on new tests that would be interesting and useful to track performance of the system on an ongoing basis. I’d obviously prefer to focus on things which have been difficult to measure accurately through other means. My setup is rather fiddly right now, but hopefully soon we can get some useful numbers going on an ongoing basis, as we do already for Firefox for Android.
the following changes have been pushed to bugzilla.mozilla.org:
the following changes have been pushed to bugzilla.mozilla.org:
Another update on getting Eideticker working with FirefoxOS. Once again this is sort of high-level, looking forward to writing something more in-depth soon now that we have the basics working. 
I finally got the last kinks out of the rig I was using to capture live video from FirefoxOS phones using the Point Grey devices last week. In order to make things reasonable I had to write some custom code to isolate the actual device screen from the rest of capture and a few other things. The setup looks interesting (reminds me a bit of something out of the War of the Worlds):
Here’s some example video of a test I wrote up to measure the performance of contacts scrolling performance (measured at a very respectable 44 frames per second, in case you wondering):
Surprisingly enough, I didn’t wind up having to write up any code to compensate for a noisy image. Of course there’s a certain amount of variance in every frame depending on how much light is hitting the camera sensor at any particular moment, but apparently not enough to interfere with getting useful results in the tests I’ve been running.
Likely next step: Create some kind of chassis for mounting both the camera and device on a permanent basis (instead of an adhoc one on my desk) so we can start running these sorts of tests on a daily basis, much like we currently do with Android on the Eideticker Dashboard.
As an aside, I’ve been really impressed with both the Marionette framework and the gaiatests python module that was written up for FirefoxOS. Writing the above test took just 5 minutes — and the code is quite straightforward. Quite the pleasant change from my various efforts in Android automation.
the following changes have been pushed to bugzilla.mozilla.org:
the following changes have been pushed to bugzilla.mozilla.org:
i’d like to draw attention to dkl’s work on bug 148564 – you can now tell bugzilla to never email you about specific bugs, even if you’re the reporter:
face is an irc bot i run on the #ateam channel.
its responses are purely random, drawing from past conversations on channel.
[16:46] <mdas> face: markov
[16:46] <face> tries again salivates takes code to the holiday with a great
[16:46] <mdas> hahaha
[16:46] <mcote> a great what??
[16:46] <mcote> face don’t leave me hanging
[16:46] * face never wrote it down apparently
[16:47] <mdas> HAHA
[16:47] <mcote> HA
the following changes have been pushed to bugzilla.mozilla.org:
Hey! Check it out, I have crash stacks in my logs now! I know, I know, I took my sweet time. I'm sorry. I hope this doesn't damage our relationship. Is there anything I can do to make it up to you? I want to help. I really do. It's just. I don't know. I guess I've been feeling under the weather lately. I must have caught a bug.
Anyway, I just wanted to let you know that I'm still here for you. I'm trying to make things right again. I just hope that one day, we'll be able to trust each other. I look forward to working more closely with you in the future, and beyond.
Sincerely,
B2G Emulator Tests
[1] https://tbpl.mozilla.org/php/getParsedLog.php?id=21215744&full=1&branch=mozilla-inbound
the following changes have been pushed to bugzilla.mozilla.org:
the following changes have been pushed to bugzilla.mozilla.org:
the following changes have been pushed to bugzilla.mozilla.org:
Last summer I wrote a bit about using Eideticker to measure the relative performance of Firefox for Android versus other browsers (Chrome, stock, etc.). At the time I was pretty optimistic about Eideticker’s usefulness as a truly “objective” measure of user experience that would give us a more accurate view of how we compared against the competition than traditional benchmarking suites (which more often than not, measure things that a user will never see normally when browsing the web). Since then, there’s been some things that I’ve discovered, as well as some developments in terms of the “state of the art” in mobile browsing that have caused me to reconsider that view — while I haven’t given up entirely on this concept (and I’m still very much convinced of eideticker’s utility as an internal benchmarking tool), there’s definitely some limitations in terms of what we can do that I’m not sure how to overcome.
Essentially, there are currently three different types of Eideticker performance tests:
In this blog post, I’m going to focus on startup and scrolling tests. Animation tests are interesting, but they are also generally the sorts of tests that are easiest to measure in synthetic ways (e.g. by putting a frame counter in your javascript code) and have thus far not been a huge focus for Eideticker development.
As it turns out, it’s unfortunately been rather difficult to create truly objective tests which measure the difference between browsers in these two categories. I’ll go over them in order.
Startup tests
There are essentially two types of startup tests: one where you measure the amount of time to get to the browser’s home screen when you explicitly launch the app (e.g. by pressing the Firefox icon in the app chooser), another where you load a web page in a browser from another app (e.g. by clicking on a link in the Twitter application).
The first is actually fairly easy to test across browsers, although we are not currently. There’s not really a good reason for that, it was just an oversight, so I filed bug 852744 to add something like this.
The second case (startup to the browser’s homescreen) is a bit more difficult. The problem here is that, in a nutshell, an apples to apples comparison is very difficult if not impossible simply because different browsers do different things when the user presses the application icon. Here’s what we see with Firefox:
And here’s what we see with Chrome:
And here’s what we see with the stock browser:
As you can see Chrome and the stock browser are totally different: they try to “restore” the browser back to its state from the last time (in Chrome’s case, I was last visiting taskjs.org, in Stock’s case, I was just on the homepage).
Personally I prefer Firefox’s behaviour (generally I want to browse somewhere new when I press the icon on my phone), but that’s really beside the point. It’s possible to hack around what chrome is doing by restoring the profile between sessions to some sort of clean “new tab” state, but at that point you’re not really reproducing a realistic user scenario. Sure, we can draw a comparison, but how valid is it really? It seems to me that the comparison is mostly only useful in a very broad “how quickly does the user see something useful” sense.
Panning tests
I had quite a bit of hope for these initially. They seemed like a place where Eideticker could do something that conventional benchmarking suites couldn’t, as things like panning a web page are not presently possible to do in JavaScript. The main measure I tried to compare against was something called “checkerboarding”, which essentially represents the amount of time that the user waits for the page to redraw when panning around.
At the time that I wrote these tests, most browsers displayed regions that were not yet drawn while panning using the page background. We figured that it would thus be possible to detect regions of the page which were not yet drawn by looking for the background color while initiating a panning action. I thus hacked up existing web pages to have a magenta background, then wrote some image analysis code to detect regions that were that color (under the assumption that magenta is only rarely seen in webpages). It worked pretty well.
The world’s moved on a bit since I wrote that: modern browsers like Chrome and Firefox use something like progressive drawing to display a lower resolution “tile” where possible in this case, so the user at least sees something resembling the actual page while panning on a slower device. To see what I mean, try visiting a slow-to-render site like taskjs.org and try panning down quickly. You should see something like this (click to expand):
Unfortunately, while this is certainly a better user experience, it is not so easy to detect and measure. For Firefox, we’ve disabled this behaviour so that we see the old checkerboard pattern. This is useful for our internal measurements (we can see both if our drawing code as well as our heuristics about when to draw are getting better or worse over time) but it only works for us.
If anyone has any suggestions on what to do here, let me know as I’m a bit stuck. There are other metrics we could still compare against (i.e. how smooth is the panning animation aka frames per second?) but these aren’t nearly as interesting.
the following changes have been pushed to bugzilla.mozilla.org:
highlights from the bugzilla 4.2 upgrade:
changes to bugzilla search
bugzilla’s searching mechanics got a complete rewrite between 4.0 and 4.2. see this blog post for more details.
dashboards
david lawrence [:dkl] has worked hard on bringing user and product dashboards to bmo, and has done an amazing job.
create attachments from the clipboard
you can now create attachments by pasting content into a text field.
restricted commenting
some bug reports are inundated with comments that make it difficult for developers to conduct technical discussions.
restricting comments provides the ability for users in the editusers group to prevent users who are not in the editbugs from making additional comments.
guidelines
impact
the following changes have been pushed to bugzilla.mozilla.org:
Just wanted to give a quick heads up that as part of the ateam’s ongoing effort to improve the documentation of our automated testing infrastructure, we now have online documentation for mozdevice, the python library we use for interacting with Android- and FirefoxOS-based devices in automated testing.
Mozdevice is used in pretty much every one of our testing frameworks that has mobile support, including mochitest, reftest, talos, autophone, and eideticker. Additionally, mozdevice is used by release engineering to clean up, monitor, and otherwise manage our hundred-odd the 1200* tegra and panda development boards that we use in tbpl. See sut_tools (old, buildbot-based, what we currently use) and mozpool (the new and shiny future).
* Thanks to Dustin Mitchell for the correction.
Quick update on my last post about finding some kind of camera suitable for use in automated performance testing of fennec and b2g with eideticker. Shortly after I wrote that, I found out about a company called Point Grey Research which manufactures custom web cameras intended for exactly the sorts of things we’re doing. Initial results with the Flea3 camera I ordered from them are quite promising:
(the actual capture is even higher quality than that would suggest… some detail is lost in the conversion to webm)
There seems to be some sort of issue with the white balance in that capture which is causing a flashing effect (I suspect this just involves flipping some kind of software setting… still trying to grok their SDK), and we’ll need to create some sort of enclosure for the setup so ambient light doesn’t interfere with the capture… but overall I’m pretty optimistic about this baby. 60 frames per second, very high resolution (1280×960), no issues with HDMI (since it’s just a USB camera), relatively inexpensive.
Throughout most of our B2G test automation deployment, we've been very conscious about not enabling too many tests simply because we didn't have enough capacity on our test slaves to run them all. Regardless it was still bad enough as it was (many of you probably experienced very long wait times for results). Thanks to releng (and especially Rail Aliiev) we are now running most of our B2G tests in Amazon AWS which means we can be much more flexible in accomodating load.
One thing this means is we don't need to be *as* cautious about enabling larger sets of tests for B2G. So if you have a mochitest, xpcshell test, marionette/webapi, reftest or crashtest that you would like to see running on B2G emulators, now is the time to make sure it's enabled!
Unfortunately not all harnesses have the same manifest format, so here is a quick guide for finding them:
Alternatively you can open a full log on tbpl of a test run for the branch/suite you are interested in and do a ctrl-f for your test. Just note that if there are multiple chunks you'll need to do this for each chunk before you can be certain.
Please file a bug in the test harness component (e.g testing/mochitest, testing/reftest, etc) and cc me (:ahal). Then write a patch to enable the test (or suite of tests) and push it to try with the syntax "try: -b o -p ics_armv7a_gecko -u <suite name> -t none". If you are trying to enable the test on a branch without try (e.g b2g18) then I can help test it and get it landed there.
If the test was specifically disabled, there's a good chance that it had failures or was intermittent. In this case I can help get you set up to run it locally so you can investigate if you want.
Working on the Firefox OS automation, it’s often been necessary to populate a device with some sample content. For example, when measuring the launch time of the contacts app it’s more realistic if we already have a bunch of contacts on our phone. To solve this, I created a small Python package called b2gpopulate, which uses Web APIs and mozdevice to push various types of content to a device with Marionette enabled.
To install b2gpopulate you will need Python and can simply run pip install b2gpopulate from the command line. If you don’t have pip installed then you can also use easy_install b2gpopulate. Running b2gpopulate is pretty straightforward, however you will need to have a Firefox OS device connected that’s running Marionette, and you will need to forward port 2828 by running adb forward tcp:2828 tcp:2828. The following example will populate the connected device with 200 of each content type:
b2gpopulate --contacts=200 --messages=200 --pictures=200 --videos=200 --music=200
Note that before pushing a database the b2g process is stopped, so don’t panic if you see your device restarting. Run b2gpopulate --help for full usage instructions.
Initially I used just the Contacts API to add/remove contacts from the device, but this is a pretty slow process, especially for a large number of contacts. After finding out about the reference workload that Gaia uses in its build I modified this to push a prebuilt database of contacts. This is then topped up using the Contacts API as needed. There are prebuilt databases for 200, 500, 1000, and 2000 contacts.
The most recent addition to b2gpopulate is messages. Like contacts, this pushes a prebuilt database of 200, 500, 1000, or 2000. Unlike the contacts, there is currently no option to top this up.
This uses mozdevice to push a reference picture or video to the device and then performs a remote copy. In a future version I would like to alternate through a number of reference files so there’s some variance.
This has changed in the version of b2gpopulate I released today. Previously it worked in exactly the same way as the pictures and videos, but because the metadata files doesn’t vary, the music app doesn’t distinguish between them. Now, the metadata is modified for each file using mutagen, and the album/artist is changed every ten tracks.
I suspect there will be a need for more content types in the future. For example, we could potentially add events, alarms, history, favourites, bookmarks, emails, etc. If your interested in contributing, you can find the repository on GitHub.
the following changes have been pushed to bugzilla.mozilla.org:
If you’re not already familiar with the Firefox endurance tests, these are Mozmill tests that repeat a small snippet of user interaction over and over again while gathering metrics. This allows us to detect if there’s a memory leak in an very localised area, or if there’s a memory regression within the areas tested. I’ve blogged about them a few times.
We’ve known for a while that the results we’ve been getting aren’t entirely realistic, and this is due to the fact that we only wait for 0.1 seconds between each iteration. This doesn’t give Firefox any time to perform tasks such as garbage collection. Unfortunately we couldn’t just increase this delay as that would cause other Mozmill tests to be queued behind the much longer running endurance tests.
So now that we have our new VMWare ESX cluster in place (which has given us an awesome three VMs per platform) we’ve configured Jenkins to run endurance tests on just one node per platform. This allows other Mozmill tests to continue on the remaining available nodes. We were then finally able to increase the delay to 5 seconds.
The results are as we had hoped. The memory usage has dropped, and the duration has increased. Also, the individual testrun results became a lot less erratic. This can be seen in the following charts:
It should now be much easier for us to spot regressions, and hopefully we’ll have less false positives! If you’re interested in the latest endurance results, you can find them in our Mozmill Dashboard, along with the endurance charts.
Related bugs/issues:
Last September the panda board were deemed ready to run tests. The next steps were to start integrating them into buildbot and making them 100% automated. This task turned into a much larger project and the end results was developing a smoketest which yielded a cleaner integration point with the automation.
The core of the android automation to date has been on the NVidia Tegra 250 developer kit. This has been running quite successfully with 3-4% total failure rate (product, test harness, tests, infrastructure, hardware). Our goal for testing on Android 4.0 was to test on the panda boards which also have a NEON chipset. Essentially this is just like adding more tegra boards to our automation, and for the most part that was true.
The main problems we faced came about when dealing with installing, rebooting, and overall management of the device. For our tegras, this is all in a set of python code call sut_tools. These sut_tools handle all the device management and with a few modifications we were able to do that for the panda boards.
While the tests and harnesses ran fine on a panda board at my desk, getting them to work smoothly with the sut_tools and the buildbot scripts proved to be quite a challenge. After about 10 weeks of solid work and many bugs fixed in the android kernel, system libraries, Firefox and of course our harnesses we were able to get this going fairly reliably with <10% total failure rate when we first turned these tests on in late December.
In order to prove this was working, we developed a smoketest which would run on the production foopies (host to control the panda boards, 12 at a time) and production panda boards. In fact this ended up being a way to diagnose boards, script changes and help debug overall test failures. The original smoketest was going to be ‘run some tests on a given panda board for 24 hours’. The resulting smoketest is a reuse of the exact tools we use in automation for cleanup, verification, installation, and uninstalling the product from the device under test. We also run a set of production mochitests, so we mimic a real job being pushed with about 98% accuracy.
To run these, it is pretty easy:
While this sounds straightforward, there is a bit more required in order to test a new panda board or what we normally do a chassis of new panda boards. As it stands now, I run an instance of smoketest.py in a different terminal window for every panda I am interested in testing. Usually this is 6-8 at a time, but this can easily be done for 1 or 12 without concern.
I usually run this in a loop of 100:
Then I grep the logs looking for failure messages or more specifically count how many success messages I have. If I have >95% success rate across all my logs, this is a good sign that things are ready to roll.
In the future, it would be nice to make smoketest.py have a better reporting and looping system. There is also the need to get us to 99% success rate running a controlled smoketest. One thing that would make this easier would be a tool to launch on a given set of machines and report back information and query the log files for easier parsing and status.
the following changes have been pushed to bugzilla.mozilla.org:
The other day I got asked if I knew of a tool that would notice changes to IDs of elements and update Selenium tests accordingly because there was an incurred maintenance cost in updating these all the time because the test will fail.
The tl;dr; is there isn't a tool and I don't think there should ever be one
Why?
HTML documents are not complex things, far from it, so when we change them we should think about how this is going to impact everything that hangs off a page.
If a designer or front end engineer changes the ID of elements then they go around updating the JavaScript references to that element and then update any CSS that works with those elements!
The non-obvious thing that they tend to forget is the test automation like, Selenium, that uses that element too!
Why? Well 9/10 times its because they don't have any ownership in the process! They don't submit tests nor do they run tests so they don't see the problem.
Now you are probably going to say "We use XPath/ CSS/ Names because (insert excuse here)!"
The point is still the same, your testing process and your development process are silo'ed and you're feeling the affects of this! Just remember everyone involved has a stake in the final product and QA aren't gate keepers for the product.
It’s a little difficult to get your hands on a device that can run Firefox OS right now, but if you’re interested in running the UI tests a device is not essential. This guide will show you how to run the tests on the nightly desktop client builds we provide.
The Firefox OS desktop client lets you run Gaia (the UI for Firefox OS) and web apps in a Gecko-based environment somewhat similar to an actual device. There are certain limitations of the desktop client, including: it doesn’t emulate device hardware (camera, battery, etc), it doesn’t support carrier based operations such as sending/receiving messages or calls, and it relies on the network connection of the machine it’s running on.
You can download the latest build of the desktop client from this location, but make sure you download the appropriate file for your operating system. Unfortunately, due to bug 832469 the nightly desktop builds do not currently work on Windows, so you will need either Mac or Linux (a virtual machine is fine) to continue:
Once downloaded, you will need to extract the contents to a local folder. For the purposes of the rest of this guide, I’ll refer to this location as $B2G_HOME.
Marionette is a test framework built into Gecko that allows remote control of the application. The Gaia UI tests use Marionette to launch applications and simulate a user interacting with them. By default, this is enabled in the desktop client but it is necessary for us to set a preference in the default profile before we can run the tests.
Add the following line to your gaia/profile/user.js file, which on Mac is located in $B2G_HOME/B2G.app/Contents/MacOS and on Linux in $B2G_HOME/b2g.
user_pref('marionette.force-local', true);
You can start Firefox OS by double clicking $B2G_HOME/B2G.app (Mac) or running $B2G_HOME/b2g/b2g (Linux). If everything went well, you should see the ‘powered by’ screen shortly followed by the first launch app. Complete the configuration steps and optionally follow the tour, and you will be presented with the lock screen. Unlock by dragging the bar up and clicking the padlock. You should be presented with the home screen (shown here).
Take a moment to familiarise yourself with Firefox OS. Launch a couple of applications, change some settings. You’ll soon discover the limitations of the simulator. Probably the most noticeable difference is that there’s no home/power/volume buttons as there would be on a device. The most useful of these is the home button, which allows you to return the to the home screen or to switch between open apps. You should be able to use the home key on your keyboard as a substitute. Here are some more usage tips.
Now you’ve got the simulator running, you can clone and run the automated UI tests against it. You will need to have git and Python installed (I recommend using version 2.7), and I highly recommend using virtual environments.
First, clone the gaia-ui-tests repository using the following command line, where $WORKSPACE is your local workspace folder:
cd $WORKSPACE git clone git://github.com/mozilla/gaia-ui-tests.git gaia-ui-tests cd gaia-ui-tests
If you’re using virtual environments, create a new environment and activate it. You will only need to create it once, but will need to activate it whenever you wish to run the tests:
virtualenv .env source .env/bin/activate
Now you need to install the test harness (gaiatest) and all of it’s dependencies:
python setup.py develop
Once this is done, you will have everything you need to run the tests. Because we’re running against the desktop client we must filter out all tests that are not appropriate. This list may grow, but it currently includes tests that use: antenna, bluetooth, carrier, camera, sdcard, and wifi. You will probably also want to exclude any tests that are expected to fail (xfail). To run the tests, use the following command:
gaiatest --address=localhost:2828 --type=b2g-antenna-bluetooth-carrier-camera-sdcard-wifi-xfail gaiatest/tests/manifest.ini
You should then start to see the tests running, with output similar to the following:
starting httpd running webserver on http://199.91.170.216:47413/ TEST-START test_settings.py test_get_all_settings (test_settings.TestSettings) ... ok test_set_named_setting (test_settings.TestSettings) ... ok test_set_volume (test_settings.TestSettings) ... ok ----------------------------------------------------------------- Ran 3 tests in 3.234s OK
The first tests that run are unit tests for the gaiatest harness, so you won’t immediately see much happening in the simulator. You may encounter test failures, and we’re currently focusing on getting these resolved. You may also encounter bug 844498, which has the nasty side-effect of causing all remaining tests to fail. If this happens just try running the suite again for now.
The video shows a full suite run against the simulator. Note that where tests time out I have either cropped the video or increased the speed. This is just to keep the video shorter.
Now you can run the tests, you’re in a great position to help us out! Our first focus is to get all the tests passing against the desktop build, but then we need to identify missing areas of coverage that are relevant to the simulator.
To contribute, you will need to set up a github account and then fork the main gaia-ui-tests repository. You will then need to update your local clone so it’s associated with your fork rather than the main one. You can do this with the following commands, replacing $USERNAME with your github username:
git config remote.origin.url git@github.com:$USERNAME/gaia-ui-tests.git git fetch origin git remote prune origin
You can now create a branch, and make your changes. Once done, you should commit your changes and push them to your fork before submitting a pull request. I’m not going to cover these steps in detail here, as they’re fairly standard git practices and will be covered in far better detail elsewhere. In fact, github:help has some fantastic documentation.
If you’re looking for a task, you should first check the desktop issues list on github. If there’s nothing available there, see if you can find an area that needs more coverage. Feel free to add an issue and a comment to say you’ll work on it.
You can also ask us for tasks! There are several mailing lists that you can sign up to: Automation Development, Web QA, and B2G QA. We’re also on IRC, and you can find us in #automation, #mozwebqa, and #appsqa all on irc.mozilla.org.
the following changes have been pushed to bugzilla.mozilla.org:
[ For more information on the Eideticker software I'm referring to, see this entry ]
Ok, so as I mentioned last time I’ve been looking into making Eideticker work for devices without native HDMI output by capturing their output with some kind of camera. So far I’ve tried four different DSLRs for this task, which have all been inadequate for different reasons. I was originally just going to write an email about this to a few concerned parties, but then figured I may as well structure it into a blog post. Maybe someone will find it useful (or better yet, have some ideas.)
Elmo MO-1
This is the device I mentioned last time. Easy to set up, plays nicely with the Decklink capture card we’re using for Eideticker. It seemed almost perfect, except for that:
Canon Rebel T4i
This is the first camera that was recommended to me at the camera shop I’ve been going to. It does have an HDMI output signal, but it’s not “clean”. This blog post explains the details better than I could. Next.
Nikon D600
Supposedly does native clean 720p output, but unfortunately the output is in a “box” so isn’t recognized by the Decklink cards that we’re using (which insist on a full 1280×720 HDMI signal to work). Next.
Nikon D800
It is possible to configure this one to not put a box around the output, so the Decklink card does recognize it. Except that the camera shuts off the HDMI signal whenever the input parameters change on the card or the signal input is turned on, which essentially makes it useless for Eideticker (this happens every time we start the Eideticker harness). Quite a shame, as the HDMI signal is quite nice otherwise.
–
To be clear, with the exception of the Elmo all the devices above seem like fine cameras, and should more than do for manual captures of B2G or Android phones (which is something we probably want to do anyway). But for Eideticker, we need something that works in automation, and none of the above fit the bill. I guess I could explore using a “real” video camera as opposed to a DSLR acting like one, though I suspect I might run into some of the same sorts of issues depending on how the HDMI output of those devices behaves.
Part of me wonders whether a custom solution wouldn’t work better. How complicated could it be to construct your own digital camera anyway? 
Hook up a fancy camera sensor to a pandaboard, get it to output through the HDMI port, and then we’re set? Or better yet, maybe just get a fancy webcam like the Playstation Eye and hook it up directly to a computer? That would eliminate the need for our expensive video capture card setup altogether.
Recently there was a bit of a rant on the Selenium users mailing list about how there were a few bugs irritating the person and, because they are only testers with not enough development experience, didnt feel that they could help with fixing these issues.
Note: this person was not rude and was not trolling so was happy to reach out.
One thing that I want people to know is that you don't have to be a brilliant developer to work on Open Source projects. You don't even have to have any experience as a software developer. You are now wondering how you can help?
By submitting a patch, even with mistakes an inexperienced would do, you are solving the problem. Once you have submitted it one thing is going to happen. Someone is going to look at your patch, eventually and reply. The reply will be we landed it or your patch needs some work. The latter is a good thing, even if you see it as a negative, since you will be learning.
Now, and is the most important thing to remember, is that the developer looking at your patch could very well learn something from you! Committers on Open Source projects tend to have come up through the ranks, unless they created the project, and have learnt what code does what where. They have learnt from other developers and want to share the knowledge back.
Going through all of these processes is a good thing. As people in software we have to be constantly learning or we fall behind. Once you are behind it can be hard to get employment in some companies you would love to work.
Next time you feel negative about contributing to Open Source, try change the thought to how you can learn something new and make a difference to your favourite projects. Selenium has getting involved bugs and Mozilla has its good first bugs that people can solve.
Don't be afraid to contribute to numerous projects because you will likely learn from a number of different people. This way you will get many years worth of meaningful experience instead of just many years of doing the same thing over and over.
following months of planning and hard work from both the BMO and IT teams, we will be upgrading bugzilla.mozilla.org to bugzilla version 4.2, with a tentative date of the 8th of march.
test drive it at https://bugzilla-dev.allizom.org/
the database is a sanitised snapshot of the production database so should be useful for testing to make sure the information is displayed properly and changeable. being that is it sanitised, private bugs, products, groups, attachments, and comments will not be present, and some features which rely on groups will not function.
email is disabled on this instance, which results in the normal account creation/verification process being unavailable. however you can use persona to create an account if required.
please point your various scripts and third party applications that use the XMLRPC/JSON API at the test server to make sure they continue to function properly.
a test instance of the BzAPI is available for testing your tools that need it – https://api-dev.bugzilla.mozilla.org/allizom/
as part of this upgrade, bugzilla.mozilla.org will be migrating from its current home in the PHX data-centre to shiny new infrastructure in the SCL3 data-centre. we will be running on a larger cluster with faster servers.
there are also numerous new features/fixes that are part of the upstream version 4.2, see the release notes for more information.
we are asking for everyone to get involved as much as possible with testing and feedback. file any bug reports in the production bugzilla system in the bugzilla.mozilla.org product.
thanks,
the bmo team
the following changes have been pushed to bugzilla.mozilla.org:
the following changes have been pushed to bugzilla.mozilla.org:
Could CSS 3, while is a great thing for the Internet and for web developers, be making websites that are extremely hard to automate?
As most of you know, in Selenium WebDriver we try an emulate what elements that a user can interact with. This means that we do a lot of DOM walking and gathering important little bits about the CSS on each of the elements to make sure that they are visible. You can see the details of what we do in the Determining Visibility section of the W3C Browser Automation Specification. Unfortunately every so often we get a situation where people start raising bugs about their Selenium WebDriver script with respect to allowing or allowing interaction with elements.
Recently we have started getting issues with how we handle CSS transforms. The first bug report came from the Web QA Team creating tests for Firefox OS. They were automating the Music app and noticed that tests were saying that an element was visible when it wasn't. I recreate the issue and make a minimised test case. You can see the element is miles away from the rest of the DOM below in the minimised test case below.
Marionette, the future of FirefoxDriver and what we use to automate Firefox OS, thinks that is visible. So down the slippery slope we go, add in support CSS Transforms, because that is the root cause for this bug. Now if you are wondering how complex transforms can get have a look at this MDN article.
It's just Algebraic transforms, no big deal right? But it can be when you start thinking in terms of cascading. Parent, or even great great grandparent element of the element we need to interact with could have the style applied. Now we need to walk the DOM from the element we want all the way back to body.
NOTE: Selenium has to walk the DOM a few times during various parts of the visibility checks which on large pages can be very expensive (therefore slow!). I am looking at you in particular large table element! We do the same when getting the text of elements
So while we are doing all of this work we used to assume that anything that mutates the element, like moving it across the page or changing its visibility, is going to happen in JavaScript and since the visibility code for Selenium is written in JavaScript, we will be blocking the JavaScript thread. CSS 3 has created things that have meant that we don't need JavaScript anymore to do animations
That means working with Selenium on something that is pure CSS animations might not work as you expect. So next time you decide to have a play with CSS 3 and it's awesomeness, have a play with Selenium at the same time and see how we fare. I suspect there are a lot of cases we don't cater for and we should!
[ For more information on the Eideticker software I'm referring to, see this entry ]
Here’s a long overdue update on where we’re at with Eideticker for FirefoxOS. While we’ve had a good amount of success getting useful, actionable data out of Eideticker for Android, so far we haven’t been able to replicate that success for FirefoxOS. This is not for lack of trying: first, Malini Das and then me have been working at it since summer 2012.
When it comes right down to it, instrumenting Eideticker for B2G is just a whole lot more complex. On Android, we could take the operating system (including support for all the things we needed, like HDMI capture) as a given. The only tricky part was instrumenting the capture so the right things happened at the right moment. With FirefoxOS, we need to run these tests on entire builds of a whole operating system which was constantly changing. Not nearly as simple. That said, I’m starting to see light at the end of the tunnel.
Platforms
We initially selected the pandaboard as the main device to use for eideticker testing, for two reasons. First, it’s the same hardware platform we’re targeting for other b2g testing in tbpl (mochitest, reftest, etc.), and is the platform we’re using for running Gaia UI tests. Second, unlike every other device that we’re prototyping FirefoxOS on (to my knowledge), it has HDMI-out capability, so we can directly interface it with the Eideticker video capture setup.
However, the panda also has some serious shortcomings. First, it’s obviously not a platform we’re shipping, so the performance we’re seeing from it is subject to different factors that we might not see with a phone actually shipped to users. For the same reason, we’ve had many problems getting B2G running reliably on it, as it’s not something most developers have been hacking on a day to day basis. Thanks to the heroic efforts of Thomas Zimmerman, we’ve mostly got things working ok now, but it was a fairly long road to get here (several months last fall).
More recently, we became aware of something called an Elmo which might let us combine
the best of both worlds. An elmo is really just a tiny mounted video camera with a bunch of outputs, and is I understand most commonly used to project documents in a classroom/presentation setting. However, it seems to do a great job of capturing mobile phones in action as well.
The nice thing about using an external camera for the video capture part of eideticker is that we are no longer limited to devices with HDMI out — we can run the standard set of automated tests on ANYTHING. We’ve already used this to some success in getting some videos of FirefoxOS startup times versus Android on the Unagi (a development phone that we’re using internally) for manual analysis. Automating this process may be trickier because of the fact that the video capture is no longer “perfect”, but we may be able to work around that (more discussion about this later).
FirefoxOS web page tests
These are the same tests we run on Android. They should give us an idea of roughly where our performance when browsing / panning web sites like CNN. So far, I’ve only run these tests on the Pandaboard and they are INCREDIBLY slow (like 1-3 frames per second when scrolling). So much so that I have to think there is something broken about our hardware acceleration on this platform.
FirefoxOS application tests
These are some new tests written in a framework that allows you to script arbitrary interactions in FirefoxOS, like launching applications or opening the task switcher.
I’m pretty happy with this. It seems to work well. The only problems I’m seeing with this is with the platform we’re running these tests on. With the pandaboard, applications look weird (since the screen resolution doesn’t remotely resemble the 320×480 resolution on our current devices) and performance is abysmal. Take, for example, this capture of application switching performance, which operates only at roughly 3-4 fps:
So what now?
I’m not 100% sure yet (partly it will depend on what others say as well as my own investigation), but I have a feeling that capturing video of real devices running FirefoxOS using the Elmo is the way forward. First, the hardware and driver situation will be much more representative of what we’ll actually be shipping to users. Second, we can flash new builds of FirefoxOS onto them automatically, unlike the pandaboards where you currently either need to manually flash and reset (a time consuming and error prone process) or set up an instance of mozpool (which I understand is quite complicated).
The main use case I see with eideticker-on-panda would be where we wanted to run a suite of tests on checkin (in tbpl-like fashion) and we’d need to scale to many devices. While cool, this sounds like an expensive project (both in terms of time and hardware) and I think we’d do better with getting something slightly smaller-scale running first.
So, the real question is whether or not the capture produced by the Elmo is amenable to the same analysis that we do on the raw HDMI output. At the very least, some of eideticker’s image analysis code will have to be adapted to handle a much “noisier” capture. As opposed to capturing the raw HDMI signal, we now have to deal with the real world and its irritating fluctuations in ambient light levels and all that the rest. I have no doubt it is *possible* to compensate for this (after all this is what the human eye/brain does all the time), but the question is how much work it will be. Can’t speak for anyone else at Mozilla, but I’m not sure if I really have the time to start a Ph.D-level research project in computational vision. I’m optimistic that won’t be necessary, but we’ll just have to wait and see.
the following changes have been pushed to bugzilla.mozilla.org:
the following changes have been pushed to bugzilla.mozilla.org:
As many of you know I have recently released the 2nd edition of my Selenium book. When I wrote my first book I looked for a way to do version control of my book. Being a developer and test type, I used git to do all of my version control but I found that I would sometimes forget to commit and push my changes. This broke my own rule of keeping commits small.
The other problem is that if the text editor crashes, or even the OS crashes or I get a hard drive crash, there is a chance that I might only be able to go back to my last commit and push. Since I already said I am rubbish at doing the commit and push thing with documents git was definitely out of question.
I went to look for a different solution. I remembered that Dropbox keeps a history of files when syncing. So I went and tested the theory with a directory within my Dropbox folder. After a few times of playing with a file and syncing and reverting, I thought I would give it a go with my first chapter. I instantly fell in love with this approach. My publisher wanted me to use Microsoft Word to write all of chapters I instantly reverted to my old university habits of ctrl+s every few minutes. And every time I saved my chapter Dropbox would instantly synchronise the file and update the history of the file.
This meant that just by saving my file I was getting the small commit chunks I like without the worry of remembering to push my changes. I have lost the ability to do commit messages but in this scenario I honestly don't care. All I worry about is making sure I have the most up to date files I can if something were to go wrong.
I highly recommend people using Dropbox in this way.
the following changes have been pushed to bugzilla.mozilla.org:
currently searching using quicksearch on bugzilla.mozilla.org queries comments as well as bug metadata. on janurary 24th we will be changing this to exclude comments by default.
if you enter a term into the quicksearch field in bugzilla’s header or footer, the current behavour is for bugzilla to search fields and comments on all open bugs. due to the quantity of bugs and comments on bugzilla, this can result in queries taking longer than expected. we will be changing the default behavour to no longer include comments.
if you want to retain the old behavour, login and change “Include comments when performing quick searches (slower)” in user preferences to “on”.
alternatively you can include “++comments” or “–comments” as a quicksearch term to exclude or include comments respectively.
the following changes have been pushed to bugzilla.mozilla.org:
Mozilla Automation and Testing: Signal from Noise, 2012
We've written up what we've been doing as part of the huge effort of the Signal from Noise project.
Look at:
Building Software can be a very expensive thing to do! Especially if you are building a web application that needs to be deployed.
An EC2 micro instance or Linode 512 machine will cost ~USD20 a month. This then needs to have all monitoring apps installed and maintained which costs more money. This might not be a lot to some of us but growing up in South Africa, that always felt like a lot of money. When you are building a new app for Firefox OS, paying for a server to serve your app might not be in your costs for building, testing and deploying your app.
What if you could limit your development and deployment costs to £0? Note your app needs to be open source to make the cost £0.
The current "go-to" source control at the moment is GitHub. I quite like it and for our free CI and deployment, we are going to have to have to use GitHub.
The FirefoxOS Marketplace team have created some template projects called mortar that give you all the necessary pieces to build an app. There are a number of different templates, from normal to games, for all your different needs.
Travis-CI is my favourite CI as a service at the moment. They are running the company as an open source company so I have a lot of love for these guys but I am digressing.
If you are using mortar then your .travis.yml will look like the following
language: node_js node_js: - 0.8
And you will also need to update your package.json file to specify how to run tests. I wrote my tests with python so it's the following
"scripts": {
"test": "py.test -n 3"
}
Now, if you want to maximise the use of your app you will need to make sure it works on as many browsers as possible. My friends at Saucelabs have recently created an Open Sauce account giving you free usage and it works well with Travis CI. To get it working you just add the following to your .travis.yml
env:
global:
- secure: "sauce user name"
- secure: "sauce labs key"
matrix:
- SAUCE_PLATFORM='Windows 2003' SAUCE_BROWSER_VERSION=18
You can see a working example here.
And the have something like the following in your test setup. It will also set the browser size to the default FirefoxOS view port
def setUp(self):
try:
os.environ['SAUCE_USERNAME']
desired_capabilities = webdriver.DesiredCapabilities.FIREFOX
desired_capabilities['version'] = os.environ['SAUCE_BROWSER_VERSION']
desired_capabilities['platform'] = os.environ['SAUCE_PLATFORM']
desired_capabilities['name'] = 'Testing Get In The Habit'
self.driver = webdriver.Remote(
desired_capabilities=desired_capabilities,
command_executor="http://%s:%s@ondemand.saucelabs.com:80/wd/hub" % \
(os.environ['SAUCE_USERNAME'],
os.environ['SAUCE_ACCESS_KEY'])
)
except Exception as e:
print e
self.driver = webdriver.Firefox()
self.driver.set_window_size(320, 480)
self.driver.get("http://localhost:8008")
Now that we have built and tested our app it's time to deploy!
One of the things about our apps for Firefox OS is that it can be packaged on to the phone or can be served from a web server. GitHub pages make this a rather trivial exercise and also allows you to grow your user base without having to know how to maintain servers. You can also have a look at things like Launcher.io / AppFog / Heroku that have free tiers. If you need to have a backend for your app for synchronising of data then I highly recommend one of them.
As B2G continues to trod onwards to its release, there is still a lot of confusion about the level and state of test coverage it has. Back in November we started running mochitests, reftests and marionette/webapi tests on ARM emulators. Now we've also added xpcshell tests and for the most part we have these nice green letters to look at on TBPL that make us feel good about ourselves. But what is really being run? What is the meaning behind these letters "M", "R", "Mn" and "X"? Are there any causes for concern? Are there other tests being run that don't show up on TBPL? What are the current automation priorities? What are the next platforms to use after emulators?
This blog post aims to answer these questions and more. It is a comprehensive snapshot of the current state of automated testing on B2G.
Pandaboards are still the future of automated testing on B2G. We've hit many problems with them over the course of the last two quarters, but all the pieces are starting to fall together. In fact we have tests running on pandas on the Cedar branch already (N.B that these are to test the infrastructure, not the product).
Mozpool is the system that will be used to control and assign pandas. The build system will send a job request to mozpool which will analyze the available devices and return the IP of a device that meets all of the requirements. Before doing so, it will invoke lifeguard which will perform diagnostic tests on the device and remove it from the pool if it is unsuitable for testing. Lifeguard will use BlackMobileMagic to perform it's low-level operations on the device, such as diagnosing network issues, restarting, retrieving device info etc. All of these components are currently completed, tested and awaiting the test harnesses.
Getting Gaia UI tests running on pandas is well under way. After that we will be shifting focus on the B2G unittests (mochitest, reftest, marionette/webapi and xpcshell tests). Bug 807126 comes to mind as an important bug that we'll need to complete before we can even start running unittests on pandaboards. It is currently on my radar for sometime in Q1 but has been slipping down my priority list lately.
A subset of mochitest-plain are being run on the emulator. There are no plans for mochitest-chrome. Mochitests will also be used for B2G permissions testing. Mochitests are rolled out to all branches and are being staged on the Cedar branch. These are denoted "M" on TBPL.
See the full list of mochitests running on b2g. Currently it's only some of the DOM and layout tests, but these are in the process of getting expanded (bug 793045).
Overall mochitests have been pretty stable save for a few intermittent harness issues.
try: -b o -p ics_armv7a_gecko -u mochitests -t none
Reftests are being run against the ARM emulators. They are rolled out to all branches and are being staged on Cedar. These are denoted "R" on TBPL.
Only the reftest-sanity tests!!! Yes, there is practically no test coverage here at the moment. The tracking bug to expand the set of tests is bug 811779. The patch in this bug should give a relatively green run of all the reftests but we simply don't have the capacity to turn them on. Instead I'm in the process of triaging a subset of reftests that Chris Jones deemed "important" on Cedar. These should be ready to turn on soon which is why there are so many chunks.
See the main reftest tracking bug for a full list of issues associated with reftests on B2G. Some of the highlights include:
try: -b o -p ics_armv7a_gecko -u reftest-1,reftest-2,reftest-3,reftest-4,reftest-5,reftest-6,reftest-7,reftest-8,reftest-9,reftest-10 -t none
Marionette and WebAPI tests are a combination of marionette unittests for testing marionette itself and some B2G webapi tests. These are rolled out to all branches and are being staged on Cedar. They are denoted with an "Mn" on TBPL.
All of the marionette unit tests. In addition there are many other webapi tests being run. These include tests for telephony, battery, sms, network and more.
The webapi tests tend to be much more crashy than any of the other unit tests. Currently there are a lot of instability issues caused by B2G process crashes and full out emulator crashes.
try: -b o -p ics_armv7a_gecko -u marionette-webapi -t none
XPCShell tests were just recently added. They are rolled out on all branches and are being staged on Cedar. These are denoted "X" on TBPL.
The xpcshell tests being run include the update tests, the ril tests, the debugger tests and a handful of others. This is just a small subset of tests that were chosen to start out with. If you know of any other tests that should be getting run feel free to let me know (or just add them yourself after verifying that they pass).
The xpcshell tests seem to be quite reliable on B2G. There are a few open bugs, but nothing near as bleak as e.g reftests or marionette/webapi.
try: -b o -p ics_armv7a_gecko -u xpcshell -t none
The gaia UI tests (aka gaia smoketests) are a set of tests being run by the WebQA team. They are running automatically in a Jenkins instance but are not currently being reported in TBPL. You can see the results (you must be on the MV network to see that link, sorry).
You can check out the current set of tests being run from github.
There are some test stability issues and some issues that appear to be legitimate failures. There are ~13 tests (roughly half) that are currently passing and stable on the pandaboards. The others require things which a pandaboard doesn't have (e.g camera) and can't be run.
Some of the Gaia integration tests are being run by the WebQA team in a Jenkins CI. Others are being run manually on shipping devices by gaia developers. These are targetted to run on pandaboards after the Gaia UI tests are finished and running stable.
Eideticker is a performance testing harness that captures HDMI out and performs frame by frame analysis. There should be B2G specific eideticker tests running by end of Q4 or early Q1.
These tests are currently running on unagis and are reporting data to datazilla. Once the remaining panda issues have been ironed out, these will provide some basic per push performance numbers.
face is an irc bot i run on the #ateam channel.
its responses are purely random, drawing from past conversations on channel.
<jedp> hello ateam, is there anyone there at this hour?
<face> jedp, greetings
<jedp> face hola1
* face gets a cup of tea
<jedp> face I want to send an observer message with marionette. Can it be done?
* face swoons
<jedp> face likes tea
* face can certainly file one if you want. It’s whatever 12 are in the first case we built
<jedp> face – or if marionette doesn’t have a way to send observer messages, maybe with SpecialPowers?
* face feels loved
<jedp> face I tried SpecialPowers.sendSyncMessage, thinking that would be there from this, but no dice: https://developer.mozilla.org/en-US/docs/SpecialPowers
* face goes afk
<glob> jedp, sorry, face is actually a bot 
<jedp> glob i was just coming to that conclusion
the following changes have been pushed to bugzilla.mozilla.org:
the following changes have been pushed to bugzilla.mozilla.org:
face is an irc bot i run on the #ateam channel.
its responses are purely random, drawing from past conversations on channel.
<glob> face, say.. do you know perl?
* face bows
<glob> face, great! want to work on bugzilla’s search code?
* face disbelieves, because that would be Dumb
<glob> crap.
the following changes have been pushed to bugzilla.mozilla.org:
the following changes have been pushed to bugzilla.mozilla.org:
I bet that the moment most people decide they actually do like git is when they start using 'rebase' regularly. I definitely do not completely understand the git model, but rebase shows that there is some seriously cool stuff going on.
Anyway, I've come upon a rebasing dilemma. The reasons for not rebasing a public repo are clear, but pushing to a remote origin (e.g. github) is also a form of backup. My master branches are for collaboration, but my dev branches are essentially just to back up my home computer, and occasionally for feedback. I rebase dev branches regularly, to keep my commits together for eventual merging to master. I occasionally switch around or squash commits too, where it adds clarity to the history. So, somewhat shamefully, I find myself using 'git push -f' a lot on branches other than master.
I guess I could get a paid account and fork private dev repos, but branches seem a lot more convenient, and I don't really want to hide anything from anyone, as embarrasing as some of my work in progress sometimes seems to me.
Maybe the moral of this story is "never pull from my dev branches"?
today’s out-of-band push is to address the double-comment issue when setting the needinfo flag.
the following changes have been pushed to bugzilla.mozilla.org:
the following changes have been pushed to bugzilla.mozilla.org:
Perils of Version Pegging in Python Packaging
When working on an ecosystem of python packages where some packages depend on other packages, it becomes a question what versions of the dependencies to require. There are three basic choices:
There is no magic bullet: all of these strategies have advantages and disadvantages. In general, the API of dependencies will change and a consumer of a particular version will only work with a certain range of versions of the dependency. Because it is in general unknown whether the next version of a dependency will break the API for consuming software, there is not a blanket strategy whereby compatability can be guaranteed via a setup.py file.
Considering the cases, case 1. allows for the most flexibility: if any version of dependency (bar) is registered, the dependency is satisfied. (Otherwise, the latest version of the dependency will be downloaded and installed from e.g. http://pypi.python.org/ .) However, case 1. is very vulnerable to API changes in the dependency: it does nothing to ensure that the dependency is compatible with the consuming software. Assuming that the latest versions of a set of packages are internally compatible, a fresh install will give an internally compatible set of packages. However, if a package is updated there is nothing to guarantee that the API is compatible.
Case 2 is the most strict: the consuming package demands a particular version of a dependency. If this strategy is followed for all dependencies, it can be assured that for a particular version of the consuming software (foo) that a compatible version of the dependency (bar) is used. However, this is done at the price of losing forwards compatability. If a new version of the dependency (bar) is available, it will not be used regardless of compatability.
Case 3 seeks to balance the alternatives: the consuming packages demands a version of a dependency of at least a given version. This protects from using an API that is too old for the package of interest. This strategy also allows newer versions of the software to be installed without complaining. If the API hasn't changed, then this is good. However, this still does not protect from API changes. If the newest version of bar has a different API from the minimum version specified in foo's setup.py, while setup.py won't complain, the software will not work. Ideally, one would be able to note post-facto that there was an API-breaking change in the new version and that all software pegged to bar >= 0.1 should really be bar >= 0.1, bar < 1.0. However, once a distribution of (e.g.) foo is released, it cannot meaningfully be re-released.
This week the A*Team came together as part of the larger Boot2Gecko work week in SF. We worked with a bunch of people across various teams and got a bunch of stuff landed.
TBPL
We’ve been working toward automation on TBPL for a while. This week we worked with Aki from releng to get mochitest, reftest, and emulator-based WebAPI tests running on Mozilla-Central, Try, Services-Central, Mozilla-Inbound, Fx-Team, Cedar, and Ash. We are set to add them to Mozilla-Aurora on Monday.
Gaia tests/QA Support
The Stephen Donner’s Web QA team is working on automating the Gaia smoketests. This week we pivoted hard to get those stood up in CI. We have them running in our Jenkins instance on an Unagi device and reporting to Autolog.
We met with Geo’s Web API team and made some fixes to the B2G mochitest suite so that they are unblocked and can continue creating webAPI mochitests.
Rob Wood and Dave Hunt continue working closely with the QA team to help them generate more tests. This week, Rob added ten more webSMS emulator tests alone.
PandaBoards
Pandaboards are our on-change solution for automation that needs to run on device, like the Gaia automated smoketests. There are many moving parts to stand these up at scale. Here is a run down of what happened this week:
Eideticker Performance System
This week, Will Lachance got all the existing Eideticker benchmarks running using a pandaboard hooked to an Eideticker system. He has uploaded a screencast of the tasks.js panning test being performed on B2G here.
Next Steps
We’re not done yet. Here are the things we’re working on next:
Contrary to popular belief, we've been running mochitests, reftests, marionette tests and webapi tests on B2G in some form of continuous integration or another for about 5 months now. They've been reporting results to a TBPL look-alike called autolog, and were run on Amazon EC2 VM's with emulators. This was a temporary solution to get something stood up quickly while we moved towards our ultimate B2G automation goal - tests running on Pandaboards and reporting to TBPL.
As of this week, while there are still no tests running on Pandaboards, I'm happy to say we have emulators running mochitests, reftests and marionette/webapi tests, all reporting to TBPL. It might seem surprising (dismaying?) that it has taken as long as it has to get to this point. Especially seeing as it's only emulators running very bare subsets of tests. The fact is I am saddened as well, though not surprised. I could go on a very long rant about how we managed to get this far without proper continuous integration, but I'll leave that for another post. The short version is that automation tends to be very fickle and error prone by nature. When the thing you are testing is in a constant state of flux (like B2G was) it makes for a bad combination. If the thing you are testing has many different components, each with differing development processes, it gets worse. When you throw in platforms with poor performance characteristics (like emulators and devices) and when said platforms need to be controlled remotely, you are bound to become sad.
But the point of this post isn't to make you sad, it's to assure you that things are getting better. In the near to mid-future look for:
Contrary to popular belief, we've been running mochitests, reftests, marionette tests and webapi tests on B2G in some form of continuous integration or another for about 5 months now. They've been reporting results to a TBPL look-alike called autolog, and were run on Amazon EC2 VM's with emulators. This was a temporary solution to get something stood up quickly while we moved towards our ultimate B2G automation goal - tests running on Pandaboards and reporting to TBPL.
As of this week, while there are still no tests running on Pandaboards, I'm happy to say we have emulators running mochitests, reftests and marionette/webapi tests, all reporting to TBPL. It might seem surprising (dismaying?) that it has taken as long as it has to get to this point. Especially seeing as it's only emulators running very bare subsets of tests. The fact is I am saddened as well, though not surprised. I could go on a very long rant about how we managed to get this far without proper continuous integration, but I'll leave that for another post. The short version is that automation tends to be very fickle and error prone by nature. When the thing you are testing is in a constant state of flux (like B2G was) it makes for a bad combination. If the thing you are testing has many different components, each with differing development processes, it gets worse. When you throw in platforms with poor performance characteristics (like emulators and devices) and when said platforms need to be controlled remotely, you are bound to become sad.
But the point of this post isn't to make you sad, it's to assure you that things are getting better. In the near to mid-future look for:
One of the biggest things that always shocks me is the way people approach problems. Software developers are essentially pattern matchers using experience to figure out how to solve problems that are placed in front of them.
This can unfortunately lead them to a problem where developers use their current knowledge to solve problems. Imagine that you are in QA and your only experience of automation is using Selenium. You are told that you need to test a REST API, or as I have seen in forums to test stored procedures in your database, you go for the tool that you know and feel comfortable with. The downside to this is that you could possibly pick the wrong tool!
If you are thinking that people don't use Selenium to test Databases/SOAP/REST think again and go have a read of the Selenium User Forum!
Selenium is a wonderful tool! I am not just saying that because I have written two books or because part of my time is to try support the project, I say it because I love watching the tests run! Unfortunately it uses a browser to drive a web application. This means that you have the entire stack to worry about when testing. Actually you have more than your stack to worry about when testing. A Browser is a very complex bit of software. Both Google and Mozilla have shown that it can be its own operating system with Chrome OS and Firefox OS respectively.
That is a lot of lines of code that are run just to make sure that your tests do what they are expected. When you need a browser you use it and then make sure everything is working. The browser start up time annoys you but for the coverage you are getting it is acceptable.
But when you are testing something that isn't rendering there is a lot of wasted parts of the browser stack that are called and aren't really used.
So when running your tests thing about how much is being invoked and how much really needs to be invoked. Try not to be wasteful with the resources that are consumed. When little resources are consumed in a test then the tests will run run quicker making feed back loops take less time.
the following changes have been pushed to bugzilla.mozilla.org:
Last week there was a very large gathering for the W3C in Lyon, France. The TPAC conference gets together all working groups and allows them to share their specifications with everyone else and demo implementations if there are some complete.
The Browser Automation Working group was there too to discuss moving the current specification further. We had 2 days of meetings and below is a break down of the topics we discussed.
On the first day we discussed the following
Discussion of open issues in the spec:
The main items that came from the work this week are:
We will be cleaning up a few sections and hopefully be pushing the next version of the specification in the next few weeks. If you have thoughts or you spot any errors please let us know!
Keeping wiki pages up to date is a hard problem, but recently we found out that people were having trouble finding out what projects we were working on. Obviously we can't help people with their problems if they can't figure out what we do, so I spent some time today updating the A-Team's Project Central. All the projects we are working on are there, along with owners' IRC nicks and links to project pages and/or docs. We also have links to our quarterly goals as well as to SmartSheet pages with details of our progress.
We'll do our best to keep it up to date!
It has been two years to the day that I joined Mozilla. I have mentioned in the past that I came in having missed the contributor to employee route and, as I said then, feel like I have missed out.
So what have the last two years been like? If I was only allowed one word it would be "Rollercoaster". Now I know that you are probably thinking that I chose it for the cliche of ups and downs. I chose the word because it is full on, and at a tremendous speed, and with out a doubt, fun!
So what have I done in the last 2 years at Mozilla? I have worked in 3 different teams. I started out in the WebQA team helping standardise the way they write their Selenium tests. They have gone on to do much much more than I expected and I love watching what this team does. It is after all my first home in Mozilla.
The next team that I was in was Automation Services. This was a team that I co-lead with Henrik Skupin and we did some really good things. This team was then merged with the Automation and Tools team, the A-Team as they are known internally, which is my current team.
I have worked on everything from web applications to Firefox Mobile to Firefox itself. I am co-editing a W3C standard with Simon Stewart to make sure that web developers and software testers can increase the quality of the products that they deliver to the web. I have worked on Frameworks and production code and I have written a lot of tests.
All the while knowing that everything I do is making a difference to the way the web is being used and worked on. It is one of the main things that drives to me to want to work at Mozilla. One of the other main things that I have found interesting, and to be honest still shocks me, is the warmth that people exhibit when they find that you work for Mozilla.
So after two years at Mozilla I can honestly say that I am working with a large number of brilliant people, though I have always been lucky with my previous employers, who continue to teach me, guide me and to give me opportunities to show how brilliant our craft and industry is!
P.s. Want to help Mozilla? Have a look at What can I do for Mozilla?
I promised a few people I’d blog about this, so here you go.
To help with the business of making Android or FirefoxOS devices do our bidding, Mozilla Automation & Tools developed a Python library called mozdevice which allows you to control these devices either using the Android Debug Bridge protocol (which is actually not Android specific: FirefoxOS devices use it too) or the System Under Test protocol (a Mozilla-specific thing).
Anyone familiar with debugging these devices is doubtless familiar with adb, which provides a command line interface that allows you to push/pull files, run a shell, etc. To help test our python code (as well as expand the scope of what’s possible on the command line), I created a similar utility a few months ago called “dm” which provides a command-line interface to the aforementioned mozdevice code. It’s shipped as part of mozdevice, and testing it out is pretty simple if you have virtualenv installed:
virtualenv mozdevice cd mozdevice ./bin/pip install mozdevice source bin/activate dm --help
I generally use this utility for two things:
Testing out mozdevice code. For example, today we discovered an (unfortunate) bug in devicemanagerADB’s killProcess routine. It was easy to verify both the problem and my fix did what I expected by starting my custom build of fennec and running this command:
dm --package-name org.mozilla.fennec_wlach killapp org.mozilla.fennec_wlach
(yes, it’s a bit unfortunate that this bug occurred in the first place: devicemanagerADB really needs unit tests)
Day-to-day menial tasks, like getting device info/status, capturing screenshots, etc. You can get a full list of what this utility is capable of by running –help. E.g.:
(mozbase)wlach@eideticker:~/src/eideticker$ dm --help Usage: dm [options][] device commands: info [os|id|uptime|systime|screen|memory|processes] - get information on a specified aspect of the device (if no argument given, print all available information) install - push this package file to the device and install it killapp - kills any processes with a particular name on device launchapp - launches application on device ls - list files on device ps - get information on running processes on device pull [remote] - copy file/dir from device push - copy file/dir to device rm - remove file from device rmdir - recursively remove directory from device screencap - capture screenshot of device in action shell - run shell command on device Options: -h, --help show this help message and exit -v, --verbose Verbose output from DeviceManager --host=HOST Device hostname (only if using TCP/IP) -p PORT, --port=PORT Custom device port (if using SUTAgent or adb-over-tcp) -m DMTYPE, --dmtype=DMTYPE DeviceManager type (adb or sut, defaults to adb) -d HWID, --hwid=HWID HWID --package-name=PACKAGENAME Packagename (if using DeviceManagerADB)
Before you ask, yes, it’s technically possible to do much of this with the original adb utility. But (1) some of our internal stuff can’t use adb a variety of reasons and (2) some of the tasks above (e.g. launching an app, getting a screenshot) involve considerably more typing with adb than with dm. So it’s still a win.
Happy command-lining!
the following changes have been pushed to bugzilla.mozilla.org:
Development of an extensive test suite to verify the functionality of the first group of Firefox OS WebAPIs is underway. Ultimately the test suite will run on the B2G device emulator on a per-check-in basis.
To develop a WebAPI test which will run on the B2G device emulator, which automation framework is best suited for the task: Mochitest or Marionette? This blog post will help answer that question, and provide some useful emulator tips.
The Mochitest automation framework has access to gecko and the WebAPIs but cannot access the lower-level emulated ‘hardware’ features and commands that are offered in the B2G emulator. The Marionette automation framework can access both the WebAPIs and the emulated ‘hardware’ and associated emulator commands. Therefore, if the WebAPI test under development requires direct access to the emulated hardware support and special emulator functions, then Marionette should be used; otherwise Mochitest is sufficient for the task.
For example, if you are developing a new WebAPI test for the ContactsAPI, Mochitest is the best choice as no special emulated hardware access or functions is required to test Contacts. However, if you are developing a WebAPI test for the WebTelephony API, Marionette is the automation framework of choice because the B2G device emulator has special commands that allow the partial-simulation of phone calls.
The B2G device emulator offers console commands that can help emulate specific hardware features, and therefore be used to verify some of the WebAPIs. To see a list of commands (or to try out some of them yourself) start-up the B2G device emulator, then open a terminal window and connect to the emulator’s android console via telnet:
$ telnet localhost 5554
Where ’5554′ is the default emulator port (displayed on the emulator window title). Once connected to the emulator’s android console, simply type ‘help’ to see a list of available commands. As an example, to use the emulator to simulate an incoming phone call, issue the following command in the android emulator console:
gsm call 5551112222
Where ’5551112222′ is the phone number of the simulated remote mobile from which the incoming call is originated. Once the above command is issued, notice that the emulator gui (gaia) indicates that there is an incoming call from the specified number. The simulated call can then be answered or rejected using more emulator console commands (or via the gaia interface itself).
To send a console command to the emulator from within a WebAPI test, Marionette provides the ‘runEmulatorCmd’ function. For example, to simulate an incoming call on the emulator use the following js code in your WebAPI test:
let inNumber = “5551112222″;
runEmulatorCmd(“gsm call ” + inNumber);
As an optional argument to ‘runEmulatorCmd’, a callback function can be provided which will be invoked after the emulator processes the command, and the emulator console output will be provided. For example:
runEmulatorCmd(“gsm list”, function(result) {
log(“Current call list: ” + result);
});
The ‘gsm list’ command asks the emulator to output a list of currently simulated phone calls and their status. In the above example the resulting console output of the command (list of simulated calls, or just ‘ok’ if no calls exist) is provided to the callback function. For more information see Marionette JavaScript Tests in MDN.
Additional information about the Automation Development team’s efforts to help with WebAPI test development can be found on our team’s projects wiki. If you are interested in contributing please contact me or find us in the #automation room on IRC.
(Thanks JGriffin and MDas for answering my various questions on these subjects.)
i’ve been working on a bugzilla enhancement which allows you to tag individual comments with arbitrary strings.
it’s tracked in bug 793963, currently waiting for review, and will be back-ported to bugzilla.mozilla.org.
comment tagging features:
automatic collapsing of comments
the bugzilla administrator can configure a list of comment tags which will result in those comments being collapsed by default when a bug is loaded.
this allows obsolete or irrelevant comments to be hidden from the information stream.
comment grouping/threading
bugzilla will show a list of all comment tags in use on the bug, and clicking on a tag will expand those comments while collapsing all others.
this allows for simple threading of comments without diverging significantly from the current bugzilla user interface, api, and schema. you’ll be able to tag all comments relating to the same topic, and remove comments no longer relevant to that thread by removing the tag.
highlighting importing comments
on bugs with a lot of information, it can be time consuming for people not directly involved in the bug to find the relevant comments. applying comment tags to the right comments assists this, and may negate the need for information to be gathered outside of bugzilla.
for example:
implementation notes
Just a few quick notes on how to avoid a class of errors I’ve been seeing in Mozilla’s automation over the last year. Since python interprets code dynamically, it’s pretty easy for problems like undefined variables to slip through, especially if they’re in a codepath that isn’t frequently tested. The most recent example I found was in some cleanup-after-error code for remote mochitest/reftest, which tried to call “self.cleanup” from a standalone method.
def main():
...
try:
dm.recordLogcat()
retVal = mochitest.runTests(options)
logcat = dm.getLogcat()
except:
print "TEST-UNEXPECTED-FAIL | %s | Exception caught while running tests." % sys.exc_info()[1]
mochitest.stopWebServer(options)
mochitest.stopWebSocketServer(options)
try:
self.cleanup(None, options)
except:
pass
testing/mochitest/runtestsremote.py
We’re calling cleanup as if it were a class variable, but we’re not inside any class! It’s easy to see what will happen if you try to run some similar code from the python console:
>>> self.cleanup() Traceback (most recent call last): File "", line 1, in NameError: name 'self' is not defined
However, because we’re in a blanket try…except, we will never see an error. The cleanup code will never be called, instead the exception is immediately caught and subsequently ignored. Probably not the end of the world in this case (there are other parts of our mobile automation which will perform the same cleanup later), but it’s easy to imagine where this would be a more serious problem.
There’s two very easy ways to help stop errors like this before they hit our code:
Try to avoid using a blanket try…except. In addition to catching legitimate problems which we want to ignore (in the remote case for example, devicemanager exceptions), it also catches (and thus obscures) things like syntax, name, or type errors. Instead, try just catching the specific exception you’re looking for. For example, we might rewrite the case above as:
try: mochitest.cleanup(None, options) except devicemanager.DMError: print "WARNING: Device error while cleaning up"
pyflakes, pyflakes, pyflakes. Pyflakes is a fantastic tool for analyzing your python code for common problems. It’s kind of analagous to jslint, for those of you familiar with that. Here’s what happens when we run pyflakes against the offending code:
wlach@eideticker:~/src/mozilla-central$ pyflakes testing/mochitest/runtestsremote.py testing/mochitest/runtestsremote.py:7: 'time' imported but unused testing/mochitest/runtestsremote.py:481: undefined name 'self' testing/mochitest/runtestsremote.py:500: undefined name 'self'
I’ve found pyflakes to be an indispensable part of my workflow. I generally run it after making any substantial change to a python file, and certainly before pushing anything to be consumed by others.
Ultimately there’s no substitute for actually thoroughly testing your code, no matter what language you’re using. But using the right techniques and tools can certainly make your life easier.
[ for those wondering, a fix for the issue mentioned in this post is part of bug 801652 ]
An amusingly frequent pattern:
git clone https://github.com/mozilla/new-python-project
# ... right, virtualenv
mkdir src
mv new-python-project src
virtualenv new-python-project
cd new-python-project
mv ../src .
. bin/activate
# get to work
I really ought to make a script to clone new Python projects...
the following changes have been pushed to bugzilla.mozilla.org:
So it is official, the 2nd version of my book is officially out! It has been a lot of work but I am happy with the current result. Selenium 2 Testing tools is a beginners guide that walks the user from creating easy throw away tests with Selenium IDE to being able to use Selenium WebDriver on desktop Operating Systems to being able to run them on mobile devices.
It is a beginners guide so if you know how to code you should be able to get started and by the end of the book be pretty confident at using Selenium WebDriver on a number of different platforms and devices.
You can buy the book from Packt Publishing or from Amazon.com or from Amazon.co.uk.
I hope that you enjoy it!
an excerpt from https://wiki.mozilla.org/BMO/Integration_Best_Practice
a few notes about the needinfo flag which was added to bugzilla.mozilla.org last week, and enabled across all products today:
