Matjaž HorvatTerminology Search in Pontoon

New release of Pontoon is out the door. It’s mostly a bugfix release eliminating annoying glitches like broken contributor profile links. Thank you for your first contribution to Pontoon, Benoit! :-)

Some new features are also available, e.g. displaying warnings on unsaved translations as suggested by flod. And — Terminology Search is now also available as a standalone feature, making it easier to access. It works similarly as the Search tab in the out-of-context translation panel.

Translations are taken from:

Byron Joneshappy bmo push day!

the following changes have been pushed to bugzilla.mozilla.org:

  • [1159166] When I ask for review from someone who is not accepting reviews, my red request count in the top left becomes their request count
  • [1159307] Can we add the Rank field to Product “Toolkit”
  • [1151745] add ui to minimise steps required to move bugs between products
  • [1157124] don’t report sql search errors to senty
  • [1153100] add mozreview’s table to bug-modal
  • [1159282] form.dev-engagement-event: stop creating a second “discussion” bug

discuss these changes on mozilla.tools.bmo.


Filed under: bmo, mozilla

Air MozillaMozilla Weekly Project Meeting

Mozilla Weekly Project Meeting The Monday Project Meeting

Mozilla Science LabMozilla Science Lab Week in Review, April 27 – May 3

The Week in Review is our weekly roundup of what’s new in open science from the past week. If you have news or announcements you’d like passed on to the community, be sure to share on Twitter with @mozillascience and @billdoesphysics, or join our mailing list and get in touch there.

Awards & Grants

  • Applications for the PLOS Early Career Travel Award Program are now open; ten $500 awards are available to help early career researchers publishing in PLOS attend meetings and conferences to present their work.

Tools & Resources

Blogs & Papers

  • A study led by the Center for Open Science that attempted to replicate the findings of 100 journal articles in psychology has concluded, with data posted online; 39 of the articles investigated were reproduced, with substantial similarities found in several dozen more.
  • The Joint Research Centre of the European Commission has released an interim report on their ongoing work in ‘Analysis of emerging reputation mechanisms for scholars'; the report maps an ontology of research-related activities onto the reputation-building activities that attempt to capture them, and reviews the social networks that attempt to facilitate this construction of reputation on the web.
  • Alyssa Goodman et al published Ten Simple Rules for the Care and Feeding of Scientific Data in PLOS Computational Biology. In it, the authors touch not only on raw data, but the importance of permanent identifiers by which to identify it, and the context provided by publishing workflows in addition to code.
  • David Takeuchi wrote about his concerns that the American federal government’s proposed FIRST act will curtail funding for the social sciences, and place too much emphasis on perceived relevance at the expense of reproducibility.
  • The Georgia Tech Computational Linguistics Lab blogged about the results of a recent graduate seminar where students were set to reproducing the results of several papers in computational social science. The author makes several observations on the challenges faced, including the difficulties in reproducing results based on social network or other proprietary information, and on the surprising robustness of machine-learning driven analyses.
  • Cobi Smith examined both the current state and future importance of open government data in Australia.

Meetings & Conferences

Mike ConleyElectrolysis and the Big Tab Spinner of Doom

Have you been using Firefox Nightly and seen this big annoying spinner?

Big Tab Spinner of Doom in an e10s tab

Aw, crap. You again.

I hate that thing. I hate it.

Me, internally, when I see the spinner.

And while we’re working on making the spinner itself less ugly, I’d like to eliminate, or at least reduce its presence to the absolute minimum.

How do I do that? Well, first, know your enemy.

What does it even mean?

That big spinner means that the graphics part of Gecko hasn’t given us a frame yet to paint for this browser tab. That means we have nothing yet to show for the tab you’ve selected.

In the single-process Firefox that we ship today, this graphics operation of preparing a frame is something that Firefox will block on, so the tab will just not switch until the frame is ready. In fact, I’m pretty sure the whole browser will become unresponsive until the frame is ready.

With Electrolysis / multi-process Firefox, things are a bit different. The main browser process tells the content process, “Hey, I want to show the content associated with the tab that the user just selected”, and the content process computes what should be shown, and when the frame is ready, the parent process hears about it and the switch is complete. During that waiting time, the rest of the browser is still responsive – we do not block on it.

So there’s this window of time where the tab switch has been requested, and when the frame is ready.

During that window of time, we keep showing the currently selected tab. If, however, 300ms passes, and we still haven’t gotten a frame to paint, that’s when we show the big spinner.

So that’s what the big spinner means – we waited 300ms, and we still have no frame to draw to the screen.

How bad is it?

I suspect it varies. I see the spinner a lot less on my Windows machine than on my MacBook, so I suspect that performance is somehow worse on OS X than on Windows. But that’s purely subjective. We’ve recently landed some Telemetry probes to try to get a better sense of how often the spinner is showing up, and how laggy our tab switching really is. Hopefully we’ll get some useful data out of that, and as we work to improve tab switch times, we’ll see improvement in our Telemetry numbers as well.

Where is the badness coming from?

This is still unclear. And I don’t think it’s a single thing – many things might be causing this problem. Anything that blocks up the main thread of the content process, like slow JavaScript running on a web-site, can cause the spinner.

I also seem to see the spinner when I have “many” tabs open (~30), and have a build going on in the background (so my machine is under heavy load).

Maybe we’re just doing things inefficiently in the multi-process case. I recently landed profile markers for the Gecko Profiler for async tab switching, to help figure out what’s going on when I experience slow tab switch. Maybe there are optimizations we can make there.

One thing I’ve noticed is that there’s this function in the graphics layer, “ClientTiledLayerBuffer::ValidateTile”, that takes much, much longer in the content process than in the single-process case. I’ve filed a bug on that, and I’ll ask folks from the Graphics Team this week.

How you can help

If you’d like to help me find more potential causes, Profiles are very useful! Grab the Gecko Profiler add-on, make sure it’s enabled, and then dump a profile when you see the big spinner of doom. The interesting part will be between two markers, “AsyncTabSwitch:Start” and “AsyncTabSwitch:Finish”. There are also markers for when the parent process displays the spinner – “AsyncTabSwitch:SpinnerShown” and “AsyncTabSwitch:SpinnerHidden”. The interesting stuff, I believe, will be in the “Content” section of the profile between those markers. Here are more comprehensive instructions on using the Gecko Profiler add-on.

And here’s a video of me demonstrating how to use the profiler, and how to attach a profile to the bug where we’re working on improving tab switch times:

And here’s the link I refer you to in the video for getting the add-on.

So hopefully we’ll get some useful data, and we can drive instances of this spinner into the ground.

I’d really like that.

Gregory SzorcReporting Mercurial Issues

I semi-frequently stumble upon conversations in hallways and on irc.mozilla.org about issues people are having with Mercurial. These conversations periodically involve a legitimate bug with Mercurial. Unfortunately, these conversations frequently end without an actionable result. Unless someone files a bug, pings me, etc, the complaints disappear into ether. That's not good for anyone and only results in bugs living longer than they should.

There are posters around Mozilla offices that say if you see something, file something. This advice does not just apply to Mozilla projects!

If you encounter an issue in Mercurial, please take the time to report it somewhere meaningful. The Reporting Issues with Mercurial page from the Mercurial for Mozillians guide tells you how to do this.

It is OK to complain about something. But if you don't inform someone empowered to do something about it, you are part of the problem without being part of the solution. Please make the incremental effort to be part of the solution.

Laura HilligerOpen Web Leadership

Over the last couple of weeks, we’ve been talking about an organizing structure for future (and current) Teach Like Mozilla content and curriculum. This stream of curriculum is aimed at helping leaders gain the competencies and skills needed for teaching, organizing and sustaining learning for the web. We’ve been short-handing this work “Open Fluency” after I wrote a post about the initial thinking.

Last week, in our biweekly community call, we talked about the vision for our call. In brief, we want to:

“Work together to define leadership competencies and skills, as well as provide ideas and support to our various research initiatives.”

We decided to change the naming of this work to “Open Web Leadership”, with a caveat that we might find a better name sometime in the future. We discussed leadership in the Mozilla context and took some notes on what we view as “leadership” in our community. We talked about the types of leadership we’ve seen within the community, noted that we’ve seen all sorts, and, in particular, had a lengthy conversation about people confusing management with leadership.

We decided that as leaders in the Mozilla Community, we want to be collaborative, effective, supported, compassionate for people’s real life situations. We want to inspire inquiry and exploration and ensure that our community can make independent decisions and take ownership. We want to be welcoming and encouraging, and we are especially interested in making sure that as leaders, we encourage new leaders to come forward, grow and participate.

I believe it was Greg who wrote in the call etherpad:

“Open Web Leaders engage in collaborative design while serving as a resource to others as we create supportive learning spaces that merge multiple networks, communities, and goals.”

Next, we discussed what people need to feel ownership and agency here in the Mozilla community. People expressed some love for the type of group work we’re doing with Open Web Leadership, pointing out that working groups who make decisions together fuels their own participation. It was pointed out that the chaos of the Mozilla universe should be a forcing function for creating on-boarding materials for getting involved, and that a good leader:

“Makes sure everyone “owns” the project”

There’s a lot in that statement. Giving ownership and agency to your fellow community members requires open and honest communication, not one time but constantly. No matter how much we SAY it, our actions (or lack of action) color how people view the work (as well as each other).

After talking about leadership, we added the progressive “ing” form to the verbs we’re using to designate each Open Web Leadership strand. I think this was a good approach as to me it signifies that understanding, modeling and uniting to TeachTheWeb are ongoing and participatory practices. Or, said another way, lifelong learning FTW! Our current strands are:

  • Understanding Participatory Learning (what you need to know)
  • Modeling Processes and Content (how you wield what you know)
  • Uniting Locally and Globally (why you wield what you know)

We established a need for short, one line descriptors on each strand, and decided that the competency “Open Thinking” is actually a part of “Open Practices”. We’ll refine and further develop this in future calls!

As always, you’re invited to participate. There are tons of thought provoking Github issues you can dive into (coding skills NOT required), and your feedback, advice, ideas and criticisms are all welcome.

Gregory SzorcMercurial 3.4 Released

Mercurial 3.4 was released on May 1 (following Mercurial's time-based schedule of releasing a new version every 3 months).

3.4 is a significant release for a few reasons.

First, the next version of the wire protocol (bundle2) has been marked as non-experimental on servers. This version of the protocol paves over a number of deficiencies in the classic protocol. I won't go into low-level details. But I will say that the protocol enables some rich end-user experiences, such as having the server hand out URLs for pre-generated bundles (e.g. offload clones to S3), atomic push operations, and advanced workflows, such as having the server rebase automatically on push. Of course, you'll need a server running 3.4 to realize the benefits of the new protocol. hg.mozilla.org won't be updated until at least June 1.

Second, Mercurial 3.4 contains improvements to the tags cache to make performance concerns a thing of the past. Due to the structure of the Firefox repositories, the previous implementation of the tags cache could result in pauses of dozens of seconds during certain workflows. The problem should go away with Mercurial 3.4. Please note that on first use of Mercurial 3.4, your repository may perform a one-time upgrade of the tags cache. This will spin a full CPU core and will take up to a few minutes to complete on Firefox repos. Let it run to completion and performance should not be an issue again. I wrote the patches to change the tags cache (with lots of help from Pierre-Yves David, a Mercurial core contributor). So if you find anything wrong, I'm the one to complain to.

Third, the HTTP interface to Mercurial (hgweb) now has JSON output for nearly every endpoint. The implementation isn't yet complete, but it is better than nothing. But, it should be good enough for services to start consuming it. Again, this won't be available on hg.mozilla.org until the server is upgraded on June 1 at the earliest. This is a feature I added to core Mercurial. If you have feature requests, send them my way.

Fourth, a number of performance regressions introduced in Mercurial 3.3 were addressed. These performance issues frequently manifested during hg blame operations. Many Mozillians noticed them on hg.mozilla.org when looking at blame through the web interface.

For a more comprehensive list of changes, see my post about the 3.4 RC and the official release notes.

3.4 was a significant release. There are compelling reasons to upgrade. That being said, there were a lot of changes in 3.4. If you want to wait until 3.4.1 is released (scheduled for June 1) so you don't run into any regressions, nobody can fault you for that.

If you want to upgrade, I recommend reading the Mercurial for Mozillians Installation Page.

Daniel StenbergHTTP/2 in curl, status update

http2 logoI’m right now working on adding proper multiplexing to libcurl’s HTTP/2 code. So far we’ve only done a single stream per connection and while that works fine and is HTTP/2, applications will still want more when switching to HTTP/2 as the multiplexing part is one of the key components and selling features of the new protocol version.

Pipelining means multiplexed

As a starting point, I’m using the “enable HTTP pipelining” switch to tell libcurl it should consider multiplexing. It makes libcurl work as before by default. If you use the multi interface and enable pipelining, libcurl will try to re-use established connections and just add streams over them rather than creating new connections. Yes this means that A) you need to use the multi interface to get the full HTTP/2 stuff and B) the curl tool won’t be able to take advantage of it since it doesn’t use the multi interface! (An old outstanding idea is to move the tool to use the multi interface and this would yet another reason why this could be a good idea.)

We still have some decisions to make about how we want libcurl to act by default – especially when we can expect application to use both HTTP/1.1 and HTTP/2 at the same time. Since we don’t know if the server supports HTTP/2 until after a certain point in the negotiation, we need to decide on how to do when we issue N transfers at once to the same server that might speak HTTP/2… Right now, we get the best HTTP/2 behavior by telling libcurl we only want one connection per host but that is probably not ideal for an application that might use a mix of HTTP/1.1 and HTTP/2 servers.

Downsides with abusing pipelining

There are some drawbacks with using that pipelining switch to allow multiplexing since users may very well want HTTP/2 multiplexing but not HTTP/1.1 pipelining since the latter is just riddled with interop problems.

Also, re-using the same options for limited connections to host names etc for both HTTP/1.1 and HTTP/2 may not at all be what real-world applications want or need.

One easy handle, one stream

libcurl API wise, each HTTP/2 stream is its own easy handle. It makes it simple and keeps the API paradigm very much in the same way it works for all the other protocols. It comes very natural for the libcurl application author. If you setup three easy handles, all identifying a resource on the same server and you tell libcurl to use HTTP/2, it makes perfect sense that all these three transfers are made using a single connection.

As multiplexed data means that when reading from the socket, there is data arriving that belongs to other streams than just a single one. So we need to feed the received data into the different “data buckets” for the involved streams. It gives us a little internal challenge: we get easy handles with no socket activity to trigger a read, but there is data to take care of in the incoming buffer. I’ve solved this so far with a special trigger that says that there is data to take care of, that it should make a read anyway that then will get the data from the buffer.

Server push

HTTP/2 supports server push. That’s a stream that gets initiated from the server side without the client specifically asking for it. A resource the server deems likely that the client wants since it asked for a related resource, or similar. My idea is to support server push with the application setting up a transfer with an easy handle and associated options, but the URL would only identify the server so that it knows on which connection it would accept a push, and we will introduce a new option to libcurl that would tell it that this is an easy handle that should be used for the next server pushed stream on this connection.

Of course there are a few outstanding issues with this idea. Possibly we should allow an easy handle to get created when a new stream shows up so that we can better deal with a dynamic number of  new streams being pushed.

It’d be great to hear from users who have ideas on how to use server push in a real-world application and how you’d imagine it could be used with libcurl.

Work in progress code

My work in progress code for this drive can be found in two places.

First, I do the libcurl multiplexing development in the separate http2-multiplex branch in the regular curl repo:

https://github.com/bagder/curl/tree/http2-multiplex.

Then, I put all my test setup and test client work in a separate repository just in case you want to keep up and reproduce my testing and experiments:

https://github.com/bagder/curl-http2-dev

Feedback?

All comments, questions, praise or complaints you may have on this are best sent to the curl-library mailing list. If you are planning on doing a HTTP/2 capable applications or otherwise have thoughts or ideas about the API for this, please join in and tell me what you think. It is much better to get the discussions going early and work on different design ideas now before anything is set in stone rather than waiting for us to ship something semi-stable as the closer to an actual release we get, the harder it’ll be to change the API.

Not quite working yet

As I write this, I’m repeatedly doing 99 parallel HTTP/2 streams with no data corruption… But there’s a lot more to be done before I’ll call it a victory.

Andy McKayTFSA

In the budget the Conservatives increased the TFSA in allowance from $5,500 to $10,000. This was claimed to be:

11 million Cdns use Tax-Free Savings Accounts. #Budget2015 will increase limit from $5500 to $10000. Help for low and middle income Cdns.

Wai Young MP

"Low income" really? According to Revenue Canada we can see that most people are not to maxing out their TFSA room. In fact since 2009, the amount of unused contribution has been growing each year.

Year Aveage unused contribution Change
2009 $1,156.29
2010 $3,817.25 $2,660.96
2011 $6,692.37 $2,875.12
2012 $9,969.19 $3,276.83

People are having trouble keeping up with TFSA contributions as it is. But what's low income? Depends how you define it, there's a few ways.

LICO is an income threshold below which a family will likely devote a larger share of its income to the necessities of food, shelter and clothing than an average family would

Statscan

And "Thus for 2011, the 1992 based after-tax LICO for a family of four living in an community with a population between 30,000 and 99,999 is $30,487, expressed in current dollars.". That is after tax.

Of that income by definition, over 50% of the families incomes is on food, shelter and clothing. Meaning that there's $15,243 left. Maybe, these are all averages and many people will be way, way worse off.

Is $10,000 in TFSA reasonable for families who have less than $15,243 a year? No. It benefits people with more money and the ability to save. Further we can see that the actual amount of money going into TFSA has been dropping every year since its creation and the unused contribution has been growing.

There isn't really a good justification for increasing the TFSA except as a way of helping the rich just before the election.

Mike ConleyThings I’ve Learned This Week (April 27 – May 1, 2015)

Another short one this week.

You can pass DOM Promises back through XPIDL

XPIDL is what we use to define XPCOM interfaces in Gecko. I think we’re trying to avoid XPCOM where we can, but sometimes you have to work with pre-existing XPCOM interfaces, and, well, you’re just stuck using it unless you want to rewrite what you’re working on.

What I’m working on lately is nsIProfiler, which is the interface to “SPS”, AKA the Gecko Profiler. nsIProfiler allows me to turn profiling on and off with various features, and then retrieve those profiles to send to a file, or to Cleopatra1.

What I’ve been working on recently is Bug 1116188 – [e10s] Stop using sync messages for Gecko profiler, which will probably have me adding new methods to nsIProfiler for async retrieval of profiles.

In the past, doing async stuff through XPCOM / XPIDL has meant using (or defining a new) callback interface which can be passed as an argument to the async method.

I was just about to go down that road, when ehsan (or was it jrmuizel? One of them, anyhow) suggested that I just pass a DOM Promise back.

I find that Promises are excellent. I really like them, and if I could pass a Promise back, that’d be incredible. But I had no idea how to do it.

It turns out that if I can ensure that the async methods are called such that there is a JS context on the stack, I can generate a DOM Promise, and pass it back to the caller as an “nsISupports”. According to ehsan, XPConnect will do the necessary magic so that the caller, upon receiving the return value, doesn’t just get this opaque nsISupports thing, but an actual DOM Promise. This is because, I believe, that DOM Promise is something that is defined via WebIDL. I think. I can’t say I fully understand the mechanics of XPConnect2, but this all sounded wonderful.

I even found an example in our new Service Worker code:

From dom/workers/ServiceWorkerManager.cpp (I’ve edited the method to highlight the Promise stuff):

// If we return an error code here, the ServiceWorkerContainer will
// automatically reject the Promise.
NS_IMETHODIMP
ServiceWorkerManager::Register(nsIDOMWindow* aWindow,
                               nsIURI* aScopeURI,
                               nsIURI* aScriptURI,
                               nsISupports** aPromise)
{
  AssertIsOnMainThread();

  // XXXnsm Don't allow chrome callers for now, we don't support chrome
  // ServiceWorkers.
  MOZ_ASSERT(!nsContentUtils::IsCallerChrome());

  nsCOMPtr<nsPIDOMWindow> window = do_QueryInterface(aWindow);

  // ...

  nsCOMPtr<nsIGlobalObject> sgo = do_QueryInterface(window);
  ErrorResult result;
  nsRefPtr<Promise> promise = Promise::Create(sgo, result);
  if (result.Failed()) {
    return result.StealNSResult();
  }

  // ...

  nsRefPtr<ServiceWorkerResolveWindowPromiseOnUpdateCallback> cb =
    new ServiceWorkerResolveWindowPromiseOnUpdateCallback(window, promise);

  nsRefPtr<ServiceWorkerRegisterJob> job =
    new ServiceWorkerRegisterJob(queue, cleanedScope, spec, cb, documentPrincipal);
  queue->Append(job);

  promise.forget(aPromise);
  return NS_OK;
}

Notice that the outparam aPromise is an nsISupports**, and yet, I do believe the caller will end up handling a DOM Promise. Wicked!


  1. Cleopatra is the web application that can be used to browse a profile retrieved via nsIProfiler 

  2. Like being able to read the black speech of Mordor, there are few who can. 

Mike ConleyThe Joy of Coding (Ep. 12): Making “Save Page As” Work

After giving some updates on the last bug we were working on together, I started a new bug: Bug 1128050 – [e10s] Save page as… doesn’t always load from cache. The problem here is that if the user were to reach a page via a POST request, attempting to save that page from the Save Page item in the menu would result in silent failure1.

Luckily, the last bug we were working on was related to this – we had a lot of context about cache keys swapped in already.

The other important thing to realize is that fixing this bug is a bandage fix, or a wallpaper fix. I don’t think those are official terms, but it’s what I use. Basically, we’re fixing a thing with the minimum required effort because something else is going to fix it properly down the line. So we just need to do what we can to get the feature to limp along until such time as the proper fix lands.

My proposed solution was to serialize an nsISHEntry on the content process side, deserialize it on the parent side, and pass it off to nsIWebBrowserPersist.

So did it work? Watch the episode and find out!

I also want to briefly apologize for some construction noise during the video – I think it occurs somewhere halfway through minute 20 of the video. It doesn’t last long, I promise!

Episode Agenda

References

Bug 1128050 – [e10s] Save page as… doesn’t always load from cache – Notes


  1. Well, it’d show something in the Browser Console, but for a typical user, I think that’s still a silent failure. 

Mozilla Release Management TeamFirefox 38 beta8 to beta9

In this beta, 16 changesets are test-only or NPOTB changes. Besides those patches, we took graphic fixes, stabilities improvements and polish fixes.

  • 38 changesets
  • 87 files changed
  • 713 insertions
  • 287 deletions

ExtensionOccurrences
h22
cpp14
js12
cc6
html4
txt2
py2
jsm2
ini2
xml1
sh1
list1
java1
css1
c1
build1

ModuleOccurrences
ipc13
dom11
js10
browser8
media5
image5
gfx5
mobile3
layout3
toolkit2
mfbt2
xpcom1
uriloader1
tools1
testing1
services1
python1
parser1
netwerk1
mozglue1
docshell1
config1

List of changesets:

Ryan VanderMeulenBug 1062496 - Disable browser_aboutHome.js on OSX 10.6 debug. a=test-only - 657cfe2d4078
Ryan VanderMeulenBug 1148224 - Skip timeout-prone subtests in mediasource-duration.html on Windows. a=test-only - 82de02ddde1b
Ehsan AkhgariBug 1095517 - Increase the timeout of browser_identity_UI.js. a=test-only - 611ca5bd91d4
Ehsan AkhgariBug 1079617 - Increase the timeout of browser_test_new_window_from_content.js. a=test-only - 1783df5849c7
Eric RahmBug 1140537 - Sanity check size calculations. r=peterv, a=abillings - a7d6b32a504c
Hiroyuki IkezoeBug 1157985 - Use getEntriesByName to search by name attribute. r=qdot, a=test-only - 55b58d5184ce
Morris TsengBug 1120592 - Create iframe directly instead of using setTimeout. r=kanru, a=test-only - a4f506639153
Gregory SzorcBug 1128586 - Properly look for Mercurial version. r=RyanVM, a=NPOTB - 49abfe1a8ef8
Gregory SzorcBug 1128586 - Prefer hg.exe over hg. r=RyanVM, a=NPOTB - a0b48af4bb54
Shane TomlinsonBug 1146724 - Use a SendingContext for WebChannels. r=MattN, r=markh, a=abillings - 56d740d0769f
Brian HackettBug 1138740 - Notify Ion when changing a typed array's data pointer due to making a lazy buffer for it. r=sfink, a=sledru - e1fb2a5ab48d
Seth FowlerBug 1151309 - Part 1: Block until the previous multipart frame is decoded before processing another. r=tn, a=sledru - 046c97d2eb23
Seth FowlerBug 1151309 - Part 2: Hide errors in multipart image parts both visually and internally. r=tn, a=sledru - 0fcbbecc843d
Alessio PlacitelliBug 1154518 - Make sure extended data gathering (Telemetry) is disabled when FHR is disabled. r=Gijs, a=sledru - cb2725c612b2
Bas SchoutenBug 1151821 - Make globalCompositeOperator work correctly when a complex clip is pushed. r=jrmuizel, a=sledru - 987c18b686eb
Bas SchoutenBug 1151821 - Test whether simple canvas globalCompositeOperators work when a clip is set. r=jrmuizel, a=sledru - 1bbb50c6a494
Bob OwenBug 1087565 - Verify the child process with a secret hello on Windows. r=dvander, a=sledru - c1f04200ed98
Randell JesupBug 1157766 - Mismatched DataChannel initial channel size in JSEP database breaks adding channels. r=bwc, a=sledru - a8fb9422ff13
David MajorBug 1130061 - Block version 1.5 of vwcsource.ax. r=bsmedberg, a=sledru - 053da808c6d9
Martin ThomsonBug 1158343 - Temporarily enable TLS_RSA_WITH_AES_128_CBC_SHA for WebRTC. r=ekr, a=sledru - d10817faa571
Margaret LeibovicBug 1155083 - Properly hide reader view tablet on landscape tablets. r=bnicholson, a=sledru - f7170ad49667
Steve FinkBug 1136309 - Rename the spidermonkey build variants. r=terrence, a=test-only - 604326355be0
Mike HommeyBug 1142908 - Avoid arm simulator builds being considered cross-compiled. r=sfink, a=test-only - 517741a918b0
Jan de MooijBug 1146520 - Fix some minor autospider issues on OS X. r=sfink, a=test-only - 620cae899342
Steve FinkBug 1146520 - Do not treat osx arm-sim as a cross-compile. a=test-only - a5013ed3d1f0
Steve FinkBug 1135399 - Timeout shell builds. r=catlee, a=test-only - b6bf89c748b7
Steve FinkBug 1150347 - Fix autospider.sh --dep flag name. r=philor, a=test-only - b8f7eabd31b9
Steve FinkBug 1149476 - Lengthen timeout because we are hitting it with SM(cgc). r=me (also jonco for a more complex version), a=test-only - 16c98999de0b
Chris PearceBug 1136360 - Backout 3920b67e97a3 to fix A/V sync regressions (Bug 1148299 & Bug 1157886). r=backout a=sledru - 4ea8cdc621e8
Patrick BrossetBug 1153463 - Intermittent browser_animation_setting_currentTime_works_and_pauses.js. r=miker, a=test-only - c31c2a198a71
Andrew McCreightBug 1062479 - Use static strings for WeakReference type names. r=ehsan, a=sledru - 5d903629f9bd
Michael ComellaBug 1152314 - Duplicate action bar configuration in code. r=liuche, a=sledru - cdfd06d73d17
Ethan HuggBug 1158627 - WebRTC return error if GetEmptyFrame returns null. r=jesup, a=sledru - f1cd36f7e0e1
Jeff MuizelaarBug 1154703 - Avoid using WARP if nvdxgiwrapper.dll is around. a=sledru - 348c2ae68d50
Shu-yu GuoBug 1155474 - Consider the input to MThrowUninitializedLexical implicitly used. r=Waldo, a=sledru - daaa2c27b89f
Jean-Yves AvenardBug 1149605 - Avoid potential integers overflow. r=kentuckyfriedtakahe, a=abillings - fcfec0caa7be
Ryan VanderMeulenBacked out changeset daaa2c27b89f (Bug 1155474) for bustage. - 0a1accb16d39
Shu-yu GuoBug 1155474 - Consider the input to MThrowUninitializedLexical implicitly used. r=Waldo, a=sledru - ff65ba4cd38a

Christian HeilmannStart of my very busy May speaking tour and lots of //build videos to watch

I am currently in the Heathrow airport lounge on the first leg of my May presenting tour. Here is what lies ahead for me (with various interchanges in other countries in between to get from one to the other):

  • 02-07/05/2015 – Mountain View, California for Spartan Summit (Microsoft Edge now)
  • 09/05/2015 – Tirana, Albania for Oscal (opening keynote)
  • 11/05/2015 – Düsseldorf, Germany for Beyond Tellerand
  • 13-14/05/2015 – Verona, Italy – JSDay (opening keynote)
  • 15/05/2015 – Thessaloniki, Greece – DevIt (opening keynote)
  • 18/05/2015 – Amsterdam, The Netherlands – PhoneGap Day (MC)
  • 27/05/2015 – Copenhagen, Denmark – At The Frontend
  • 29/05/2015 – Prague, Czech Republic – J and Beyond

All packed and ready to go

I will very likely be too busy to answer a lot of requests this month, and if you meet me, I might be disheveled and unkempt – I never have more than a day in a hotel. The good news is that I have written 3 of these talks so far.

To while away the time on planes with my laptop being flat, I just downloaded lots of videos from build to watch (you can do that on each of these pages, just do the save-as), so I am up to speed with that. Here’s my list, in case you want to do the same:

Morgan PhillipsTo Serve Developers

The neatest thing about release engineering, is the fact that our pipeline forms the primary bridge between users and developers. On one end, we maintain the CI infrastructure that engineers rely on for thorough testing of their code, and, on the other end, we build stable releases and expose them for the public to download. Being in this position means that we have the opportunity to impact the experiences of both contributors and users by improving our systems (it also makes working on them a lot of fun).

Lately, I've become very interested in improving the developer experience by bringing our CI infrastructure closer to contributors. In short, I would like developers to have access to the same environments that we use to test/build their code. This will make it:
  • easier to run tests locally
  • easier to set up a dev environment
  • easier to reproduce bugs (especially environment dependent bugs)

[The release pipeline from 50,000ft]

How?

The first part of my plan revolves around integrating release engineering's CI system with a tool that developers are already using: mach; starting with a utility called: mozbootstrap -- a system that detects its host operating system and invokes a package manager for installing all of the libraries needed to build firefox desktop or firefox android.

The first step here was to make it possible to automate the bootstrapping process (see bug: 1151834 "allow users to bootstrap without any interactive prompts"), and then integrate it into the standing up of our own systems. Luckily, at the moment I'm also porting some of our Linux builds from buildbot to TaskCluster (see bug: 1135206), which necessitates scrapping our old chroot based build environments in favor of docker containers. This fresh start has given me the opportunity begin this transition painlessly.

This simple change alone strengthens the interface between RelEng and developers, because now we'll be using the same packages (on a given platform). It also means that our team will be actively maintaining a tool used by contributors. I think it's a huge step in the right direction!

What platforms/distributions are you supporting?

Right now, I'm only focusing on Linux, though in the future I expect to support OSX as well. The bootstrap utility supports several distributions (Debian/Ubuntu/CentOS/Arch), though, I've been trying to base all of release engineering's new docker containers on Ubuntu 14.04 -- as such, I'd consider this our canonical distribution. Our old builders were based on CentOS, so it would have been slightly easier to go with that platform, but I'd rather support the platform that the majority of our contributors are using.

What about developers who don't use Ubuntu 14.04, and/or have a bizarre environment

One fabulous side effect of using TaskCluster is that we're forced to create docker containers for running our jobs, in fact, they even live in mozilla-central. That being the case, I've started a conversation around integrating our docker containers into mozbootstrap, giving it the option to pull down a releng docker container in lieu of bootstrapping a host system.

On my own machine, I've been mounting my src directory inside of a builder and running ./mach build, then ./mach run within it. All of the source, object files, and executables live on my host machine, but the actual building takes place in a black box. This is a very tidy development workflow that's easy to replicate and automate with a few bash functions [which releng should also write/support].


[A simulation of how I'd like to see developers interacting with our docker containers.]

Lastly, as the final nail in the coffin of hard to reproduce CI bugs, I'd like to make it possible for developers to run our TaskCluster based test/build jobs on their local machines. Either from mach, or a new utility that lives in /testing.

If you'd like to follow my progress toward creating this brave new world -- or heckle me in bugzilla comments -- check out these tickets:

Christopher ArnoldCalling Android users: Help Mozilla Map the World!

Many iPhone users may have wondered why Apple prompts them with a message saying “Location accuracy is improved when Wi-Fi is turned on” each time they choose to turn Wi-Fi off.  Why does a phone that has GPS (Global Positioning Satellite) capability need to use Wi-Fi to determine it’s location?

The reason is fairly simple.  There are of course thousands of radio frequencies traveling through the walls of buildings all around us.  What makes Wi-Fi frequency (or even bluetooth) particularly useful for location mapping is that the frequency travels a relatively short distance before it decays, due to how low energy the Wi-Fi wavelengths are.  A combination of three or more Wi-Fi signals can be used in a very small area by a phone to triangulate locations on a map in the same manner that earthquake shockwave strengths can be used to triangulate epicenters.  Wi-Fi hubs don't need to transmit their locations to be useful.  Most are oblivious of their location.  It is the phone's interpretations of their signal strength and inferred location that creates the value to the phone's internal mapping capabilities.  No data that goes over the Wi-Fi frequency is  relevant to using radio for triangulation.  It is merely the signal strength/weakness that makes it useful for triangulation.  (Most Wi-Fi hubs are password protected and the data sent over them is encrypted.) 

Being able to let phone users determine their own location is of keen interest to developers who can’t make location-based-services work without fairly precise location determinations.  The developers don't want to track the users per se.  They want the users to be able to self-determine location when they request a service at a precise location in space.  (Say requesting a Lyft ride or checking in at a local eatery.)  There are a broad range of businesses that try to help phones accurately orient themselves on maps.  The data that each application developer uses may be different across a range of phones.  Android, Windows and iPhones all have different data sources for this, which can make it frustrating to have consistency of app experience for many users, even when they’re all using the same basic application.

At Mozilla, we think the best way to solve this problem is to create an open source solution.  We are app developers ourselves and we want our users to have consistent quality of experience, along with all the websites that our users access using our browsers and phones.  If we make location data accessible to developers, we should be able to help Internet users navigate their world more consistently.  By doing it in an open source way, dozens of phone vendors and app developers can utilize this open data source without cumbersome and expensive contracts that are sometimes imposed by location service vendors.  And as Mozilla we do this in a way that empowers users to make personal choice as to whether they wish to participate in data contribution or not.

How can I help?  There are two ways Firefox users can get involved.  (Several ways that developers can help.)  We have two applications for Android that have the capability to “stumble” Wi-Fi locations.

The first app is called “Mozilla Stumbler” and is available for free download in the Google Play store. (https://play.google.com/store/apps/details?id=org.mozilla.mozstumbler)  By opening MozStumbler and letting it collect radio frequencies around you, you are able to help the location database register those frequencies so that future users can determine their location.  None of the data your Android phone contributes can be specifically tied to you.  It’s collecting the ambient radio signals just for the purpose of determining map accuracy.  To make it fun to use MozStumbler, we have also created a leaderboard for users to keep track of their contributions to the database. 


Second app is our Firefox mobile browser that runs on the Android operating system.  (If it becomes possible to stumble on other operating systems, I’ll post an update to this blog.)  You need to take a couple of steps to enable background stumbling on your Firefox browser.  Specifically, you have to opt-in to share location data to Mozilla.  To do this, first download Firefox on your Android device.  On the first run you should get a prompt on what data you want to share with Mozilla.  If you bypassed that step, or installed Firefox a long time ago, here’s how to find the setting:



1) Click on the three dots at the right side of the Firefox browser chrome then select "Settings" (Above image)

2) Select Mozilla (Right image)

Check the box that says “Help Mozilla map the world! Share approximate Wi-Fi and cellular location of your device to improve our geolocation services.” (Below image)

If you ever want to change your settings, you can return to the settings of Firefox, or you can view your Android device's main settings menu on this path: Settings>Personal>Location which is the same place where you can see all the applications you've previously granted access to look up your physical location.

The benefit of the data contributed is manifold:
1) Firefox users on PCs (which do not have GPS sensors) will be able to determine their positions based on the frequency of the WiFi hotspots they use rather than having to continually require users to type in specific location requests. 
2) Apps on Firefox Operating System and websites that load in Firefox that use location services will perform more accurately and rapidly over time.
3) Other developers who want to build mobile applications and browsers will be able to have affordable access to location service tools.  So your contribution will foster the open source developer community.

And in addition to the benefits above, my colleague Robert Kaiser points out that even devices with GPS chips can benefit from getting Wi-Fi validation in the following way:
"1) When trying to get a location via GPS, it takes some time until the chip actually has seen signals from enough satellites to determine a location ("get a fix"). Scanning the visible wi-fi signals is faster than that, so getting an initial location is faster that way (and who wants to wait even half a minute until the phone can even start the search for the nearest restaurant or cafe?).
2) The location from this wifi triangulation can be fed into the GPS system, which enables it to know which satellites it roughly should expect to see and therefore get a fix on those sooner (Firefox OS at least is doing that).
3) In cities or buildings, signals from GPS satellites get reflected or absorbed by walls, often making the GPS position inaccurate or not being able to get a fix at all - while you might still see enough wi-fi signals to determine a position."

Thank you for helping improve Mozilla Location Services.

If you'd like to read more about Mozilla Location Services please visit:
https://location.services.mozilla.com/
To see how well our map currently covers your region, visit:
https://location.services.mozilla.com/map#2/15.0/10.0
If you are a developer, you can also integrate our open source code directly into your own app to enable your users to stumble for fun as well.  Code is available here: https://github.com/mozilla/DemoStumbler
For an in-depth write-up on the launch of the Mozilla Location Service please read Hanno's blog here: http://blog.hannosch.eu/2013/12/mozilla-location-service-what-why-and.html
For a discussion of the issues on privacy management view Gervase's blog:
http://blog.gerv.net/2013/10/location-services-and-privacy/








Robert AccetturaOn Deprecating HTTP

Mozilla announced:

There’s pretty broad agreement that HTTPS is the way forward for the web. In recent months, there have been statements from IETF, IAB (even the other IAB), W3C, and the US Government calling for universal use of encryption by Internet applications, which in the case of the web means HTTPS.

I’m on board with this development 100%. I say this as a web developer who has, and will face some uphill battles to bring everything into HTTPS land. It won’t happen immediately, but the long-term plan is 100% HTTPS . It’s not the easiest move for the internet, but it’s undoubtedly the right move for the internet.

A brief history

The lack of encryption on the internet is not to different from the weaknesses in email and SMTP that make spam so prolific. Once upon a time the internet was mainly a tool of academics, trust was implicit and ethics was paramount. Nobody thought security was of major importance. Everything was done in plain text for performance and easy debugging. That’s why you can use telnet to debug most older popular protocols.

In 2015 the landscape has changed. Academic use of the internet is a small fraction of its traffic. Malicious traffic is a growing concern. Free sharing of information, the norm in the academic world is the exception in some of the places the internet reaches.

Protecting the user

Users deserve to be protected as much as technology will allow. Some folks claim “non-sensitive” data exist. I disagree with this as it’s objective and a matter of personal perspective. What’s sensitive to someone in a certain situation is not sensitive to others. Certain topics that are normal and safe to discuss in most of the world are not safe in others. Certain search queries are more sensitive than others (medical questions, sensitive business research). A web developer doesn’t have a good grasp of what is sensitive or not. It’s specific to the individual user. It’s not every network admin’s right to know if someone on their network browsed and/or purchased pregnancy tests or purchased a book on parenting children with disabilities on Amazon. The former may not go over well at a “free” conservative school in the United States for example. More than just credit card information is considered “sensitive data” in this case. Nobody should be so arrogant as to think they understand how every person on earth might come across their website.

Google and Yahoo took the first step to move search to HTTPS (Bing still seems to be using HTTP oddly enough). This is the obvious second step to protecting the world’s internet users.

Protecting the website’s integrity

Michelangelo David - CensoredUnfortunately you can no longer be certain a user sees a website as you intended it as a web developer. Sorry, but it doesn’t work that way. For years ISP’s have been testing the ability to do things like insert ads into webpages. As far as I’m aware in the U.S. there’s nothing explicitly prohibiting replacing ads. Even net neutrality rules seem limited to degrading or discriminating against certain traffic, not modifying payloads.

I’m convinced the next iteration of the great firewall will not explicitly block content, but censor it. It will be harder to detect than just being denied access to a website. The ability to do large-scale processing like this is becoming more practical. Just remove the offending block of text or image. Citizens of oppressed nations will possibly not notice a thing.

There’s also been attempts to “optimize” images and video. Again even net-neutrality is not entirely clear assuming this isn’t targeted to competitors for example.

But TLS isn’t perfect!

True, but let’s be honest, it’s 8,675,309 times better than using nothing. CA’s are a vulnerability, they are a bottleneck, and a potential target for governments looking to control information. But browsers and OS’s allow you to manage certificates. The ability to stop trusting CA’s exists. Technology will improve over time. I don’t expect us to be still using TLS 1.1 and 1.2 in 2025. Hopefully substantial improvements get made over time. This argument is akin to not buying a computer because there will be a faster one next year. It’s the best option today, and we can replace it with better methods when available.

SSL Certificates are expensive!

First of all, domain validation certificates can be found for as little as $10. Secondly, I fully expect these prices to drop as demand increases. Domain verification certificates have virtually no cost as it’s all automated. The cheaper options will experience substantial growth as demand grows. There’s no limit in “supply” except computing power to generate them. A pricing war is inevitable. It would happen even faster if someone like Google bought a large CA and dropped prices to rock bottom. Certificates will get way cheaper before it’s essential. $10 is the early adopter fee.

But XYZ doesn’t support HTTPS!

True, not everyone is supporting it yet. That will change. It’s also true some (like CDN’s) are still charging insane prices for HTTPS. It’s not practical for everyone to switch today. Or this year. But that will change as well as demand increases. Encryption overhead is nominal. Once again pricing wars will happen once someone wants more than their shopping cart served over SSL. The problem today is demand is minimal, but those who need it must have it. Therefore price gouging is the norm.

Seriously, we need to do this?

Yes, seriously. HTTPS is the right direction for the Internet. There’s valid arguments for not switching your site over today, but those roadblocks will disappear and you should be re-evaluating where you stand periodically. I’ve moved a few sites including this blog (SPDY for now, HTTP/2 soon) to experience what would happen. It was largely a smooth transition. I’ve got some sites still on HTTP. Some will be on HTTP for the foreseeable future due to other circumstances, others will switch sooner. This doesn’t mean HTTP is dead tomorrow, or next year. It just means the future of the internet is HTTPS, and you should be part of it.

Comment Count

Kim MoirMozilla pushes - April 2015

Here's April 2015's  monthly analysis of the pushes to our Mozilla development trees. You can load the data as an HTML page or as a json file.  


Trends
The number of pushes decreased from those recorded in the previous month with a total of 8894.  This is due to the fact that gaia-try is managed by taskcluster and thus these jobs don't appear in the buildbot scheduling databases anymore which this report tracks.


Highlights

  • 8894 pushes
  • 296 pushes/day (average)
  • Highest number of pushes/day: 528 pushes on Apr 1, 2015
  • 17.87 pushes/hour (highest average)

General Remarks

  • Try has around 58% of all the pushes now that we no longer track gaia-try
  • The three integration repositories (fx-team, mozilla-inbound and b2g-inbound) account around 28% of all the pushes.

Records

  • August 2014 was the month with most pushes (13090  pushes)
  • August 2014 had the highest pushes/day average with 422 pushes/day
  • July 2014 had the highest average of "pushes-per-hour" with 23.51 pushes/hour
  • October 8, 2014 had the highest number of pushes in one day with 715 pushes  



Note
I've changed the graphs to only track 2015 data.  Last month they were tracking 2014 data as well but it looked crowded so I updated them.  Here's a graph showing the number of pushes over the last few years for comparison.



Daniel Stenbergtalking curl on the changelog

The changelog is the name of a weekly podcast on which the hosts discuss open source and stuff.

Last Friday I was invited to participate and I joined hosts Adam and Jerod for an hour long episode about curl. It all started as a response to my post on curl 17 years, so we really got into how things started out and how curl has developed through the years, how much time I’ve spent on it and if I could mention a really great moment in time that stood out over the years?

They day before, they released the little separate teaser we made about about the little known –remote-name-all command line option that basically makes curl default to do -O on all given URLs.

The full length episode can be experienced in all its glory here: https://changelog.com/153/

Brian BirtlesWhat do we do with SMIL?

Earlier this week, Blink announced their intention to deprecate SMIL. I thought they were going to replace their native implementation with a Javascript one so this was a surprise to me.

Prompted by this, the SVG WG decided it would be better to split the animation features in SVG2 out into a separate spec. (This was something I started doing a while ago, calling it Animation Elements, but I haven’t had time to follow up on it recently.)

I’ve spent quite a lot of time working on SMIL in Gecko (Firefox) so I’m probably more attached to it than most. I also started work on Web Animations specifically to address Microsoft’s concern that we needed a unified model for animations on the Web and I was under the impression they were finally open to the idea of a Javascript implementation of SMIL in Edge.

I’m not sure what will happen next, but it’s interesting to think about what we would lose without SMIL and what we could do to fix that. Back in 2011 I wrote up a gap analysis of features missing in CSS that exist in SVG animation. One example, is that even with CSS Animations, CSS Transitions, Web Animations and the Motion Path module, we still couldn’t create a font using SVG-in-OpenType where the outlines of the glyphs wiggle. That’s because even though Web Animations lets you animate attributes (and not just CSS properties), that feature is only available via the script API and you can’t run script in some contexts like font glyph documents.

So what would we need? I think some of the following might be interesting specs:

  • Path animation module – We need some means of animating path data such as the ‘d’ attribute on an SVG element. With SMIL this is actually really hard—you need to have exactly the same number and type of segments in order to interpolate between two paths. Tools could help with this but there aren’t any yet.

    It would be neat to be able to interpolate between, say, a and a . Once you allow different numbers of segments you probably need a means of annotating anchor points so you can describe how the different paths are supposed to line up.
    (If, while we’re at it, we could define a way of warping paths that would be great for doing cartoons!)

  • Animation group module – SMIL lets you sequence and group animations so they play perfectly together. That’s not easy with CSS at the moment. Web Animations level 2 actually defines grouping and synchronization primitives for this but there’s no proposed CSS syntax for it.

    I think it would be useful if CSS Animations Level 2 added a single level of grouping, something like an animation-group property where all animations with a matching group name were kept in lock-step (with animation-group-reset to create new groups of the same name). A subsequent level could extend that to the more advanced hierarchies of groups described in Web Animations level 2.

  • Property addition – SMIL lets you have independent animations target the same element and add together. For example, you can have a ‘spin’ animation and a ‘swell’ animation defined completely independently and then applied to the same element, and they combine together without conflict. Allowing CSS properties to add together sounds like a big change but you can actually narrow down the problem space in three ways:
    1. Most commonly you’re adding together lists: e.g. transform lists or filter lists. A solution that only lets you add lists together would probably solve a lot of use cases.
    2. Amongst lists, transform lists are the most common. For this the FXTF already resolved to add translate, rotate and scale properties in CSS transforms level 2 so we should be able to address some of those use cases in the near future.
    3. While it would be nice to add properties together in static contexts like below, if it simplifies the solution, we could just limit the issue to animations at first.

.blur {  filter: blur(10px); }
.sepia { filter: sepia(50%); }
<img class=”blur sepia”>

There are other things that SMIL lets you do such as change the source URL of an image in response to an arbitrary event like a click without writing any programming code but I think the above cover some of the bigger gaps? What else would we miss?


Mike HommeyUsing a git clone of gecko-dev to push to mercurial

The next branch of git-cinnabar now has minimal support for grafting, enough to allow to graft a clone of gecko-dev to mozilla-central and other Mozilla branches. This will be available in version 0.3 (which I don’t expect to release before June), but if you are interested, you can already try it.

There are two ways you can work with gecko-dev and git-cinnabar.

Switching to git-cinnabar

This is the recommended setup.

There are several reasons one would want to start from gecko-dev instead of a fresh clone of mozilla-central. One is to get the full history from before Mozilla switched to Mercurial, which gecko-dev contains. Another is if you already have a gecko-dev clone with local branches, where rebasing against a fresh clone would not be very convenient (but not impossible).

The idea here is to use gecko-dev as a start point, and from there on, use cinnabar to pull and push from/to Mercurial. The main caveat is that new commits pulled from Mercurial after this will not have the same SHA-1s as those on gecko-dev. Only the commits until the switch will. This also means different people grafting gecko-dev at different moments will have different SHA-1s for new commits. Eventually, I hope we can switch gecko-dev itself to use git-cinnabar instead of hg-git, which would solve this issue.

Assuming you already have a gecko-dev clone, and git-cinnabar next installed:

  • Change the remote URL:
    $ git remote set-url origin hg::https://hg.mozilla.org/mozilla-central

    (replace origin with the remote name for gecko-dev if it’s not origin)

  • Add other mercurial repositories you want to track as well:
    $ git remote add inbound hg::https://hg.mozilla.org/integration/mozilla-inbound
    $ git remote add aurora hg::https://hg.mozilla.org/releases/mozilla-aurora
    (...)
  • Pull with grafting enabled:
    $ git -c cinnabar.graft=true -c cinnabar.graft-refs=refs/remotes/origin/* remote update

    (replace origin with the remote name for gecko-dev if it’s not origin)

  • Finish the setup by setting push urls for all those remotes:
    $ git remote set-url --push origin hg::ssh://hg.mozilla.org/mozilla-central
    $ git remote set-url --push inbound hg::ssh://hg.mozilla.org/integration/mozilla-inbound
    $ git remote set-url --push aurora hg::ssh://hg.mozilla.org/releases/mozilla-aurora
    (...)

Now, you’re mostly done setting things up. Check out my git workflow for Gecko development for how to work from there.

Following gecko-dev

This setup allows to keep pulling from gecko-dev and thus keep the same SHA-1s. It relies on everything pulled from Mercurial existing in gecko-dev, which makes it cumbersome, which is why I don’t recommend using it. For instance, you may end up pulling from Mercurial before the server-side mirroring made things available in gecko-dev, and that will fail. Some commands such as git pull --rebase will require you to ensure gecko-dev is up-to-date first (that makes winning push races essentially impossible). And more importantly, in some cases, what you push to Mercurial won’t have the same commit SHA-1 in gecko-dev, so you’ll have to manually deal with that (fortunately, it most cases, this shouldn’t happen).

Assuming you already have a gecko-dev clone, and git-cinnabar next installed:

  • Add a remote for all mercurial repositories you want to track:
    $ git remote add central hg::https://hg.mozilla.org/mozilla-central
    $ git remote add inbound hg::https://hg.mozilla.org/integration/mozilla-inbound
    $ git remote add aurora hg::https://hg.mozilla.org/releases/mozilla-aurora
    (...)
  • For each of those remotes, set a push url:
    $ git remote set-url --push origin hg::ssh://hg.mozilla.org/mozilla-central
    $ git remote set-url --push inbound hg::ssh://hg.mozilla.org/integration/mozilla-inbound
    $ git remote set-url --push aurora hg::ssh://hg.mozilla.org/releases/mozilla-aurora
    (...)
  • For each of those remotes, set a refspec that limits to pulling the tip of the default branch:
    $ git remote.central.fetch +refs/heads/branches/default/tip:refs/remotes/central/default
    $ git remote.inbound.fetch +refs/heads/branches/default/tip:refs/remotes/inbound/default
    $ git remote.aurora.fetch +refs/heads/branches/default/tip:refs/remotes/aurora/default
    (...)

    Other branches can be added, but that must be done with care, because not all branches are exposed on gecko-dev.

  • Set git-cinnabar grafting mode permanently:
    $ git config cinnabar.graft only
  • Make git-cinnabar never store metadata when pushing:
    $ git config cinnabar.data never
  • Make git-cinnabar only graft with gecko-dev commits:
    $ git config cinnabar.graft-refs refs/remotes/origin/*

    (replace origin with the remote name for gecko-dev if it’s not origin)

  • Then pull from all remotes:
    $ git remote update

    Retry as long as you see errors about “Not allowing non-graft import”. This will keep happening until all Mercurial changesets you’re trying to pull make it to gecko-dev.

With this setup, you can now happily push to Mercurial and see your commits appear in gecko-dev after a while. As long as you don’t copy or move files, their SHA-1 should be the same as what you pushed.

Mike HommeyAnnouncing git-cinnabar 0.2.2

Git-cinnabar is a git remote helper to interact with mercurial repositories. It allows to clone, pull and push from/to mercurial remote repositories, using git.

Get it on github.

What’s new since 0.2.1?

  • Don’t require core.ignorecase to be set to false on the repository when using a case-insensitive file system. If you did set core.ignorecase to false because git-cinnabar told you to, you can now set it back to true.
  • Raise an exception when git update-ref or git fast-import return an error. Silently ignoring those errors could lead to bad repositories after an upgrade from pre-0.1.0 versions on OS X, where the default maximum number of open files is low (256), and where git update-ref uses a lot of lock files for large transactions.
  • Updated git to 2.4.0, when building with the native helper.
  • When doing git cinnabar reclone, skip remotes with remote.$remote.skipDefaultUpdate set to true.

Niko MatsakisA few more remarks on reference-counting and leaks

So there has been a lot of really interesting discussion in response to my blog post. I wanted to highlight some of the comments I’ve seen, because I think they raise good points that I failed to address in the blog post itself. My comments here are lightly edited versions of what I wrote elsewhere.

Isn’t the problem with objects and leak-safe types more general?

Reem writes:

I posit that this is in fact a problem with trait objects, not a problem with Leak; the exact same flaw pointed about in the blog post already applies to the existing OIBITs, Send, Sync, and Reflect. The decision of which OIBITs to include on any trait object is already a difficult one, and is a large reason why std strives to avoid trait objects as part of public types.

I agree with him that the problems I described around Leak and objects apply equally to Send (and, in fact, I said so in my post), but I don’t think this is something we’ll be able to solve later on, as he suggests. I think we are working with something of a fundamental tension. Specifically, objects are all about encapsulation. That is, they completely hide the type you are working with, even from the compiler. This is what makes them useful: without them, Rust just plain wouldn’t work, since you couldn’t (e.g.) have a vector of closures. But, in order to gain that flexibility, you have to state your requirements up front. The compiler can’t figure them out automatically, because it doesn’t (and shouldn’t) know the types involved.

So, given that objects are here to stay, the question is whether adding a marker trait like Leak is a problem, given that we already have Send. I think the answer is yes; basically, because we can’t expect object types to be analyzed statically, we should do our best to minimize the number of fundamental splits people have to work with. Thread safety is pretty fundamental. I don’t think Leak makes the cut. (I said some of the reasons in conclusion of my previous blog post, but I have a few more in the questions below.)

Could we just remove Rc and only have RcScoped? Would that solve the problem?

Original question.

Certainly you could remove Rc in favor of RcScoped. Similarly, you could have only Arc and not Rc. But you don’t want to because you are basically failing to take advantage of extra constraints. If we only had RcScoped, for example, then creating an Rc always requires taking some scoped as argument – you can have a global constant for 'static data, but it’s still the case that generic abstractions have to take in this scope as argument. Moreover, there is a runtime cost to maintaining the extra linked list that will thread through all Rc abstractions (and the Rc structs get bigger, as well). So, yes, this avoids the “split” I talked about, but it does it by pushing the worst case on all users.

Still, I admit to feeling torn on this point. What pushes me over the edge, I think, is that simple reference counting of the kind we are doing now is a pretty fundamental thing. You find it in all kinds of systems (Objective C, COM, etc). This means that if we require that safe Rust cannot leak, then you cannot safely integrate borrowed data with those systems. I think it’s better to just use closures in Rust code – particularly since, as annodomini points out on Reddit, there are other kinds of cases where RAII is a poor fit for cleanup.

Could a proper GC solve this? Is reference counting really worth it?

Original question.

It’ll depend on the precise design, but tracing GC most definitely is not a magic bullet. If anything, the problem around leaks is somewhat worse: GC’s don’t give any kind of guarantee about when the destructor bans. So we either have to ban GC’d data from having destructors or ban it from having borrowed pointers; either of those implies a bound very similar to Leak or 'static. Hence I think that GC will never be a “fundamental building block” for abstractions in the way that Rc/Arc can be. This is sad, but perhaps inevitable: GC inherently requires a runtime as well, which already limits its reusability.

The Servo BlogServo Continues Pushing Forward

Servo is a new prototype web browser layout engine written in Rust that was launched by Mozilla in 2012 with a new architecture to achieve high parallelism on components like layout and painting.

It has been progressing at an amazing pace, with over 120 CSS properties currently supported, and work is ongoing to implement the remaining properties. For a full list of the current set of CSS properties with initial support in Servo, check out the Google Docs spreadsheet servo team is using to track development.

The current supported properties allow Servo to be mostly operational on static sites like Wikipedia and GitHub, with a surprisingly small code footprint. It has only about 126K lines of Rust code, and the Rust compiler and libraries are about 360K lines. For comparison, in 2014 Blink had about 700K lines of C++ code, and WebKit had around 1.3M lines, including platform specific code.

Another exciting development is servo-shell, which allows the implementation and customization of a WebBrowser using only Javascript and CSS. It’s essentially a browser chrome that uses mozbrowser APIs (i.e. iFrame extensions) running on top of Servo and provides the ability to separate the browser content from loaded pages, which has led to fairly good performance so far.

Finally, Rust (the programming language used to implement Servo) is approaching the 1.0 launch and a big group of people are ready to celebrate the occasion in San Francisco.

Improving Debugging and Testing

One of the most challenging parts of developing a browser engine from scratch is re-implementing all of the CSS features, because they often have complicated interactions. For a developer to solve any layout rendering bugs they run into, they must first inspect the graphical representation of the DOM tree to see if it is correct. In case of Servo, the DOM tree will generate a FlowTree and DisplayLists while performing layout and rendering, compared to WebKit and Blink, which uses a RenderTree as graphical representation (and features DumpRenderTree tool for accessing the RenderTree). Debugging support was improved remarkably with addition of the ability to dump Optimized display lists, Flow tree, and Display List, as well as the implementation of reflow events debugging, which can be used to inform developers when and why a layout was recalculated.

Integration of the Firefox timeline has recently been started on Servo. This is a tool that allows tracking of operations performed by the web engine and is useful for debugging and profiling a site. Additionally, W3C organization has created a test suite to help in verifying CSS features across browsers, which enhances interoperability. Servo now has support for running these W3C CSS tests.

Additional Servo Highlights

General Developments

  • Servo was ported to Gonk (the low level layer of Firefox OS) last February.
  • Servo has some state of the art components (e.g. HTML5 parser, CSS parser) implemented in Rust as independent libraries, which may be beneficial to integrate with Firefox. Work has started on this integration, but whether the image decoder or the URL parser will be integrated first is undefined at this time.
  • WebGL implementation has begun.
  • Another cool feature is the visualization of parallel painting, where Servo executes in a mode in which tiles rendered by each distinct thread will have an overlay rendered on top of it. This makes it possible to visualize Servo’s parallel painting.
  • Support for displaying a placeholder when an image link is broken.
  • Cookies are now supported as well as SSL certification verification. These allow users to login to most websites that have user accounts.
  • Providing the ability to embed Servo on applications in the future is important, and work on this subject is progressing. Instead of creating a new API for developers, the community decided to use the Chromium Embedded Framework (CEF): an API that is quite successful and stable. Servo has a CEF-like API that provides the ability to embed a Servo-powered webview on native apps, as demonstrated by Miniservo on Mac and Linux. Work on supporting the API has been progressing well.

Improved HTML/CSS Support

The Road Ahead

As you can see, Servo has advanced remarkably in the last few months by implementing many new features that benefit both Servo developers as well as future users. It is moving at a fast pace, implementing support for several of the web features needed by any modern browser engine while proving that Rust, as a systems-level programing language, is up to the task of writing a web engine from scratch.

Are you interested in contributing to Servo or just curious to give it a try? Visit the project site, or feel free to chat with the developers on #servo on the mozilla IRC server.

This post was originally published on the Samsung OSG Blog, which is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Andrew SutherlandTalk Script: Firefox OS Email Performance Strategies

Last week I gave a talk at the Philly Tech Week 2015 Dev Day organized by the delightful people at technical.ly on some of the tricks/strategies we use in the Firefox OS Gaia Email app.  Note that the credit for implementing most of these techniques goes to the owner of the Email app’s front-end, James Burke.  Also, a special shout-out to Vivien for the initial DOM Worker patches for the email app.

I tried to avoid having slides that both I would be reading aloud as the audience read silently, so instead of slides to share, I have the talk script.  Well, I also have the slides here, but there’s not much to them.  The headings below are the content of the slides, except for the one time I inline some code.  Note that the live presentation must have differed slightly, because I’m sure I’m much more witty and clever in person than this script would make it seem…

Cover Slide: Who!

Hi, my name is Andrew Sutherland.  I work at Mozilla on the Firefox OS Email Application.  I’m here to share some strategies we used to make our HTML5 app Seem faster and sometimes actually Be faster.

What’s A Firefox OS (Screenshot Slide)

But first: What is a Firefox OS?  It’s a multiprocess Firefox gecko engine on an android linux kernel where all the apps including the system UI are implemented using HTML5, CSS, and JavaScript.  All the apps use some combination of standard web APIs and APIs that we hope to standardize in some form.

Firefox OS homescreen screenshot Firefox OS clock app screenshot Firefox OS email app screenshot

Here are some screenshots.  We’ve got the default home screen app, the clock app, and of course, the email app.

It’s an entirely client-side offline email application, supporting IMAP4, POP3, and ActiveSync.  The goal, like all Firefox OS apps shipped with the phone, is to give native apps on other platforms a run for their money.

And that begins with starting up fast.

Fast Startup: The Problems

But that’s frequently easier said than done.  Slow-loading websites are still very much a thing.

The good news for the email application is that a slow network isn’t one of its problems.  It’s pre-loaded on the phone.  And even if it wasn’t, because of the security implications of the TCP Web API and the difficulty of explaining this risk to users in a way they won’t just click through, any TCP-using app needs to be a cryptographically signed zip file approved by a marketplace.  So we do load directly from flash.

However, it’s not like flash on cellphones is equivalent to an infinitely fast, zero-latency network connection.  And even if it was, in a naive app you’d still try and load all of your HTML, CSS, and JavaScript at the same time because the HTML file would reference them all.  And that adds up.

It adds up in the form of event loop activity and competition with other threads and processes.  With the exception of Promises which get their own micro-task queue fast-lane, the web execution model is the same as all other UI event loops; events get scheduled and then executed in the same order they are scheduled.  Loading data from an asynchronous API like IndexedDB means that your read result gets in line behind everything else that’s scheduled.  And in the case of the bulk of shipped Firefox OS devices, we only have a single processor core so the thread and process contention do come into play.

So we try not to be a naive.

Seeming Fast at Startup: The HTML Cache

If we’re going to optimize startup, it’s good to start with what the user sees.  Once an account exists for the email app, at startup we display the default account’s inbox folder.

What is the least amount of work that we can do to show that?  Cache a screenshot of the Inbox.  The problem with that, of course, is that a static screenshot is indistinguishable from an unresponsive application.

So we did the next best thing, (which is) we cache the actual HTML we display.  At startup we load a minimal HTML file, our concatenated CSS, and just enough Javascript to figure out if we should use the HTML cache and then actually use it if appropriate.  It’s not always appropriate, like if our application is being triggered to display a compose UI or from a new mail notification that wants to show a specific message or a different folder.  But this is a decision we can make synchronously so it doesn’t slow us down.

Local Storage: Okay in small doses

We implement this by storing the HTML in localStorage.

Important Disclaimer!  LocalStorage is a bad API.  It’s a bad API because it’s synchronous.  You can read any value stored in it at any time, without waiting for a callback.  Which means if the data is not in memory the browser needs to block its event loop or spin a nested event loop until the data has been read from disk.  Browsers avoid this now by trying to preload the Entire contents of local storage for your origin into memory as soon as they know your page is being loaded.  And then they keep that information, ALL of it, in memory until your page is gone.

So if you store a megabyte of data in local storage, that’s a megabyte of data that needs to be loaded in its entirety before you can use any of it, and that hangs around in scarce phone memory.

To really make the point: do not use local storage, at least not directly.  Use a library like localForage that will use IndexedDB when available, and then fails over to WebSQLDatabase and local storage in that order.

Now, having sufficiently warned you of the terrible evils of local storage, I can say with a sorta-clear conscience… there are upsides in this very specific case.

The synchronous nature of the API means that once we get our turn in the event loop we can act immediately.  There’s no waiting around for an IndexedDB read result to gets its turn on the event loop.

This matters because although the concept of loading is simple from a User Experience perspective, there’s no standard to back it up right now.  Firefox OS’s UX desires are very straightforward.  When you tap on an app, we zoom it in.  Until the app is loaded we display the app’s icon in the center of the screen.  Unfortunately the standards are still assuming that the content is right there in the HTML.  This works well for document-based web pages or server-powered web apps where the contents of the page are baked in.  They work less well for client-only web apps where the content lives in a database and has to be dynamically retrieved.

The two events that exist are:

DOMContentLoaded” fires when the document has been fully parsed and all scripts not tagged as “async” have run.  If there were stylesheets referenced prior to the script tags, the script tags will wait for the stylesheet loads.

load” fires when the document has been fully loaded; stylesheets, images, everything.

But none of these have anything to do with the content in the page saying it’s actually done.  This matters because these standards also say nothing about IndexedDB reads or the like.  We tried to create a standards consensus around this, but it’s not there yet.  So Firefox OS just uses the “load” event to decide an app or page has finished loading and it can stop showing your app icon.  This largely avoids the dreaded “flash of unstyled content” problem, but it also means that your webpage or app needs to deal with this period of time by displaying a loading UI or just accepting a potentially awkward transient UI state.

(Trivial HTML slide)

<link rel=”stylesheet” ...>
<script ...></script>
DOMContentLoaded!

This is the important summary of our index.html.

We reference our stylesheet first.  It includes all of our styles.  We never dynamically load stylesheets because that compels a style recalculation for all nodes and potentially a reflow.  We would have to have an awful lot of style declarations before considering that.

Then we have our single script file.  Because the stylesheet precedes the script, our script will not execute until the stylesheet has been loaded.  Then our script runs and we synchronously insert our HTML from local storage.  Then DOMContentLoaded can fire.  At this point the layout engine has enough information to perform a style recalculation and determine what CSS-referenced image resources need to be loaded for buttons and icons, then those load, and then we’re good to be displayed as the “load” event can fire.

After that, we’re displaying an interactive-ish HTML document.  You can scroll, you can press on buttons and the :active state will apply.  So things seem real.

Being Fast: Lazy Loading and Optimized Layers

But now we need to try and get some logic in place as quickly as possible that will actually cash the checks that real-looking HTML UI is writing.  And the key to that is only loading what you need when you need it, and trying to get it to load as quickly as possible.

There are many module loading and build optimizing tools out there, and most frameworks have a preferred or required way of handling this.  We used the RequireJS family of Asynchronous Module Definition loaders, specifically the alameda loader and the r-dot-js optimizer.

One of the niceties of the loader plugin model is that we are able to express resource dependencies as well as code dependencies.

RequireJS Loader Plugins

var fooModule = require('./foo');
var htmlString = require('text!./foo.html');
var localizedDomNode = require('tmpl!./foo.html');

The standard Common JS loader semantics used by node.js and io.js are the first one you see here.  Load the module, return its exports.

But RequireJS loader plugins also allow us to do things like the second line where the exclamation point indicates that the load should occur using a loader plugin, which is itself a module that conforms to the loader plugin contract.  In this case it’s saying load the file foo.html as raw text and return it as a string.

But, wait, there’s more!  loader plugins can do more than that.  The third example uses a loader that loads the HTML file using the ‘text’ plugin under the hood, creates an HTML document fragment, and pre-localizes it using our localization library.  And this works un-optimized in a browser, no compilation step needed, but it can also be optimized.

So when our optimizer runs, it bundles up the core modules we use, plus, the modules for our “message list” card that displays the inbox.  And the message list card loads its HTML snippets using the template loader plugin.  The r-dot-js optimizer then locates these dependencies and the loader plugins also have optimizer logic that results in the HTML strings being inlined in the resulting optimized file.  So there’s just one single javascript file to load with no extra HTML file dependencies or other loads.

We then also run the optimizer against our other important cards like the “compose” card and the “message reader” card.  We don’t do this for all cards because it can be hard to carve up the module dependency graph for optimization without starting to run into cases of overlap where many optimized files redundantly include files loaded by other optimized files.

Plus, we have another trick up our sleeve:

Seeming Fast: Preloading

Preloading.  Our cards optionally know the other cards they can load.  So once we display a card, we can kick off a preload of the cards that might potentially be displayed.  For example, the message list card can trigger the compose card and the message reader card, so we can trigger a preload of both of those.

But we don’t go overboard with preloading in the frontend because we still haven’t actually loaded the back-end that actually does all the emaily email stuff.  The back-end is also chopped up into optimized layers along account type lines and online/offline needs, but the main optimized JS file still weighs in at something like 17 thousand lines of code with newlines retained.

So once our UI logic is loaded, it’s time to kick-off loading the back-end.  And in order to avoid impacting the responsiveness of the UI both while it loads and when we’re doing steady-state processing, we run it in a DOM Worker.

Being Responsive: Workers and SharedWorkers

DOM Workers are background JS threads that lack access to the page’s DOM, communicating with their owning page via message passing with postMessage.  Normal workers are owned by a single page.  SharedWorkers can be accessed via multiple pages from the same document origin.

By doing this, we stay out of the way of the main thread.  This is getting less important as browser engines support Asynchronous Panning & Zooming or “APZ” with hardware-accelerated composition, tile-based rendering, and all that good stuff.  (Some might even call it magic.)

When Firefox OS started, we didn’t have APZ, so any main-thread logic had the serious potential to result in janky scrolling and the impossibility of rendering at 60 frames per second.  It’s a lot easier to get 60 frames-per-second now, but even asynchronous pan and zoom potentially has to wait on dispatching an event to the main thread to figure out if the user’s tap is going to be consumed by app logic and preventDefault called on it.  APZ does this because it needs to know whether it should start scrolling or not.

And speaking of 60 frames-per-second…

Being Fast: Virtual List Widgets

…the heart of a mail application is the message list.  The expected UX is to be able to fling your way through the entire list of what the email app knows about and see the messages there, just like you would on a native app.

This is admittedly one of the areas where native apps have it easier.  There are usually list widgets that explicitly have a contract that says they request data on an as-needed basis.  They potentially even include data bindings so you can just point them at a data-store.

But HTML doesn’t yet have a concept of instantiate-on-demand for the DOM, although it’s being discussed by Firefox layout engine developers.  For app purposes, the DOM is a scene graph.  An extremely capable scene graph that can handle huge documents, but there are footguns and it’s arguably better to err on the side of fewer DOM nodes.

So what the email app does is we create a scroll-region div and explicitly size it based on the number of messages in the mail folder we’re displaying.  We create and render enough message summary nodes to cover the current screen, 3 screens worth of messages in the direction we’re scrolling, and then we also retain up to 3 screens worth in the direction we scrolled from.  We also pre-fetch 2 more screens worth of messages from the database.  These constants were arrived at experimentally on prototype devices.

We listen to “scroll” events and issue database requests and move DOM nodes around and update them as the user scrolls.  For any potentially jarring or expensive transitions such as coordinate space changes from new messages being added above the current scroll position, we wait for scrolling to stop.

Nodes are absolutely positioned within the scroll area using their ‘top’ style but translation transforms also work.  We remove nodes from the DOM, then update their position and their state before re-appending them.  We do this because the browser APZ logic tries to be clever and figure out how to create an efficient series of layers so that it can pre-paint as much of the DOM as possible in graphic buffers, AKA layers, that can be efficiently composited by the GPU.  Its goal is that when the user is scrolling, or something is being animated, that it can just move the layers around the screen or adjust their opacity or other transforms without having to ask the layout engine to re-render portions of the DOM.

When our message elements are added to the DOM with an already-initialized absolute position, the APZ logic lumps them together as something it can paint in a single layer along with the other elements in the scrolling region.  But if we start moving them around while they’re still in the DOM, the layerization logic decides that they might want to independently move around more in the future and so each message item ends up in its own layer.  This slows things down.  But by removing them and re-adding them it sees them as new with static positions and decides that it can lump them all together in a single layer.  Really, we could just create new DOM nodes, but we produce slightly less garbage this way and in the event there’s a bug, it’s nicer to mess up with 30 DOM nodes displayed incorrectly rather than 3 million.

But as neat as the layerization stuff is to know about on its own, I really mention it to underscore 2 suggestions:

1, Use a library when possible.  Getting on and staying on APZ fast-paths is not trivial, especially across browser engines.  So it’s a very good idea to use a library rather than rolling your own.

2, Use developer tools.  APZ is tricky to reason about and even the developers who write the Async pan & zoom logic can be surprised by what happens in complex real-world situations.  And there ARE developer tools available that help you avoid needing to reason about this.  Firefox OS has easy on-device developer tools that can help diagnose what’s going on or at least help tell you whether you’re making things faster or slower:

– it’s got a frames-per-second overlay; you do need to scroll like mad to get the system to want to render 60 frames-per-second, but it makes it clear what the net result is

– it has paint flashing that overlays random colors every time it paints the DOM into a layer.  If the screen is flashing like a discotheque or has a lot of smeared rainbows, you know something’s wrong because the APZ logic is not able to to just reuse its layers.

– devtools can enable drawing cool colored borders around the layers APZ has created so you can see if layerization is doing something crazy

There’s also fancier and more complicated tools in Firefox and other browsers like Google Chrome to let you see what got painted, what the layer tree looks like, et cetera.

And that’s my spiel.

Links

The source code to Gaia can be found at https://github.com/mozilla-b2g/gaia

The email app in particular can be found at https://github.com/mozilla-b2g/gaia/tree/master/apps/email

(I also asked for questions here.)

Doug BelshawWeb Literacy Map v1.5 is now live at teach.mozilla.org

Mozilla has soft-launched teach.mozilla.org. This provides a new home for the Web Literacy Map, which now stands at v1.5.

Web Literacy Map v1.5

While I’m a bit sad at the lack of colour compared to the previous version, at least it’s live and underpinning the ‘Teach Like Mozilla’ work!


Questions? Comments? I’m @dajbelshaw or you can email me: mail@dougbelshaw.com

Gregory SzorcAutomatically Redirecting Mercurial Pushes

Managing URLs in distributed version control tools can be a pain, especially if multiple repositories are involved. For example, with Mozilla's repository-based code review workflow (you push to a special review repository to initiate code review - this is conceptually similar to GitHub pull requests), there exist separate code review repositories for each logical repository. Figuring out how repositories map to each other and setting up remote paths for each new clone can be a pain and time sink.

As of today, we can now do something better.

If you push to ssh://reviewboard-hg.mozilla.org/autoreview, Mercurial will automatically figure out the appropriate review repository and redirect your push automatically. In other words, if we have MozReview set up to review whatever repository you are working on, your push and review request will automatically go through. No need to figure out what the appropriate review repo is or configure repository URLs!

Here's what it looks like:

$ hg push review
pushing to ssh://reviewboard-hg.mozilla.org/autoreview
searching for appropriate review repository
redirecting push to ssh://reviewboard-hg.mozilla.org/version-control-tools/
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 1 changes to 1 files
remote: Trying to insert into pushlog.
remote: Inserted into the pushlog db successfully.
submitting 1 changesets for review

changeset:  11043:b65b087a81be
summary:    mozreview: create per-commit identifiers (bug 1160266)
review:     https://reviewboard.mozilla.org/r/7953 (draft)

review id:  bz://1160266/gps
review url: https://reviewboard.mozilla.org/r/7951 (draft)
(visit review url to publish this review request so others can see it)

Read the full instructions for more details.

This requires an updated version-control-tools repository, which you can get by running mach mercurial-setup from a Firefox repository.

For those that are curious, the autoreview repo/server advertises a list of repository URLs and their root commit SHA-1. The client automatically sends the push to a URL sharing the same root commit. The code is quite simple.

While this is only implemented for MozReview, I could envision us doing something similar for other centralized repository-centric services, such as Try and Autoland. Stay tuned.

Mozilla Release Management TeamFirefox 38 beta8 to beta9

This beta is not busy as it might seem. 17 patches were test-only or NPOTB (Not Part Of The Build) changes.

  • 38 changesets
  • 87 files changed
  • 713 insertions
  • 287 deletions

ExtensionOccurrences
h22
cpp14
js12
cc6
html4
txt2
py2
jsm2
ini2
xml1
sh1
list1
java1
css1
c1
build1

ModuleOccurrences
ipc13
dom11
js10
browser8
media5
image5
gfx5
mobile3
layout3
toolkit2
mfbt2
xpcom1
uriloader1
tools1
testing1
services1
python1
parser1
netwerk1
mozglue1
docshell1
config1

List of changesets:

Ryan VanderMeulenBug 1062496 - Disable browser_aboutHome.js on OSX 10.6 debug. a=test-only - 657cfe2d4078
Ryan VanderMeulenBug 1148224 - Skip timeout-prone subtests in mediasource-duration.html on Windows. a=test-only - 82de02ddde1b
Ehsan AkhgariBug 1095517 - Increase the timeout of browser_identity_UI.js. a=test-only - 611ca5bd91d4
Ehsan AkhgariBug 1079617 - Increase the timeout of browser_test_new_window_from_content.js. a=test-only - 1783df5849c7
Eric RahmBug 1140537 - Sanity check size calculations. r=peterv, a=abillings - a7d6b32a504c
Hiroyuki IkezoeBug 1157985 - Use getEntriesByName to search by name attribute. r=qdot, a=test-only - 55b58d5184ce
Morris TsengBug 1120592 - Create iframe directly instead of using setTimeout. r=kanru, a=test-only - a4f506639153
Gregory SzorcBug 1128586 - Properly look for Mercurial version. r=RyanVM, a=NPOTB - 49abfe1a8ef8
Gregory SzorcBug 1128586 - Prefer hg.exe over hg. r=RyanVM, a=NPOTB - a0b48af4bb54
Shane TomlinsonBug 1146724 - Use a SendingContext for WebChannels. r=MattN, r=markh, a=abillings - 56d740d0769f
Brian HackettBug 1138740 - Notify Ion when changing a typed array's data pointer due to making a lazy buffer for it. r=sfink, a=sledru - e1fb2a5ab48d
Seth FowlerBug 1151309 - Part 1: Block until the previous multipart frame is decoded before processing another. r=tn, a=sledru - 046c97d2eb23
Seth FowlerBug 1151309 - Part 2: Hide errors in multipart image parts both visually and internally. r=tn, a=sledru - 0fcbbecc843d
Alessio PlacitelliBug 1154518 - Make sure extended data gathering (Telemetry) is disabled when FHR is disabled. r=Gijs, a=sledru - cb2725c612b2
Bas SchoutenBug 1151821 - Make globalCompositeOperator work correctly when a complex clip is pushed. r=jrmuizel, a=sledru - 987c18b686eb
Bas SchoutenBug 1151821 - Test whether simple canvas globalCompositeOperators work when a clip is set. r=jrmuizel, a=sledru - 1bbb50c6a494
Bob OwenBug 1087565 - Verify the child process with a secret hello on Windows. r=dvander, a=sledru - c1f04200ed98
Randell JesupBug 1157766 - Mismatched DataChannel initial channel size in JSEP database breaks adding channels. r=bwc, a=sledru - a8fb9422ff13
David MajorBug 1130061 - Block version 1.5 of vwcsource.ax. r=bsmedberg, a=sledru - 053da808c6d9
Martin ThomsonBug 1158343 - Temporarily enable TLS_RSA_WITH_AES_128_CBC_SHA for WebRTC. r=ekr, a=sledru - d10817faa571
Margaret LeibovicBug 1155083 - Properly hide reader view tablet on landscape tablets. r=bnicholson, a=sledru - f7170ad49667
Steve FinkBug 1136309 - Rename the spidermonkey build variants. r=terrence, a=test-only - 604326355be0
Mike HommeyBug 1142908 - Avoid arm simulator builds being considered cross-compiled. r=sfink, a=test-only - 517741a918b0
Jan de MooijBug 1146520 - Fix some minor autospider issues on OS X. r=sfink, a=test-only - 620cae899342
Steve FinkBug 1146520 - Do not treat osx arm-sim as a cross-compile. a=test-only - a5013ed3d1f0
Steve FinkBug 1135399 - Timeout shell builds. r=catlee, a=test-only - b6bf89c748b7
Steve FinkBug 1150347 - Fix autospider.sh --dep flag name. r=philor, a=test-only - b8f7eabd31b9
Steve FinkBug 1149476 - Lengthen timeout because we are hitting it with SM(cgc). r=me (also jonco for a more complex version), a=test-only - 16c98999de0b
Chris PearceBug 1136360 - Backout 3920b67e97a3 to fix A/V sync regressions (Bug 1148299 & Bug 1157886). r=backout a=sledru - 4ea8cdc621e8
Patrick BrossetBug 1153463 - Intermittent browser_animation_setting_currentTime_works_and_pauses.js. r=miker, a=test-only - c31c2a198a71
Andrew McCreightBug 1062479 - Use static strings for WeakReference type names. r=ehsan, a=sledru - 5d903629f9bd
Michael ComellaBug 1152314 - Duplicate action bar configuration in code. r=liuche, a=sledru - cdfd06d73d17
Ethan HuggBug 1158627 - WebRTC return error if GetEmptyFrame returns null. r=jesup, a=sledru - f1cd36f7e0e1
Jeff MuizelaarBug 1154703 - Avoid using WARP if nvdxgiwrapper.dll is around. a=sledru - 348c2ae68d50
Shu-yu GuoBug 1155474 - Consider the input to MThrowUninitializedLexical implicitly used. r=Waldo, a=sledru - daaa2c27b89f
Jean-Yves AvenardBug 1149605 - Avoid potential integers overflow. r=kentuckyfriedtakahe, a=abillings - fcfec0caa7be
Ryan VanderMeulenBacked out changeset daaa2c27b89f (Bug 1155474) for bustage. - 0a1accb16d39
Shu-yu GuoBug 1155474 - Consider the input to MThrowUninitializedLexical implicitly used. r=Waldo, a=sledru - ff65ba4cd38a

Nick Cameronrustfmt - call for contributions

I've been experimenting with a rustfmt tool for a while now. Its finally in working shape (though still very, very rough) and I'd love some help on making it awesome.

rustfmt is a reformatting tool for Rust code. The idea is that it takes your code, tidies it up, and makes sure it conforms to a set of style guidelines. There are similar tools for C++ (clang format), Go (gofmt), and many other languages. Its a really useful tool to have for a language, since it makes it easy to adhere to style guidelines and allows for mass changes when guidelines change, thus making it possible to actually change the guidelines as needed.

Eventually I would like rustfmt to do lots of cool stuff like changing glob imports to list imports, or emit refactoring scripts to rename variables to adhere to naming conventions. In the meantime, there are lots of interesting questions about how to lay out things like function declarations and match expressions.

My approach to rustfmt is very incremental. It is usable now and gives good results, but it only touches a tiny subset of language items, for example function definitions and calls, and string literals. It preserves code elsewhere. This makes it immediately useful.

I have managed to run it on several crates (or parts of crates) in the rust distro. It also bootstraps, i.e., you can rustfmt on rustfmt before every check-in, in fact this is part of the test suite.

It would be really useful to have people running this tool on their own code or on other crates in the rust distro, and filing issues and/or test cases where things go wrong. This should actually be a useful tool to run, not just a chore, and will get more useful with time.

It's a great project to hack on - you'll learn a fair bit about the Rust compiler's frontend and get a great understanding of more corners of the language than you'll ever want to know about. It's early days too, so there is plenty of scope for having a big impact on the project. I find it a lot of fun too! Just please forgive some of the hackey code that I've already written.

Here is the rustfmt repo on GitHub. I just added a bunch of information to the repo readme which should help new contributors. Please let me know if there is other information that should go in there. I've also created some good issues for new contributors. If you'd like to help out and need help, please ping me on irc (I'm nrc).

Anthony HughesThe Testday Brand

Over the last few months I’ve been surveying people who’ve participated in testdays. The purpose of this effort is to develop an understanding of the current “brand” that testdays present. I’ve come to realize that our goal to “re-invigorate” the testdays program was based on assumptions that testdays were both well-known and misunderstood. I wanted to cast aside that assumption and make gains on a new plan which includes developing a positive brand.

The survey itself was quite successful as I received over 200 responses, 10x what I normally get out of my surveys. I suspect this was because I kept it short, under a minute to complete; something I will keep in mind for the future.

Who Shared the Most?

testday-responses

When looking out who responded most, the majority were unaffiliated with QA (53%). Of the 47% who were affiliated with QA nearly two thirds were volunteers.

How do these see themselves?

testday-mozillians

When looking at how respondents self-identified, only people who identified as volunteers did not self-identify as a Mozillian. In terms of vouching, people affiliated with QA seem to have a higher proportion of vouched Mozillians than those unaffiliated with QA. This tells me that we need to be doing a better job of converting new contributors into Mozillians and active into vouched Mozillians.

What do they know about Testdays?

testday-familiarity

When looking at how familiar people are with the concept of testdays, people affiliated with QA are most aware while people outside of QA are least aware. No group of people are 100% familiar with testdays which tells me we need to do a better job of educating people about testdays.

What do they think about Testdays?

testday-keywords

Most respondents identified testdays with some sort of activity (30%), a negative feeling (22%), a community aspect (15%), or a specific product (15%). Positive characteristics were lowest on the list (4%). This was probably the most telling question I asked as it really helps me see the current state of the brand of testdays and not just for the responses I received. Reading between the lines, looking for what is not there, I can see testdays relate poorly to anything outside the scope of blackbox testing on Firefox (eg. automation, services, web qa, security qa, etc).

Where do I go from here?

1. We need to diversify the testday brand to be more about testing Firefox and expand it to enable testing across all areas in need.

2. We need to solve some of the negative brand associations by making activities more understandable and relevant, by having shorter events more frequently, , and by rewarding contributions (even those who do work that doesn’t net a bug).

3. We need to teach people that testdays are about more than just testing. Things like writing tests, writing new documentation, updating and translating existing documentation, and mentoring newcomers is all part of what testdays can enable.

4. Once we’ve identified the brand we want to put forward, we need to do a much better job of frequently educating and re-educating people about testdays and the value they provide.

5. We need to enable testdays to facilitate converting newcomers into Mozillians and active contributors into vouched Mozillians.

My immediate next step is to have the lessons I’ve learned here integrated into a plan of action to rebrand testdays. Rest assured I am going to continue to push my peers on this, to be an advocate for improving the ways we collaborate, and to continually revisit the brand to make sure we aren’t losing sight of reality.

I’d like to end with a thank you to everyone who took the time to respond to my survey. As always, please leave a comment below if you have any interesting insights or questions.

Thank you!

Air MozillaQuality Team (QA) Public Meeting

Quality Team (QA) Public Meeting This is the meeting where all the Mozilla quality teams meet, swap ideas, exchange notes on what is upcoming, and strategize around community building and...

Niko MatsakisOn reference-counting and leaks

What’s a 1.0 release without a little drama? Recently, we discovered that there was an oversight in one of the standard library APIs that we had intended to stabilize. In particular, we recently added an API for scoped threads – that is, child threads which have access to the stack frame of their parent thread.

The flaw came about because, when designing the scoped threads API, we failed to consider the impact of resource leaks. Rust’s ownership model makes it somewhat hard to leak data, but not impossible. In particular, using reference-counted data, you can construct a cycle in the heap, in which case the components of that cycle may never be freed.

Some commenters online have taken this problem with the scoped threads API to mean that Rust’s type system was fundamentally flawed. This is not the case: Rust’s guarantee that safe code is memory safe is as true as it ever was. The problem was really specific to the scoped threads API, which was making flawed assumptions; this API has been marked unstable, and there is an RFC proposing a safe alternative.

That said, there is an interesting, more fundamental question at play here. We long ago decided that, to make reference-counting practical, we had to accept resource leaks as a possibility. But some recent proposals have suggested that we should place limits on the Rc type to avoid some kinds of reference leaks. These limits would make the original scoped threads API safe. However, these changes come at a pretty steep price in composability: they effectively force a deep distinction between “leakable” and “non-leakable” data, which winds up affecting all levels of the system.

This post is my attempt to digest the situation and lay out my current thinking. For those of you don’t want to read this entire post (and I can’t blame you, it’s long), let me just copy the most salient paragraph from my conclusion:

This is certainly a subtle issue, and one where reasonable folk can disagree. In the process of drafting (and redrafting…) this post, my own opinion has shifted back and forth as well. But ultimately I have landed where I started: the danger and pain of bifurcating the space of types far outweighs the loss of this particular RAII idiom.

All right, for those of you who want to continue, this post is divided into three sections:

  1. Section 1 explains the problem and gives some historical background.
  2. Section 2 explains the “status quo”.
  3. Section 3 covers the proposed changes to the reference-counted type and discusses the tradeoffs involved there.

Section 1. The problem in a nutshell

Let me start by summarizing the problem that was uncovered in more detail. The root of the problem is an interaction between the reference-counting and threading APIs in the standard library. So let’s look at each in turn. If you’re familiar with the problem, you can skip ahead to section 2.

Reference-counting as the poor man’s GC

Rust’s standard library includes the Rc and Arc types which are used for reference-counted data. These are widely used, because they are the most convenient way to create data whose ownership is shared amongst many references rather than being tied to a particular stack frame.

Like all reference-counting systems, Rc and Arc are vulnerable to reference-count cycles. That is, if you create a reference-counted box that contains a reference to itself, then it will never be collected. To put it another way, Rust gives you a lot of safety guarantees, but it doesn’t protect you from memory leaks (or deadlocks, which turns out to be a very similar problem).

The fact that we don’t protect against leaks is not an accident. This was a deliberate design decision that we made while transitioning from garbage-collected types (@T and @mut T) to user-defined reference counting. The reason is that preventing leaks requires either a runtime with a cycle collector or complex type-system tricks. The option of a mandatory runtime was out, and the type-system tricks we explored were either too restrictive or too complex. So we decided to make a pragmatic compromise: to document the possibility of leaks (see, for example, this section of the Rust reference manual) and move on.

In practice, the possibility of leaks is mostly an interesting technical caveat: I’ve not found it to be a big issue in practice. Perhaps because problems arose so rarely in practice, some things—like leaks—that should not have been forgotten were… partially forgotten. History became legend. Legend became myth. And for a few years, the question of leaks seemed to be a distant, settled issue, without much relevance to daily life.

Thread and shared scopes

With that background on Rc in place, let’s turn to threads. Traditionally, Rust threads were founded on a “zero-sharing” principle, much like Erlang. However, as Rust’s type system evolved, we realized we could do much betterthe same type system rules that ensured memory safe in sequential code could be used to permit sharing in parallel code as well, particularly once we adopted RFC 458 (a brilliant insight by pythonesque).

The basic idea is to start a child thread that is tied to a particular scope in the code. We want to guarantee that before we exit that scope, the thread will be joined. If we can do this, then we can safely permit that child thread access to stack-allocated data, so long as that data outlives the scope; this is safe because Rust’s type-system rules already ensure that any data shared between multiple threads must be immutable (more or less, anyway).

So the question then is how can we designate the scope of the children threads, and how can we ensure that the children will be joined when that scope exits. The original proposal was based on closures, but in the time since it was written, the language has shifted to using more RAII, and hence the scoped API is based on RAII. The idea is pretty simple. You write a call like the following:

1
2
3
4
5
fn foo(data: &[i32]) {
  ...
  let guard = thread::scoped(|| /* body of the child thread */);
  ...
}

The scoped function takes a closure which will be the body of the child thread. It returns to you a guard value: running the destructor of this guard will cause the thread to be joined. This guard is always tied to a particular scope in the code. Let’s call the scope 'a. The closure is then permitted access to all data that outlives 'a. For example, in the code snippet above, 'a might be the body of the function foo. This means that the closure could safely access the input data, because that must outlive the fn body. The type system ensures that no reference to the guard exists outside of 'a, and hence we can be sure that guard will go out of scope sometime before the end of 'a and thus trigger the thread to be joined. At least that was the idea.

The conflict

By now perhaps you have seen the problem. The scoped API is only safe if we can guarantee that the guard’s destructor runs, so that the thread will be joined; but, using Rc, we can leak values, which means that their destructors never run. So, by combining Rc and scoped, we can cause a thread to be launched that will never be joined. This means that this thread could run at any time and try to access data from its parents stack frame – even if that parent has already completed, and thus the stack frame is garbage. Not good!

So where does the fault lie? From the point of view of history, it is pretty clear: the scoped API was ill designed, given that Rc already existed. As I wrote, we had long ago decided that the most practical option was to accept that leaks could occur. This implies that if the memory safety of an API depends on a destructor running, you can’t relinquish ownership of the value that carries that destructor (because the end-user might leak it).

It is totally possible to fix the scoped API, and in fact there is already an RFC showing how this can be done (I’ll summarize it in section 2, below). However, some people feel that the decision we made to permit leaks was the wrong one, and that we ought to have some limits on the RC API to prevent leaks, or at least prevent some leaks. I’ll dig into those proposals in section 3.

Section 2. What is the impact of leaks on the status quo?

So, if we continue with the status quo, and accept that resource leaks can occur with Rc and Arc, what is the impact of that? At first glance, it might seem that the possibility of resource leaks is a huge blow to RAII. After all, if you can’t be sure that the destructor will run, how can you rely on the destructor to do cleanup? But when you look closer, it turns out that the problem is a lot more narrow.

“Average Rust User”

I think it’s helpful to come at this problem from two difference perspectives. The first is: what do resource leaks mean for the average Rust user? I think the right way to look at this is that the user of the Rc API has an obligation to avoid cycle leaks or break cycles. Failing to do so will lead to bugs – these could be resource leaks, deadlocks, or other things. But leaks cannot lead to memory unsafety. (Barring invalid unsafe code, of course.)

It’s worth pointing out that even if you are using Rc, you don’t have to worry about memory leaks due to forgetting to decrement a reference or anything like that. The problem really boils down to ensuring that you have a clear strategy for avoiding cycles, which usually boils to an “ownership DAG” of strong references (though in some cases, breaking cycles explicitly may also be an option).

“Author of unsafe code”

The other perspective to consider is the person who is writing unsafe code. Unsafe code frequently relies on destructors to do cleanup. I think the right perspective here is to view a destructor as akin to any other user-facing function: in particular, it is the user’s responsibility to call it, and they may accidentally fail to do so. Just as you have to write your API to be defensive about users invoking functions in the wrong order, you must be defensive about them failing to invoke destructors due to a resource leak.

It turns out that the majority of RAII idioms are actually perfectly memory safe even if the destructors don’t run. For example, if we examine the Rust standard library, it turns out that all of the destructors therein are either optional or can be made optional:

  1. Straight-forward destructors like Box or Vec leak memory if they are not freed; clearly no worse than the original leak.
  2. Leaking a mutex guard means that the mutex will never be released. This is likely to cause deadlock, but not memory unsafety.
  3. Leaking a RefCell guard means that the RefCell will remain in a borrowed state. This is likely to cause thread panic, but not memory unsafety.
  4. Even fancy iterator APIs like drain, which was initially thought to be problematic, can be implemented in such a way that they cause leaks to occur if they are leaked, but not memory unsafety.

In all of these cases, there is a guard value that mediates access to some underlying value. The type system already guarantees that the original value cannot be accessed while the guard is in scope. But how can we ensure safety outside of that scope in the case where the guard is leaked? If you look at the the cases above, I think they can be grouped into two patterns:

  1. Ownership: Things like Box and Vec simply own the values they are protecting. This means that if they are leaked, those values are also leaked, and hence there is no way for the user to access it.
  2. Pre-poisoning: Other guards, like MutexGuard, put the value they are protecting into a poisoned state that will lead to dynamic errors (but not memory unsafety) if the value is accessed without having run the destructor. In the case of MutexGuard, the “poisoned” state is that the mutex is locked, which means a later attempt to lock it will simply deadlock unless the MutexGuard has been dropped.

What makes scoped threads different?

So if most RAII patterns continue to work fine, what makes scoped different? I think there is a fundamental difference between scoped and these other APIs; this difference was well articulated by Kevin Ballard:

thread::scoped is special because it’s using the RAII guard as a proxy to represent values on the stack, but this proxy is not actually used to access those values.

If you recall, I mentioned above that all the guards serve to mediate access to some value. In the case of scoped, the guard is mediating access to the result of a computation – the data that is being protected is “everything that the closure may touch”. The guard, in other words, doesn’t really know the specific set of affected data, and it thus cannot hope to either own or pre-poison the data.

In fact, I would take this a step farther, and say that I think that in this kind of scenario, where the guard doesn’t have a connection to the data being protected, RAII tends to be a poor fit. This is because, generally, the guard doesn’t have to be used, so it’s easy for the user to accidentally drop the guard on the floor, causing the side-effects of the guard (in this case, joining the thread) to occur too early. I’ll spell this out a bit more in the section below.

Put more generally, accepting resource leaks does mean that there is a Rust idiom that does not work. In particular, it is not possible to create a borrowed reference that can be guaranteed to execute arbitrary code just before it goes out of scope. What we’ve seen though is that, frequently, it is not necessary to guarantee that the code will execute – but in the case of scoped, because there is no direct connection to the data being protected, joining the thread is the only solution.

Using closures to guarantee code execution when exiting a scope

If we can’t use an RAII-based API to ensure that a thread is joined, what can we do? It turns out that there is a good alternative, laid out in RFC 1084. The basic idea is to restructure the API so that you create a “thread scope” and spawn threads into that scope (in fact, the RFC lays out a more general version that can be used not only for threads but for any bit of code that needs guaranteed execution on exit from a scope). This thread scope is delinated using a closure. In practical terms, this means that started a scoped thread look something like this:

1
2
3
4
5
6
7
fn foo(data: &[i32]) {
  ...
  thread::scope(|scope| {
    let future = scope.spawn(|| /* body of the child thread */);
    ...
  });
}

As you can see, whereas before calling thread::scoped started a new thread immediately, it now just creates a thread scope – it doesn’t itself start any threads. A borrowed reference to the thread scope is passed to a closure (here it is the argument scope). The thread scope offers a method spawn that can be used to start a new thread tied to a specific scope. This thread will be joined when the closure returns; as such, it has access to any data that outlives the body of the closure. Note that the spawn method still returns a future to the result of the spawned thread; this future is similar to the old join guard, because it can be used to join the thread early. But this future doesn’t have a destructor. If the thread is not joined through the future, it will still be automatically joined when the closure returns.

In the case of this particular API, I think closures are a better fit than RAII. In particular, the closure serves to make the scope where the threads are active clear and explicit; this in turn avoids certain footguns that were possible with the older, RAII-based API. To see an example of what I mean, consider this code that uses the old API to do a parallel quicksort:

1
2
3
4
5
6
7
8
fn quicksort(data: &mut [i32]) {
  if data.len() <= 1 { return; }
  let pivot = data.len() / 2;
  let index = partition(data, pivot);
  let (left, right) = data.split_at_mut(data, index);
  let _guard1 = thread::scoped(|| quicksort(left));
  let _guard2 = thread::scoped(|| quicksort(right));
}

I want to draw attention to one snippet of code at the end:

1
2
3
  let (left, right) = data.split_at_mut(data, index);
  let _guard1 = thread::scoped(|| quicksort(left));
  let _guard2 = thread::scoped(|| quicksort(right));

Notice that we have to make dummy variables like _guard1 and _guard2. If we left those variables off, then the thread would be immediately joined, which means we wouldn’t get any actual parallelism. What’s worse, the code would still work, it would just run sequentially. The need for these dummy variables, and the resulting lack of clarity about just when parallel threads will be joined, is a direct result of using RAII here.

Compare that code above to using a closure-based API:

1
2
3
4
5
  thread::scope(|scope| {
    let (left, right) = data.split_at_mut(data, index);
    scope.spawn(|| quicksort(left));
    scope.spawn(|| quicksort(right));
  });

I think it’s much clearer. Moreover, the closure-based API opens the door to other methods that could be used with scope, like convenience methods to do parallel maps and so forth.

Section 3. Can we prevent (some) resource leaks?

Ok, so in the previous two sections, I summarized the problem and discussed the impact of resource leaks on Rust. But what if we could avoid resource leaks in the first place? There have been two RFCs on this topic: RFC 1085 and RFC 1094.

The two RFCs are quite different in the details, but share a common theme. The idea is not to avoid all resource leaks altogether; I think everyone recognizes that this is not practical. Instead, the goal is to try and divide types into two groups: those that can be safely leaked, and those that cannot. You then limit the Rc and Arc types so that they can only be used with types that can safely be leaked.

This approach seems simple but it has deep ramifications. It means that Rc and Arc are no longer fully general container types. Generic code that wishes to operate on data of all types (meaning both types that can and cannot leak) can’t use Rc or Arc internally, at least not without some hard choices.

Rust already has a lot of precedent for categorizing types. For example, we use a trait Send to designate “types that can safely be transferred to other threads”. In some sense, dividing types into leak-safe and not-leak-safe is analogous. But my experience has been that every time we draw a fundamental distinction like that, it carries a high price. This distinction “bubbles up” through APIs and affects decisions at all levels. In fact, we’ve been talking about one case of this rippling effect through this post – the fact that we have two reference-counting types, one atomic (Arc) and one not (Rc), is precisely because we want to distinguish thread-safe and non-thread-safe operations, so that we can get better performance when thread safety is not needed.

What this says to me is that we should be very careful when introducing blanket type distinctions. The places where we use this mechanism today – thread-safety, copyability – are fundamental to the language, and very important concepts, and I think they carry their weight. Ultimately, I don’t think resource leaks quite fit the bill. But let me dive into the RFCs in question and try to explain why.

RFC 1085 – the Leak trait

The first of the two RFCs is RFC 1085. This RFC introduces a trait called Leak, which operates exactly like the existing Send trait. It indicates “leak-safe” data. Like Send, it is implemented by default. If you wish to make leaks impossible for a type, you can explicitly opt out with a negative impl like impl !Leak for MyType. When you create a Rc<T> or Arc<T>, either T: Leak must hold, or else you must use an unsafe constructor to certify that you will not create a reference cycle.

The fact that Leak is automatically implemented promises to make it mostly invisible. Indeed, in the prototype that Jonathan Reem implemented, he found relatively little fallout in the standard library and compiler. While encouraging, I still think we’re going to encounter problems of composability over time.

There are a couple of scenarios where the Leak trait will, well, leak into APIs where it doesn’t seem to belong. One of the most obvious is trait objects. Imagine I am writing a serialization library, and I have a Serializer type that combines an output stream (a Box<Writer>) along with some serialization state:

1
2
3
4
5
struct Serializer {
  output_stream: Box<Writer>,
  serialization_state: u32,
  ...
}

So far so good. Now someone else comes along and would like to use my library. They want to put this Serializer into a reference counted box that is shared amongst many users, so they try to make a Rc<Serializer>. Unfortunately, this won’t work. This seems somewhat surprising, since weren’t all types were supposed to be Leak by default?

The problem lies in the Box<Writer> object – an object is designed to hide the precise type of Writer that we are working with. That means that we don’t know whether this particular Writer implements Leak or not. For this client to be able to place Serializer into an Rc, there are two choices. The client can use unsafe code, or I, the library author, can modify my Serializer definition as follows:

1
2
3
4
5
struct Serializer {
  output_stream: Box<Writer+Leak>,
  serialization_state: u32,
  ...
}

This is what I mean by Leak “bubbling up”. It’s already the case that I, as a library author, want to think about whether my types can be used across threads and try to enable that. Under this proposal, I also have to think about whether my types should be usable in Rc, and so forth.

Now, if you avoid trait objects, the problem is smaller. One advantage of generics is that they don’t encapsulate what type of writer you are using and so forth, which means that the compiler can analyze the type to see whether it is thread-safe or leak-safe or whatever. Until now we’ve found that many libraries avoid trait objects partly for this reason, and I think that’s good practice in simple cases. But as things scale up, encapsulation is a really useful mechanism for simplifying type annotations and making programs concise and easy to work with.

There is one other point. RFC 1085 also includes an unsafe constructor for Rc, which in principle allows you to continue using Rc with any type, so long as you are in a position to assert that no cycles exist. But I feel like this puts the burden of unsafety into the wrong place. I think you should be able to construct reference-counted boxes, and truly generic abstractions built on reference-counted boxes, without writing unsafe code.

My allergic reaction to requiring unsafe to create Rc boxes stems from a very practical concern: if we push the boundaries of unsafety too far out, such that it is common to use an unsafe keyword here and there, we vastly weaken the safety guarantees of Rust in practice. I’d rather that we increase the power of safe APIs at the cost of more restrictions on unsafe code. Obviously, there is a tradeoff in the other direction, because if the requirements on unsafe code become too subtle, people are bound to make mistakes there too, but my feeling is that requiring people to consider leaks doesn’t cross that line yet.

RFC 1094 – avoiding reference leaks

RFC 1094 takes a different tack. Rather than dividing types arbitrarily into leak-safe and not-leak-safe, it uses an existing distinction, and says that any type which is associated with a scope cannot leak.

The goal of RFC 1094 is to enable a particular “mental model” about what lifetimes mean. Specifically, the RFC aims to ensure that if a value is limited to a particular scope 'a, then the value will be destroyed before the program exits the scope 'a. This is very similar to what Rust currently guarantees, but stronger: in current Rust, there is no guarantee that your value will be destroyed, there is only a guarantee that it will not be accessed outside that scope. Concretely, if you leak an Rc into the heap today, that Rc may contain borrowed references, and those references could be invalid – but it doesn’t matter, because Rust guarantees that you could never use them.

In order to guarantee that borrowed data is never leaked, RFC 1094 requires that to construct a Rc<T> (or Arc<T>), the condition T: 'static must hold. In other words, the payload of a reference-counted box cannot contain borrowed data. This by itself is very limiting: lots of code, including the rust compiler, puts borrowed pointers into reference-counted structures. To help with this, the RFC includes a second type of reference-counted box, ScopedRc. To use a ScopedRc, you must first create a reference-counting scope s. You can then create new ScopedRc instances associated with s. These ScopedRc instances carry their own reference count, and so they will be freed normally as soon as that count drops to zero. But if they should get placed into a cycle, then when the scope s is dropped, it will go along and “cycle collect”, meaning that it runs the destructor for any ScopedRc instances that haven’t already been freed. (Interestingly, this is very similar to the closure-based scoped thread API, but instead of joining threads, exiting the scope reaps cycles.)

I originally found this RFC appealing. It felt to me that it avoided adding a new distinction (Leak) to the type system and instead piggybacked on an existing one (borrowed vs non-borrowed). This seems to help with some of my concerns about “ripple effects” on users.

However, even though it piggybacks on an existing distinction (borrowed vs static), the RFC now gives that distinction additional semantics it didn’t have before. Today, those two categories can be considered on a single continuum: for all types, there is some bounding scope (which may be 'static), and the compiler ensures that all accesses to that data occur within that scope. Under RFC 1094, there is a discontinuity. Data which is bounded by 'static is different, because it may leak.

This discontinuity is precisely why we have to split the type Rc into two types, Rc and ScopedRc. In fact, the RFC doesn’t really mention Arc much, but presumably there will have to be both ScopedRc and a ScopedArc types. So now where we had only two types, we have four, to account for this new axis:

1
2
3
4
5
6
|-----------------++--------+----------|
|                 || Static | Borrowed |
|-----------------++--------+----------|
| Thread-safe     || Rc     | RcScoped |
| Not-thread-safe || Arc    | ArcScope |
|-----------------++--------+----------|

And, in fact, the distinction doesn’t end here. There are abstractions, such as channels, that built on Arc. So this means that this same categorization will bubble up through those abstractions, and we will (presumably) wind up with Channel and ChannelScoped (otherwise, channels cannot be used to send borrowed data to scoped threads, which is a severe limitation).

Section 4. Conclusion.

This concludes my deep dive into the question of resource leaks. It seems to me that the tradeoffs here are not simple. The status quo, where resource leaks are permitted, helps to ensure composability by allowing Rc and Arc to be used uniformly on all types. I think this is very important as these types are vital building blocks.

On a historical note, I am particularly sensitive to concerns of composability. Early versions of Rust, and in particular the borrow checker before we adopted the current semantics, were rife with composability problems. This made writing code very annoying – you were frequently refactoring APIs in small ways to account for this.

However, this composability does come at the cost of a useful RAII pattern. Without leaks, you’d be able to use RAII to build references that reliably execute code when they are dropped, which in turn allows RAII-like techniques to be used more uniformly across all safe APIs.

This is certainly a subtle issue, and one where reasonable folk can disagree. In the process of drafting (and redrafting…) this post, my own opinion has shifted back and forth as well. But ultimately I have landed where I started: the danger and pain of bifurcating the space of types far outweighs the loss of this particular RAII idiom.

Here are the two most salient points to me:

  1. The vast majority of RAII-based APIs are either safe or can be made safe with small changes. The remainder can be expressed with closures.
    • With regard to RAII, the scoped threads API represents something of a “worst case” scenario, since the guard object is completely divorced from the data that the thread will access.
    • In cases like this, where there is often no need to retain the guard, but dropping it has important side-effects, RAII can be a footgun and hence is arguably a poor fit anyhow.
  2. The cost of introducing a new fundamental distinction (“leak-safe” vs “non-leak-safe”) into our type system is very high and will be felt up and down the stack. This cannot be completely hidden or abstracted away.
    • This is similar to thread safety, but leak-safety is far less fundamental.

Bottom line: the cure is worse than the disease.

Doug BelshawGuiding Students as They Explore, Build, and Connect Online

Ian O'Byne, Greg McVerry and I have just had an article published in the Journal of Adolescent & Adult Literacy (JAAL). Entitled Guiding Students as They Explore, Build, and Connect Online it’s an attempt to situate and explain the importance of the Web Literacy Map work.

Publish

I’d have preferred it be published in an open access journal, but there was a window of opportunity that we decided to take advantage of. Happily, you can access the pre-publication version via Ian’s blog here.

Related:

Cite this article:

McVerry, J.G., Belshaw, D. & Ian O'Byrne, W. (2015). Guiding Students as They Explore, Build, and Connect Online. Journal of Adolescent & Adult Literacy, 58(8), 632–635. doi: 10.1002/jaal.411

Air MozillaProduct Coordination Meeting

Product Coordination Meeting Duration: 10 minutes This is a weekly status meeting, every Wednesday, that helps coordinate the shipping of our products (across 4 release channels) in order...

Eitan IsaacsoneSpeak Web Speech API Addon

Now that eSpeak runs pretty well in JS, it is time for a Web Speech API extension!

What is the Web Speech API? It gives any website access to speech synthesis (and recognition) functionality, Chrome and Safari already have this built-in. This extension adds speech synthesis support in Firefox, and adds eSpeak voices.

For the record, we had speech synthesis support in Gecko for about 2 years. It was introduced for accessibility needs in Firefox OS, now it is time to make sure it is supported on desktop as well.

Why an extension instead of built-in support? A few reasons:

  1. An addon will provide speech synthesis to Firefox now as we implement built-in platform-specific solutions for future releases.
  2. An addon will allow us to surface current bugs both in our Speech API implementation, and in the spec.
  3. We designed our speech synthesis implementation to be extensible with addons, this is a good proof of concept.
  4. People are passionate about eSpeak. Some people love it, some people don’t.

So now I will shut up, and let eSpeak do the talking:


Mozilla Release Management TeamFirefox 38 beta6 to beta8

This beta was harder then usual to release. For the 38.0.5 dev cycle, we decided to merge mozilla-beta into mozilla-release. As this is unusual, we encountered several issues:

  • Some l10n updates had to be run (beta2release_l10n.sh). To help with the diagnostics, we reported bug 1158126.
  • The automation tool had an expectation that, coming from mozilla-release, the version would be a release (bug 1158124). We hardcoded a temporary fix.

Because it took some time to fix these issues, we haven't been able to publish beta 7 last week. We decided to skip beta 7 from releasing and start the beta 8 build on Sunday evening. Unfortunately, this didn't go smoothly either:

  • During the merge, we update some of the configurations. This caused beta 8 build 1 to be built using the release branding. This change was backout. See bug 1158760 for more information.
  • Last but not least, because of the previous issue, we had to do a second build of 38 beta 8. This caused some CDN issue and it took a while to get that resolved. We also reported a bug to simplify this in the future.

Besides that, these two betas are regular. We disabled readling list and reader view (reader view is going to ship into 38.0.5). We took some fixes for EME and MSE.

Finally, we took some stability fixes.

  • 56 changesets
  • 131 files changed
  • 1488 insertions
  • 1911 deletions

ExtensionOccurrences
cpp36
js32
h12
jsm8
java7
ini7
css5
list3
in3
html3
jsx2
idl2
build2
xml1
txt1
sh1
mk1
hgtags1
gyp1
dep1
cc1

ModuleOccurrences
browser35
dom16
toolkit14
media12
mobile10
js10
security7
gfx6
layout5
widget3
testing3
docshell3
services2
netwerk1
mozglue1
modules1

List of changesets:

David KeelerBug 1150114 - Allow PrintableString to match UTF8String in name constraints checking. r=briansmith, a=sledru - a00d0de3202f
Justin DolskeBug 1155191 - Please disable readling list and reader view for 38. r=markh, a=me - 69173cc17556
Kai EngertBug 1156428 - Upgrade Firefox 38 to use NSS 3.18.1, a=dveditz - 8f9c08f19f6a
Patrick BrossetBug 1155172 - Intermittent browser_webconsole_notifications.js. r=past, a=test-only - 52322e98f739
Matt WoodrowBug 1154536 - Disable 4k H264 video for vista since it performs poorly. r=ajones, a=sledru - 650ed1bb5a04
Philipp KewischBug 1153192 - Cannot pass extra arguments to l10n-repack.py. r=gps, a=lmandel - d1e5b60cd47c
Chris PearceBug 1156131 - Use correct DLL on WinVista, 7, and 8 for WMF decoding in gmp-clearkey. r=edwin, a=sledru - e7210d2ce8a9
Chris PearceBug 1156131 - Expand list of WMF DLLs that are whitelisted for use by EME plugins. r=bobowen, a=sledru - 5712fefbace8
Mark HammondBug 1152193 - Ensure sync/readinglist log directory exists. r=rnewman, a=sledru - fc98815acf5f
Ed LeeBug 1156921 - Backout Suggested Tiles (Bug 1120311) from 38.0 [a=sylvestre] - d7ca3b75c842
Ryan VanderMeulenBug 1123563 - Skip test-animated-image-layers.html and test-animated-image-layers-background.html on Android and Linux. a=test-only - 1cd478c3e0b5
Hannes VerschoreBug 1140890 - Make sure the first argument cannot bail in between negative zero removal and creating result in substraction. r=nbp, a=sledru - d55fdde73ac8
Valentin GosuBug 1145812 - Fix assertion with dom.url.encode_decode_hash pref set to true. r=mcmanus, a=sledru - 5f0e381a7afd
Hannes VerschoreBug 1143878 - IonMonkey: Test conversion of MToInt32 for testing congruence. r=jandem, a=sledru - 0b3c5b65610e
Valentin GosuBug 1149913 - Disable Bug 1093611. Set pref dom.url.encode_decode_hash to true. r=honzab, a=sledru - a9be9167d92b
Chris PearceBug 1155432 - Don't flush WMF PDM task queues. r=jya, a=sledru - 0920ace0d8b0
Julian SewardBug 1153173 - Uninitialised value use in AutoJSExceptionReporter::~AutoJSExceptionReporter. r=aklotz, a=sledru - 92fb098ace7a
Jean-Yves AvenardBug 1154683 - Fix potential size overflow. r=kentuckyfriedtakahe, a=sledru - 22f8fa3a9273
Milan SreckovicBug 1133119 - ::Map should fail if the data is null, and add more null pointer checks. r=mattwoodrow, a=sledru - 90d2538212ab
Florian QuèzeBug 1109728 - Intermittent browser_devices_get_user_media.js | popup WebRTC indicator visible - Got false, expected true. r=Gijs, a=test-only - fe8c5e74565f
Florian QuèzeBug 1126107 - Intermittent browser_devices_get_user_media.js | WebRTC indicator hidden - Got true, expected false. r=Gijs, a=test-only - 8d4a0b33d32e
Jim MathiesBug 1100501 - Add StatisticsRecorder initialization to xpcshell. r=georg, a=sledru - 71d1d59db847
Jim MathiesBug 1100501 - Avoid a late shutdown of chromium's StatisticsRecorder. r=georg, a=sledru - 8661ed4cbdb9
Mark BannerBug 1153630 - Allow buttons in the Loop panel to be bigger if required as L10n needs. r=dmose, a=sledru - a6fe316e7571
Milan SreckovicBug 1154003 - More protection for failed surface drawable creation. r=bas, a=sledru - 474ffd404414
Valentin GosuBug 1139831 - End timestamps are before start timestamps. r=baku, a=sledru - 9fe28719e4fd
Mats PalmgrenBug 1152354 - Re-introduce the incremental reflow hack in nsSimplePageSequenceFrame for now, since the regressions are worse than the original problem (Bug 1108104). r=roc, a=sledru - 92a269ca564d
Garvan KeeleyBug 1155237 - Part 1: Remove contextless access to NetworkUtils, causes NPE. r=rnewman, a=sledru - 1ec2ee773b51
Garvan KeeleyBug 1155237 - Part 2: Make upload service non-sticky. r=rnewman, a=sledru - 645fc5aa6a49
Mark BannerBug 1145541. r=mikedeboer, a=sledru - db41e8e267ed
Ryan VanderMeulenBug 1108104 - Fix rebase bustage. a=bustage - df5d106c2607
Ryan VanderMeulenBug 1152354 - Remove no longer needed assertion expectation. a=orange - 50550eca1fa2
JW WangBug 1091155 - Don't check if 'playing' has fired for it depends on how fast decoding is which is not reliable. r=cpearce, a=test-only - 2161d1dc7e2b
Randell JesupBug 1151628 - Re-enable MJPEG in libyuv (especially for getUserMedia). r=glandium, a=sledru - f6448c4cf87f
Randell JesupBug 1152016 - Suppress fprintf(stderr)'s from jpeg in MJPEG decode. r=pkerr, a=sledru - 97d33db56113
Ganesh SahukariBug 1009465 - Set the read-only attribute for temporary downloads on Windows. r=paolo, a=sledru - b7d8d79c1ee5
Tom SchusterBug 1152550 - Make sure that cross-global Iterator can not be broken. r=Waldo, a=sledru - 6b096f9b31d3
Mark FinkleBug 1154960 - Fennec should explicitly block the DOM SiteSpecificUserAgent.js file from packaging. r=nalexander, a=sledru - da1d9ba28360
Richard NewmanBug 1155684 - Part 0: Disable reading list sync in confvars.sh. r=nalexander, a=sledru - 18c8180670c7
Richard NewmanBug 1155684 - Part 1-3: Remove reading list sync integration. r=nalexander, a=sledru - 309ed42a5999
Richard MartiBug 1156913 - Use highlighttext color also for :active menus. r=Gijs, a=sledru - 98086516ce8f
Edwin FloresBug 1156560 - Prefer old CDMs on update if they are in use. r=cpearce, ba=sledru - 7c66212e4c09
Ryan VanderMeulenBacked out changeset 6b096f9b31d3 (Bug 1152550) for bustage. - d20a4e36e508
Ryan VanderMeulenBug 1139591 - Skip browser_timeline_overview-initial-selection-01.js on OSX debug. a=test-only - c0624fb0b902
Ganesh SahukariBug 1022816 - OS.File will now be able to change the readOnly, hidden, and system file attributes on Windows. r=paolo, a=sledru - 8a2c933394da
Blake KaplanBug 1156939 - Don't stash a reference to a CPOW and then spin the event loop. r=mconley, a=test-only - 0efa961d5162
Jonas JenwaldBug 1112947 - Replace a setTimeout with an EventListener to fix an intermittent failure in browser/extensions/pdfjs/test/browser_pdfjs_navigation.js. r=mossop, a=test-only - b29a45098630
Jared WeinBug 1153403 - Don't allow dialogs to resize if they didn't resize in windowed preferences. r=Gijs, a=sledru - e46c9612492a
Matt WoodrowBug 1144257 - Blacklist DXVA for one NVIDIA driver that was causing crashes. r=ajones, a=sledru - 78c6b3ce2ce2
Tom SchusterBug 1152550 - Make sure that cross-global Iterator can not be broken. r=Waldo, a=sledru - 2025aa8c5b1b
travisBug 1154803 - Put our sigaction diversion in __sigaction if it exists. r=glandium, a=sledru - fd5c74651fb2
Neil RashbrookBug 968334 - Allow disabling content retargeting on child docshells only. r=smaug, ba=sledru - 38ff61772a2e
Nicolas B. PierronBug 1149119 - Use Atoms in the template object hold by Baseline. r=jandem, a=abillings - 7298f6e3943e
Nicolas B. PierronBug 1149119 - Do not inline bound functions with non-atomized arguments. r=jandem, a=abillings - 0e69c76cbbe2
Ryan VanderMeulenBacked out changeset b29a45098630 (Bug 1112947) for test bustage. - 8fc6195511e5
Rail AliievBug 1158760 - Wrong branding on the 38 Beta 8, backout d27c9211ebb3. IGNORE BROKEN CHANGESETS CLOSED TREE a=release ba=release - 9d105ed6f35a

Mike TaylorReferenceError onTouchStart is not defined jquery.flexslider.js

I was supposed to write this blog post like a year ago, but have been very busy in the last 12 months not writing this blog post. But yet here we are.

Doing some compatibility research on top Japanese sites, I ran into my old nemesis: ReferenceError: onTouchStart is not defined jquery.flexslider.js:397:12.

I first ran into this in January of 2014 in its more spooky form ReferenceError: g is not defined. Eventually I figured out it was a problem in a WooThemes jQuery plugin called FlexSlider, the real bug being the undefined behavior of function declaration hoisting in conditions (everyone just nod like that makes sense).

In JavaScript-is-his-co-pilot Juriy's words,

Another important trait of function declarations is that declaring them conditionally is non-standardized and varies across different environments. You should never rely on functions being declared conditionally and use function expressions instead.

In this case, they were conditionally declaring a function, but referencing it before said declaration in the if block, as an event handler, i.e.,

if (boop) {
  blah.addEventListener("touchstart", wowowow);

  function wowowow() {}
}

No big deal, easy fix. I wrote a patch. Some people manually patched their sites and moved on with their lives. I tried to.

We ran into this a few more times in Bugzilla and Webcompat.com land.

TC-39 bosom buddies Brendan and Allen noted that ES6 specifies things in such a way that these sites will eventually work in all ES62015 compliant browsers. Here's the bug to track that work in SpiderMonkey.

Cool! Until then, my lonely pull request is still hanging out at https://github.com/woothemes/FlexSlider/pull/986 (16 months later). The good news is FlexSlider is open source, so you're allowed to fix their bugs by manually applying that patch on your site. Then your touch-enabled slider widget stuff will work in Mobile Firefox browsers.

Christian HeilmannThe new challenges of “open”

These are my notes for my upcoming keynote at he Oscal conference in Tirana, Albania.

Today I want to talk about the new challenges of “open”. Open source, Creative Commons, and many other ideas of the past have become pretty much mainstream these days. It is cool to be open, it makes sense for a lot of companies to go that way. The issue is, that – as with anything fashionable and exciting – people are wont to jump on the bandwagon without playing the right tune. And this is one of the big challenges we’re facing.

Before we go into that, though, let’s recap the effects of going into the open with our work has.

Creating in the open is an empowering and frightening experience. The benefits are pretty obvious:

  • You share the load – people can help you with feedback, doing research for you, translating your work, building adapters to other environments for you.
  • You have a good chance your work will go on without you – as you shared, others can build upon your work when you moved on to other challenges; or got run over by a bus.
  • Sharing feels good – it’s a sort of altruism that doesn’t cost you any money and you see the immediate effect.
  • You become a part of something bigger – people will use your work in ways you probably never intended, and never thought of. Seeing this is incredibly exciting.

The downsides of working in the open are based on feedback and human interaction.

  • You’re under constant surveillance – you can’t hide things away when you openly share your work in progress. This can be a benefit as it means your product is higher quality when you’re under constant scrutiny. It can, however, also be stifling as you’re more worried about what people think about your work rather than what the work results in.
  • You have to allocate your time really well – feedback will come 24/7 and in many cases not in a format that is pleasing or – in some cases – even intelligible.
  • You have to pick your battles – people will come with all kind of requests and it is easy to get lost in pleasing the audience instead of finishing your product.
  • You have to prepare yourself for having to adhere to existing procedures – years of open source work resulted in many best practices and very outspoken people are quick to demand you adhere to them or stay off the lawn.

Hey cool kids, we’re doing the open thing!

One of the main issues with open is that people are not really aware of the amount of work it is. It is very fashionable to release products as open source. But, in many cases, this means putting the code on GitHub and hoping for a magical audience to help you and fix your problems. This is not how open source prospers.

Open Source and related ways of working does not mean you give out your work for free and leave it at that. It means that you make it available, that you nurture it and that you are open to giving up control for the benefit of the wisdom of the crowd. It is a two-way, three-way, many way exchange of data and information. You give something out, but you also get a lot back, and either deserves the same attention and respect.

More and more I find companies and individuals seeing open sourcing not as a way of working, but as an advertising and hiring exercise. Products get released openly but there is no infrastructure or people in place to deal with the overhead this means. It has become a ribbon to attach to your product – “also available on GitHub”.

We’ve been through that before – the mashup web and open APIs promised us developers that we can build great, scaling and wonderful products by using the power of the web. We pick and mix our content providers with open APIs and build our interfaces on top of that data. This died really quickly and today most APIs we played with are either shut down or became pay-to-play.

Other companies see “open” as a means to keep things alive that are not supported any longer. It’s like the mythical farm the family dog goes to when the kids ask where you take him when he gets old and sick. “Don’t worry, the product doesn’t mesh with the core business of our company any longer, but it will live on as it is now open source” is the message. And it is pretty much a useless one. We need products that are supported, maintaned and used. Simply giving stuff out for free doesn’t mean this will happen to that product, as it means a lot of work for the maintainers. In many cases shutting a product down is the more honest thing to do.

If you want to be open about it – do it our way

The other issue with open is that – ironically – open communities can come across as uninviting and aggressive. We are a passionate bunch, and care a lot about what we do. That can make us appear overly defensive and aggressive. Many long-standing open communities have methodologies in place to ensure quality that on first look can be daunting and off-putting.

Many companies understand the value of open, but are hesitant to open up their products because of this. The open community can come across as very demanding. And it is very easy to get an avalanche of bad feedback when you release something into the open but you fail to tick all the boxes. This is poison to anyone in a large company trying to release something closed into the open. You have to justify your work to the business people in the company. And if all you have to show is an avalanche of bad feedback and passive-aggressive “oh look, evilcorp is trying to play nice” comments, they are not going to be pleased with you.

We’re not competing with technology – we’re competing with marketing and tech propaganda

The biggest issue I see with open is that it has become a tool. Many of the closed environments that are in essence a replacement for the open web are powered by open technologies. This is what they are great for. The plumbing of the web runs on open. We’re a useful cog, and – to be fair – a lot of closed companies also support and give back to these products.

On the other hand, when you talk about a fully open product and try to bring it to end users, you are facing an uphill battle. Almost every open alternative to closed (or partly open systems) struggles or – if we are honest with ourselves – failed. Firefox OS is not taking the world by storm and brings connectivity to people who badly need it. The Ubuntu phone as an alternative didn’t cause a massive stir. Ello and Diaspora didn’t make a dent in the Facebooks and Twitters of this world. The Ouya game console ran into debt very quickly and now is looking to be bought out.

The reason is that we’re fooling ourselves when it comes to the current market and how it uses technology.

Longevity is dead

We love technology. We love the web. We love how it made us who we are and we celebrate the fights we fought to keep it open. We fight for freedom of choice, we fight for data retention and ownership of data and we worry where our data goes, if it will be available in the future or what happens with it.

But we are not our audience. Our audience are the digital natives. The people who see a computer, a smartphone and the internet as a given. The people who don’t even know what it means to be offline, and who watch streaming TV shows in bulk without a sense of dread at how much this costs or if it will work. If it stops working, who cares? Let’s do something else. If our phones or computers are broken, well let’s replace them. Or go to the shop and get them repaired for us. If the phone is too slow for the next version of its operating system, fine. Obviously we need to buy a better one.

The internet and technology has become a commodity, like running water and electricity. Of course, this is not the case all over the world, and in many cases also not when you’re traveling outside the country of your contracts. But, to those who never experienced this, it is nothing to worry about. Bit by bit, the web has become the new TV. Something people consume without knowing how it works or really taking part in it.

In England, where I live, it is almost impossible to get an internet connection without some digital TV deal as part of the package. The internet access is the thing we use to consume content provided to us by the same people who sold us CDs, DVDs, and BluRays. And those who consume over the internet also fill it up with content taken from this source material. Real creativity on the web, writing and publishing is on the way out. When something is always available, you stop caring for it. It is simply a given.

Closed by design, consumable by nature

This really scares me. It means that the people who always fought the open web and the free nature of software have won. Not by better solutions or by more choice. But by offering convenience. We’ve allowed companies with better marketing than us to take over and tell people that by staying in their world, everything is easy and works magically. People trade freedom of choice and ownership of their information for convenience. And that is hard to beat. When everything works, why put effort in?

The dawn of this was the format of the app. It was a genius idea to make software a consumable, perishable product. We moved away from desktop apps to web based apps a long time ago. Email, Calendaring, even document handling has gone that way and Google showed how that can be done.

With the smartphone revolution and the lack of support for open technologies in the leading platform the app was re-born: a bespoke piece of software written for a single platform in a closed format that needs OS-specific skills and tools to create. For end users, it is an icon. It works well, it looks amazing and it ties in perfectly with the OS. Which is no surprise, as it is written exclusively for it.

Consumable, perishable products are easier to sell. That’s why the market latched on to this quickly and defined it as the new, modern way to create software.

Even worse, instead of pointing out the lack of support for interoperable and standardised technology in the operating systems of smart devices, the tech press blamed said technologies for not working on them as well as the native counterparts do.

Develop where the people are

This constant re-inforcement of closed as a good business and open as not ready and hard to do has become a thing in the development world. Most products these days are not created for the web, independent of OS or platform. The first port of call is iOS, and once its been a success there, maybe Android. But only after complaining that the fragmentation is making it impossible to work. Fragmentation that has always been a given in the open world.

A fool’s errand

It seems open has lost. It has, to a degree. But there are already signs that what’s happening is not going to last. People are getting tired of apps and being constantly reminded by them to do things for them. People are getting bored of putting content in a system that doesn’t keep them excited and jump from product to product almost monthly. The big move of almost every platform towards light-weight messaging systems instead of life streams shows that there is a desperate attempt to keep people interested.

The big market people aim for is teenagers. They have a lot of time, they create a lot of interactions and they have their parent’s money to spend if they nag long enough.

The fallacy here is that many companies think that the teenagers of now will be the users of their products in the future. When I remember what I was like as a teenager, there is a small chance that this will happen.

We’re in a bubble and it is pretty much ready to burst. When the dust settles and people start wondering how anyone could be foolish enough to spend billions on dollars on companies that promise profits and pivot every few months when it didn’t come we’ll still be there. Much like we were during the first dotcom boom.

We’re here to help!

And this is what I want to close with. It looks dire for the open web and for open technologies right now. Yes, a lot is happening, but a lot is lip-service and many of the “open solutions” are trojan horses trying to lock people into a certain service infrastructure.

And this is where I need you. The open source and open in general enthusiasts. Our job now is to show that what we do works. That what we do matters. And that what we do will not only deliver now, but also in the future.

We do this by being open. By helping people to move from closed to open. Let’s be a guiding mentor, let’s push gently instead of getting up in arms when something is not 100% open. Let’s show that open means that you build for the users and the creators of now and of tomorrow – regardless of what is fashionable or shiny.

We have to move with the evolution of computing much like anybody else. And we do it by merging with the closed, not by trying to replace it. This failed and will also in the future. We’re not here to create consumables. We’re here to make sure they are made from great, sustainable and healthy parts.

Air MozillaThe Well Tempered API

The Well Tempered API Centuries ago, a revolution in music enabled compositions to still be playable hundreds of years later. How long will your software last? This talk, originally...

Kim MoirReleng 2015 program now available

Releng 2015 will take place in concert with ICSE in Florence, Italy on May 19, 2015. The program is now available. Register here!

via romana in firenze by ©pinomoscato, Creative Commons by-nc-sa 2.0



Kim MoirLess testing, same great Firefox taste!


Running a large continuous integration farm forces you to deal with many dynamic inputs coupled with capacity constraints. The number of pushes increase.  People add more tests.  We build and test on a new platform.  If the number of machines available remains static, the computing time associated with a single push will increase.  You can scale this for platforms that you build and test in the cloud (for us - Linux and Android on emulators), but this costs more money.  Adding hardware for other platforms such as Mac and Windows in data centres is also costly and time consuming.

Do we really need to run every test on every commit? If not, which tests should be run?  How often do they need to be run in order to catch regressions in a timely manner (i.e. able to bisect where the regression occurred)


Several months ago, jmaher and vaibhav1994, wrote code to analyze the test data and determine the minimum number of tests required to run to identify regressions.  They named their software SETA (search for extraneous test automation). They used historical data to determine the minimum set of tests that needed to be run to catch historical regressions.  Previously, we coalesced tests on a number of platforms to mitigate too many jobs being queued for too few machines.  However, this was not the best way to proceed because it reduced the number of times we ran all tests, not just less useful ones.  SETA allows us to run a subset of tests on every commit that historically have caught regressions.  We still run all the test suites, but at a specified interval. 

SETI – The Search for Extraterrestrial Intelligence by ©encouragement, Creative Commons by-nc-sa 2.0
In the last few weeks, I've implemented SETA scheduling in our our buildbot configs to use the data that the analysis that Vaibhav and Joel  implemented.  Currently, it's implemented on mozilla-inbound and fx-team branches which in aggregate represent around 19.6% (March 2015 data) of total pushes to the trees.  The platforms configured to run fewer pushes for both opt and debug are
  • MacOSX (10.6, 10.10)
  • Windows (XP, 7, 8)
  • Ubuntu 12.04 for linux32, linux64 and ASAN x64
  • Android 2.3 armv7 API 9

As we gather more SETA data for newer platforms, such as Android 4.3, we can implement SETA scheduling for it as well and reduce our test load.  We continue to run the full suite of tests on all platforms other branches other than m-i and fx-team, such as mozilla-central, try, and the beta and release branches. If we did miss a regression by reducing the tests, it would appear on other branches mozilla-central. We will continue to update our configs to incorporate SETA data as it changes.

How does SETA scheduling work?
We specify the tests that we would like to run on a reduced schedule in our buildbot configs.  For instance, this specifies that we would like to run these debug tests on every 10th commit or if we reach a timeout of 5400 seconds between tests.

http://hg.mozilla.org/build/buildbot-configs/file/2d9e77a87dfa/mozilla-tests/config_seta.py#l692


Previously, catlee had implemented a scheduling in buildbot that allowed us to coallesce jobs on a certain branch and platform using EveryNthScheduler.  However, as it was originally implemented, it didn't allow us to specify tests to skip, such as mochitest-3 debug on MacOSX 10.10 on mozilla-inbound.  It would only allow us to skip all the debug or opt tests for a certain platform and branch.

I modified misc.py to parse the configs and create a dictionary for each test specifying the interval at which the test should be skipped and the timeout interval.  If the tests has these parameters specified, it should be scheduled using the  EveryNthScheduler instead of the default scheduler.

http://hg.mozilla.org/build/buildbotcustom/file/728dc76b5ad0/misc.py#l2727
There are still some quirks to work out but I think it is working out well so far. I'll have some graphs in a future post on how this reduced our test load. 

Further reading
Joel Maher: SETA – Search for Extraneous Test Automation



Tanner FilipDo you host a wiki for your community? Community Ops wants to hear from you!

I'm cross-posting this to my blog, I'm hoping to get as much feedback as possible.

If you are hosting a wiki for your community rather than using wiki.mozilla.org, Community Ops has a few questions for you. If you would be so kind to reply to my post on Discourse, answering the questions I have below, we'd be extremely appreciative.

  1. How did you decide that you need a wiki?
  2. Why did you decide to host your own, rather than using the Mozilla Wiki?
  3. How did you choose your Wiki software (MediaWiki, TikiWiki, etc.)?
  4. What could make your wiki better? For example, would you like any extensions, or technical support?

Thank you in advance for taking the time to answer these questions!

Gervase MarkhamHSBC: Bad Security

I would like to use a stronger word than “bad” in the title, but decency forbids.

HSBC has, or used to have, a compulsory 2-factor system for logging in to their online banking. It used a small widget called a Secure Key. This is good. Now, they have rolled out an Android/iOS/Blackberry app alternative. This is also good, on balance.

However, at the same time, they have instituted a system where you can log on and see all your banking information and even take some actions without the key, just using a password. This is bad. Can I opt out, and say “no, I’d like to always use the key, please?” No, it seems I can’t. Compulsory lowered security for me. Even if I don’t use the password, that login mechanism will always be there.

OK, so I go to set a password. Never mind, I think, I’ll pick something long and complicated. But no; the guidance says:

Your password is not case sensitive and must be between 8 and 30 characters. It must include letters and numbers.

So the initial passphrase I picked was both too long, and didn’t include a number. However, the only error it gives is “This data is invalid”. I tried several other variants of my thought-of passphrase, but couldn’t get it to accept it. Painful reverse-engineering showed that the space character is also forbidden. Thank you so much, HSBC.

I finally find a password it’ll accept and click “Continue”. But, no. “Your session is invalidated – please log in again.” It’s taken so long to find a password it’ll accept that it has timed me out.

Air MozillaMartes mozilleros

Martes mozilleros Reunión bi-semanal para hablar sobre el estado de Mozilla, la comunidad y sus proyectos. Bi-weekly meeting to talk (in Spanish) about Mozilla status, community and...

Adam LoftingOptimizing for Growth

In my last post I spent some time talking about why we care about measuring retention rates, and tried to make the case that retention rate works as a meaningful measure of quality.

In this post I want to look at how a few key metrics for a product, business or service stack up when you combine them. This is an exercise for people who haven’t spent time thinking about these numbers before.

  • Traffic
  • Conversion
  • Retention
  • Referrals

If you’re used to thinking about product metrics, this won’t be new to you.

I built a simple tool to support this exercise. It’s not perfect, but in the spirit of ‘perfect is the enemy of good‘ I’ll share it in it’s current state.

>> Follow this link, and play with the numbers.

Optimizing for growth isn’t just ‘pouring’ bigger numbers into the top of the  ‘funnel‘. You need to get the right mix of results across all of these variables. And if your results for any of these measurable things are too low, your product will have a ‘ceiling’ for how many active users you can have at a single time.

However, if you succeed in optimizing your product or service against all four of these points you can find the kind of growth curve that the start-up world chases after every day. The referrals part in particular is important if you want to turn the ‘funnel’ into a ‘loop’.

Depending on your situation, improving each of these things has varying degrees of difficulty. But importantly they can all be measured, and as you make changes to the thing you are building you can see how your changes impact on each of these metrics. These are things you can optimize for.

But while you can optimize for these things, that doesn’t make it easy.

It still comes down to building things of real value and quality, and helping the right people find those things. And while there are tactics to tweak performance rates against each of these goals, the tactics alone won’t matter without the product being good too.

As an example, Dropbox increased their referral rate by rewarding users with extra storage space for referring their friends. But that tactic only works if people like Dropbox enough to (a) want extra storage space and (b) feel happy recommending the product to their friends.

In summary:

  • Build things of quality
  • Optimize them against these measurable goals

Marco ZeheAn update on office solutions in the browser and on mobile

Regular readers of my blog may remember my January 2014 shout out to Microsoft for implementing great accessibility in their Office Online offering. Later in the year, I also gave an overview over the accessibility in Google apps. Now, in late April 2015, it is time for an update, since both have made progress. We will also take a look at what has changed in Apple’s iCloud on the web suite, and I’ll introduce an open-source alternative that is ramping up to becoming more accessible.

Google apps

The Google apps suite for both free Gmail and paid Google Apps for Business accounts has grown enormously in functionality. And so have the accessibility features. No matter which are you look at, be it Docs with its wide variety of document navigation and collaboration features, Sheets with its ever more comprehensive spreadsheet editing and navigation abilities, or Slides, which allows a blind person to create full-featured slide shows. Gmail itself and Drive are also running very smoothly nowadays. Creating a Google Form to conduct surveys is also going very strongly.

One of the most remarkable facts about this is the enhancements to documentation the Google apps have received. Docs now has dedicated help pages for navigation, formatting, collaboration, and a huge list of keyboard shortcuts for all supported platforms. Take alone the Collaborating on a document with a screen reader, and just try a few things described in there with a friend, co-worker or family member. Each time I use these features, I am totally blown away by the experience.

Docs also introduced braille support and has by now expanded this to Firefox and screen readers as well. If you use it, you’ll get braille output (of course), but may lose some announcements that are available when braille support is not enabled. I have found that a combination of both modes works well for me: Non-braille mode for editing and collaboration, and braille mode for the final proof-reading.

The iOS apps have also made huge leaps forward. If you use an external keyboard with your iPhone or iPad, you have a similarly rich set of key strokes available to you that you have on the web. Compared to where these apps were a year ago, … Uh … forget it, there is no comparison. It’s like day and night!

On Android, the situation looks good as well, within, of course, the limitations that TalkBack still imposes on the users in general. Future versions may vastly improve this situation, let’s keep our fingers crossed! Until then, I suggest you look at the help documentation for Docs with Android.

Microsoft

Microsoft has also enhanced its accessibility features. Word Online, Excel Online, and PowerPoint Online work even better than when I wrote my first article. I found that the collaboration features don’t work as smoothly for me as they do with Google Docs, but because of OneDrive and Dropbox integrations, many tasks can be accomplished using the Office for Windows suite with all its features if the browser version falls short. The start page for accessibility in Office Online gives good pointers to articles with further information.

I also found that the Outlook.com web mailer behaves more like a web page than a real application in many instances. But of course, it has tight integration with Microsoft Outlook and Windows Mail in Windows, so again, if the web version falls short for you if you use these services, you can use the desktop versions.

The iOS versions also have seen great improveents for VoiceOver. The new kid on the block, Outlook for iOS, is getting frequent updates which usually also contain VoiceOver fixes.

And some good news for all the Mac enthusiasts out there: The Microsoft Office 2016 for Mac preview received an update on April 14, 2015 which, according to this support article, also contains VoiceOver improvements for Word, Excel, and PowerPoint. Outlook is also said to be accessible via a different support article on this.

I can’t say much about the Android versions of the Microsoft Office applications, and the official documentation channels haven’t revealed anything. If you have any experience, please let me know in the comments! Especially the MS Word for Android Tablet, and friends, are the interesting ones I think, since they are more feature-rich than the Office for Android phone app.

Apple

As much as Apple is great when it comes to accessibility in their hardware devices, including the latest new device category Apple Watch, as dismal is the situation with their iCloud.com offering. This thing just doesn’t have the fit and finish that the other products have. Yes, many buttons are now labeled, and yes, in Pages on the web, lines are read when you navigate them, as well as some other information. But the overall experience is not good. The keyboard focus gets lost every so often, and unpredictably, the interface is confusing and keeps stuff around that might, in a certain situation, not even be activable. This is nothing I can recommend to any screen reader user to use productively, even after some upgrades it received over the past year.

If you want, or have to, use iWork for iCloud, use the Mac or iOS versions. They work quite OK and get the job done.

Open-Xchange

And here’s the new kid on the block that I promised you! It’s called Open-Xchange App Suite, and is actually not quite as new in the field of collaborative tools. But its accessibility efforts are fairly new, and they look promising. Open-Xchange is mostly found in web mail providers such as the Germany-based Mailbox.org or 1&1, but also used internationally. Furthermore, anyone who runs certain types of Linux distributions as a server can run their own instance, with mail and cloud storage services. It also offers standards like IMAP and SMTP, CalDav for calendar sync, CardDav for contact sync, and WebDav for access to the files. It works in the MS Office formats, so is compatible with most, if not all, other office suites.

Its accessibility features on the web are on their way to becoming really good. They’ve still got some ways to go, primarily also in the way the keyboard focus handling works and how to get some tasks done really efficiently, but Mail, some parts of Calendar, Contacts, Drive, Text and the dashboard really work quite well already. It is nothing that compares yet to what Google is offering, but it comes close to the quality of what Microsoft is offering on the web, and definitely surpasses that in some areas.

This is definitely something to keep an eye on. I certainly will be watching its progress closely.

In summary

As of this writing, Google definitely comes out strongest in terms of accessibility and fit and finish when it comes to working efficiently with their apps. Granted, it takes some getting used to, and it requires that a screen reader user know their assistive technology and are willing to learn some keyboard shortcuts and familiarize themselves with certain usability concepts. But once that is mastered, Google Apps is definitely something I can whole-heartedly recommend for online collaboration. Furthermore, if you look at new features for commercial screen readers such as JAWS, you can see that they’re paying attention and improve their support for Google apps bit by bit where support is still lacking.

Microsoft is close behind, with some areas that are better accomplished in their desktop apps or on tablets rather than on the web.

Open-Xchange still has its bumps in the road, but is on a good way to becoming a viable alternative for those who can rely on their own infrastructure and want to go to a full open-source stack.

And for Apple, I recommend staying away from the web offering and doing all the office work in iWork apps for Mac or iOS. The web stuff is just too much of a hassle still.

Gervase MarkhamTop 50 DOS Problems Solved: Renaming Directories

Q: How do I rename a sub-directory? The only way I can find is to make a new one with the new name, copy all the files over from the old one, and then delete the original!

A: As you have found, the MS-DOS REN command doesn’t work on sub-directories. For a programmer it is a relatively trivial task to write a utility to do this job, and DR DOS 6 has a RENDIR command used in the same way as REN.

The manual for MS-DOS 5.0 advises the reader to do what you’re doing already, and indeed DR DOS 5 didn’t make provision for renaming directories. You can, however, use the DOS shell program to rename directories. If you want to stick with the command line, the best alternative is to get hold of a utility program written to do the job. Such programs are commonly found in shareware/PD catalogues.

Better think carefully before choosing that directory name…

Byron Joneshappy bmo push day!

the following changes have been pushed to bugzilla.mozilla.org:

  • [1157113] Argument “” isn’t numeric in numeric eq (==) at Bugzilla/Extension/TrackingFlags/Flag.pm line 406
  • [1157395] CSRF in log in form
  • [1149659] crash signature not linking to signature summary on socorro
  • [1153102] add hooks for bug bounty display and form
  • [1154026] Pre-fill Bounty Attachment form fields for e.g. date and reporter

discuss these changes on mozilla.tools.bmo.


Filed under: bmo, mozilla

Jeff WaldenTexas license plate oral argument

Yesterday I discussed offensive speech, especially relevant to the Texas license plate situation because the design’s being seen as “offensive” was Texas’s justification for denying Texas SCV‘s request. Today I talk a bit about oral argument.

Government speech

Texas hammered on its government-speech argument, but it generally didn’t get a very receptive audience. As Lyle Denniston observed, most of the justices’ questions and comments implicitly assumed there was a free speech issue to determine — which wouldn’t be the case if Texas SCV’s design were government speech. I doubt there’s more than a vote or two for these plates being government speech, if that.

One particularly gratifying discussion of government speech occurred when Texas’s advocate attempted to assert the specialty license plates were government speech because of the level of control, and Justice Kennedy noted the circularity of the argument. It was good to see practically the first problem I noticed in Texas’s argument, was also noticed on the bench. Chief Justice Roberts’s expression of skepticism about the program having no clear, identifiable policy being articulated, instead stating Texas was doing it for money (later joined by Justice Alito on the latter point), was also welcome.

A fair bit of time was spent discussing hypothetical “Vote Republican” and “Vote Democrat” license plates, and whether a state might approve one and deny the other. It’s not clear to me (nor was it clear to the justices) that “government speech” would directly prohibit this, but various “independent rules” were observed that would prevent such (just as such electioneering would be prohibited, somehow, in official ballots).

Forums

As I suggested earlier, Texas’s specialty plates program seems to be either a designated public forum or a limited public forum. Justice Kennedy picked up on this, asking if this was a case where Texas had opened “a new public forum in a new era”. Justice Alito posed multiple hypothetical questions involving government-established places where speech might occur, and the implication I drew from his comments suggested that he also viewed such cases as public forums. Each justice also presented hypothetical cases where the government set up a place for speech to happen (a billboard with a state message on it, with a small space for private speech to take place; officially-designated soapboxes in parks), then questioned whether it could be government speech or instead a public forum.

Justice Alito also probed the nature of the license plate forum if the state accepted only colleges, then colleges plus scenic places, and so on, gradually expanding into everyone. The point being: at what dividing line is a scheme no longer government speech? Texas SCV’s answer was that every state-designed plate would be government speech — but plates designed by private entities would be those entities’ speech.

“Offensive” speech

Various justices expressed concern that approving Texas’s denial might lead to regulation of offensiveness in other forums. Justice Ginsburg characterized the “might be offensive” standard as “nebulous” and granting too much discretion. Justice Kagan worried about approval of regulation of offense spreading into more and more forums, producing more and more regulation of speech.

The true fireworks for offensiveness, of course, arose when Texas SCV’s free-speech nut lawyer rose to defend their position. In essence he argued that once Texas extended an open invitation to anybody, they no longer could control what was said. Then, in response to successive questions, he argued Texas couldn’t prohibit license plates with swastikas, “jihad” (which he initially misheard as “vegan”, to laughter), “Make pot legal”, “BONG HiTS 4 JESUS” (more laughter, and a high point of the argument), and ultimately “the most offensive racial epithet that you can imagine”. Truly it was a glorious display of zeal for freedom of speech.

A banner reading "BONG HiTS 4 JESUS", displayed by Joseph Frederick and ultimately leading to the school speech case of Morse v. Frederick
A banner “reasonably viewed as promoting illegal drug use”, or “dumb advocacy”? But maybe protected speech, on a specialty license plate (picture by Mlschafer, public domain)

Selectivity

Various justices also made comments as to Texas’s non-selectivity in approving plates. Texas approved over 400 plates and rejected only around a dozen. Clearly several justices thought that near-blanket approval weakened any argument Texas might have for the state carefully exercising discretion in every instance, and strengthened the argument that they’d opened up a public forum for speech.

Those are some of the high points of argument. If you’re interested in more detail, see the transcript.

Next time, it’s probably on to a series wrapup. But no promises yet, as I haven’t written up enough thoughts to be certain. And again, as I noted yesterday, this might end up delayed a day or two. Til next time!

Armen Zambranomozci hackday - Friday May 1st, 2015

I recently blogged about mozci and I was gladly surprised that people have curiosity about it.

I want to spend Friday fixing some issues on the tool and I wonder if you would like to join me to learn more about it and help me fix some of them.

I will be available as armenzg_mozci from 9 to 5pm EDT on IRC (#ateam channel).
I'm happy to jump on Vidyo to give you a hand understanding mozci.

I hand picked some issues that I could get a hand with.
Documentation and definition of the project in readthedocs.


Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Mozilla Science LabTeaming Up: How to Build Your Open Science Collaboration

With the Mozilla Science Global Sprint coming up soon (submissions for project ideas are open until May 11), the community has been coming up with tons of awesome suggestions for projects to work on; meanwhile, new projects on everything from collaborative LaTeX to sleuthing out what references are open access on BMC to a curated bioinformatics reaction pathway database are cropping up on Collaborate, our project to highlight interesting open source and open science projects you can jump into, and learn by doing. But all of these projects have to answer the same question: how do you present an open source project in a way that appeals to people, gets them interested, and attracts collaborators who will stick around? Here are a few tips on what we’ve learned during the first six months of Collaborate:

On Collaborate

Collaborate compiles and curates open source, open science projects, in order to make those projects discoverable and connect them with potential contributors. But once someone finds their way to your project, it’s up to what you present there to convince them to jump in. A few tips for a strong listing:

  • Seek out users. Prolific and long-term volunteer contributors to open-source projects are often those who see themselves as the future users of that project. When describing your project, make sure to illustrate who you think the users will be, how you think they will use the project, and what benefit they’re going to get out of it. Your goal should be for a visitor to read your project summary and say, ‘I would totally use that – let’s make it happen!’
  • Offer some examples. Link to examples of the project in action, that help newcomers understand what the project is and get excited about being involved. If your project is such that it can be demoed on a website, all the better – if it’s not a web-oriented project, consider producing a blogpost that has figures or examples of your work thus far.
  • Describe a roadmap. The ‘Main Deliverables’ section of a Collaborate listing is there to let you describe what sort of high-level things collaborators will get to work on on your project. Don’t worry about nitty-gritty bugs or feature descriptions – that’s what your issue tracker is for. Rather, give a bird’s eye view of the direction you’re hoping to take the work in, and give people an idea of how they could fit into that.

On GitHub

Alright! People are excited about your listing on Collaborate, so they’ve joined your project and forked your repo. How can we turn that initial excitement into enough engagement to stick around and really dig in? A few more tips:

  • Talk to your collaborators. Collaborate will open an issue introducing new collaborators as they express interest in the site; try and respond to this issue within 24 hours. Making newcomers feel welcome is crucial at this stage; let them know you’re interested in working with them, and help them figure out where they can start contributing based on their skills and interests. Same goes for new issues opened – fast engagement signals a collaboration that cares about participation.
  • Have a good Getting Started guide. Confusion on what to do first can kill enthusiasm to participate in a project very quickly. Make sure the README.md of your project contains complete, current and correct instructions on how to set the project up, including any dependencies it may involve.
  • Use that Issue Tracker. The issue tracker is where you can start getting down to the details of what tasks need attention; use liberally, to get the plan out of your head and into a place where people can jump in. Also keep in mind, different issues are suitable for different collaborators; people new to the programming language or topic are best served by detailed, small issues that lay out an exact problem and perhaps a proposed solution, while experienced contributors will be more interested in big, open-ended problems that give them a chance to use their skills and imaginations. A good collaborative project should try to have a mix of both.

All these things point in the same direction – people participate in open projects when they see the value in them, identify personally with them, have a clear path to getting involved, and feel welcomed by the existing community. As always, we’ll help you polish your project listing and repo as it goes up on Collaborate and as we get ready for the Global Sprint. There is a huge and growing list of exciting projects to get involved with; we hope you’ll join us on June 4-5 for the Sprint, and anytime on Collaborate.

Mozilla Science LabMozilla Science Lab Week in Review, April 20-26

The Week in Review is our weekly roundup of what’s new in open science from the past week. If you have news or announcements you’d like passed on to the community, be sure to share on Twitter with @mozillascience and @billdoesphysics, or join our mailing list and get in touch there.

Blogs & Papers

Government & Policy

  • The European Commission released detailed findings & policy recommendations  from its survey late last year on the perceptions, expectations and challenges presented by open science, as understood by the scientific community there.
  • The US National Oceanic and Atmospheric Administration has announced a series of industry partnerships intended to make their environmental data widely available.
  • The NSF-funded Workshop on Supporting Scientific Discovery through Norms and Practices for Software and Data Citation and Attribution released its summary report, including recommendations and action plans surrounding the establishment of clear infrastructure and standards for citing both code and data as part of the scholarly record.

Projects & Resources

This Week In RustThis Week in Rust 79

Hello and welcome to another issue of This Week in Rust! Rust is a systems language pursuing the trifecta: safety, concurrency, and speed. This is a weekly summary of its progress and community. Want something mentioned? Send me an email! Want to get involved? We love contributions.

This Week in Rust is openly developed on GitHub. If you find any errors or omissions in this week's issue, please submit a PR.

What's cooking on master?

119 pull requests were merged in the last week, and 2 RFC PRs.

Now you can follow breaking changes as they happen!

Breaking Changes

Other Changes

New Contributors

  • Andrzej Janik
  • Geoffry Song
  • Heejong Ahn
  • John Van Enk
  • Krzysztof Drewniak
  • Lee Aronson
  • Michael Rosenberg
  • Przemysław Wesołek
  • Skyler
  • tynopex
  • Young Wu

Approved RFCs

New RFCs

Betawatch!

The current beta is 1.0.0-beta.3 (5241bf9c3 2015-04-25).

There were 2 PRs this week landing backports to beta.

Notable Links

Project Updates

Upcoming Events

If you are running a Rust event please add it to the calendar to get it mentioned here. Email Erick Tryzelaar or Brian Anderson for access.

Quote of the Week

< Ms2ger> And note, unsafe code isn't for violating Rust's invariants, it's for maintaining them manually

Ms2ger in #rust.

Thanks to bluss for the tip. Submit your quotes for next week!.

Robert Kaiser"Nothing to Hide"?

I've been bothered for quite a while with people telling me they "have nothing to hide anyhow" when the topic of Internet privacy comes up.

I guess that mostly comes from the impression that the whole story is our government watching (over) us and the worst thing that can happen is incrimination. While that might threaten some things, most people do nothing that is really interesting enough for a government to go into attack mode over it (or so they believe, and very firmly so). And I even agree that most governments (including the US and EU countries) actually actively seek out what they call "terrorist activities" (even though they often stretch that term in crazy ways) and/or child abuse and similar topics that the vast majority of citizens agree are a bad thing and are not part of - and the vast majority of politicians and government workers believe they act in the best interest of their citizens when "obviously fighting that" via their different programs of privacy-undermining surveillance. That said, most people seem to be OK with their government collecting data about them as long as it's not used to incriminate them (and when that happens, it's too late to protest the practice anyhow).

A lot has been said about that since the "Snowden leaks", but I think the more obvious short-term and direct threat is in corporate surveillance, which has been swept under the rug in most discussions recently - to the joy of Facebook, Google and other major players in that area. I have also seen that when depicting some obvious scenarios resulting of that, people start to think about it much more promptly and realize the effect on their daily lives (even if those are minor issues compared to government starting a manhunt against you with terror allegations or similar).

So what I start asking is:
  • Are you OK with banks determining your credit conditions based on all his comments on Facebook and his Google searches? ("Your friends say you owe them money, and that you live beyond your means, this is gonna be difficult...")
  • Are you OK with insurances changing your rates based on all that data? ("Oh, so you 'like' all those videos about dangerous sports and that deafening music, and you have some quite aggressive or even violent friends - so you see why we need to go a bit higher there, right?")
  • Are you OK with prices for flights or products in online stores (Amazon etc.) being different depending on what other things you have done on the web? ("So, you already planned that vacation at that location, good, so we can give you a higher air rate as you' can't back out now anyhow.")
  • And, of course, envision ads in public or half-public locations being customized for whoever is in the area. ("You recently searched for engagement rings, so we'll show ads for them wherever you go." or "Hey, this is the third time today we sat down and a screen nearby shows Viagra ads." or "My dear daughter, why do we see ads for diapers everywhere we go?")
There are probably more examples, those are the ones that came to my mind so far. Even if those are smaller things, people can relate to them as they affect things in their own life and not scenarios that feel very theoretical to them.

And, of course, they are true to a degree even now. Banks are already buying data from Facebook, probably including "private" messages, for determining credit scores, insurances base rates on anything they can find out about you, flight rates as well as prices for some Amazon and other web shop products vary based on what you searched before - and ads both on your screen and even on postal mail get tailored to a profile built on all kinds of your online behavior. My questions above just take all of those another step forward - but a pretty realistic one in my opinion.

I hope thinking about questions like that makes people realize they might actually want to evade some of that and in the end they actually have something to hide.

And then, of course, that a non-profit like Mozilla, which doesn't seek to maximize money, can believably be on their side and help them regain some privacy where they - now - want to.

Vaibhav AgrawalA-Team Contributions in 2015

It has been a while since I last blogged and I had something interesting to share so I finally managed to overcome my laziness. In this post, I would like to talk about some of the projects that I have been involved with in 2015 in the A-Team at Mozilla. Before I talk about the projects, I would like to give a shout out to @jmaher, @armenzg, @kmoir, @dminor, @ahal, @gbrown, @wlach who were always there to answer my questions, help me find a way to solve problems when I was stuck and review my patches. I have worked on some exciting problems and made some great progress on the projects in the past 4 months. They are:

1) SETA – Search for Extraneous Test Automation
We run hundreds of builds and thousands of test jobs each day on our integration branches, that is , mozilla-inbound and fx-team. And as more and more platforms are added every month, the load on test machines is ever increasing. But are so many test jobs for each push required? We run the test jobs to catch failures but majority of time, the test jobs pass and the ones who indeed catch failures often have duplicate ones. SETA tries to tackle this problem by being smart about utilizing machine cycles. In SETA, we find the minimum number of jobs that are needed to find all the failures that have occurred in the last six months on integration machines. From this data, we predict that these jobs will be more likely to catch failures than others and therefore other test jobs are set to run less frequently. It is true that we will be wrong certain number of times and when we miss a failure, the sheriffs would need to backfill some jobs to find the root cause. But most of the time, it will work. Joel has done an excellent blog post giving examples and statistics that has been done in this project. This project has been deployed in Mozilla Releng production systems and we have reduced the number of jobs to roughly 150-190 jobs/push from 350-400 jobs/push per day on desktop (linux, osx, win) platforms, a 50% reduction during high load weekdays. To put this into perspective, the past week we have seen the lowest jobs per push since January 1st on both mozilla-inbound and fx-team. I see this as a huge win as it drastically reduces the load on our machines as well as reduces the time the sheriffs need to star intermittents, increasing productivity for all. And this data is for desktop platforms only, android and other platforms are yet to come, after which we should be seeing more gains.

dailyjobs

jobs/push per week since January for mozilla-inbound. Lowest on April 20th – April 26th

Project Repo: https://github.com/dminor/ouija
Project Stats: http://alertmanager.allizom.org/dailyjobs.html
Project Information: http://alertmanager.allizom.org/seta.html

2) Mozilla CI Tools
MozCI (Mozilla CI Tools) is a python library which allows one to trigger builds and test jobs on treeherder.mozilla.org via command line. It has various use cases like triggering jobs for bisecting intermittent failures, backfilling missing jobs, and personally I use it for bisecting performance regressions. This tool is also used by sheriffs and is aimed for increasing developer productivity.
Project Repo: https://github.com/armenzg/mozilla_ci_tools
Project Documentation: https://mozilla-ci-tools.readthedocs.org/en/latest/index.html

3) Firefox Performance Sheriffing
In late January 2015, I took up the role of a performance sheriff. In this role, I look at the performance data produced by the test jobs and find regressions, root causes and get bugs on file to track issues and bring it to the attention of patch authors.
Sheriff documentation: https://wiki.mozilla.org/Buildbot/Talos/Sheriffing

I have also contributed patches to some other projects like removing android*.json from mochitests (bug 1083347), A-Team bootcamp and Mozregression. If you are looking to contribute to open source projects, I think this is a great time to start contributing to Automation and Tools team at Mozilla and make a big impact. For me, it has been one of the most productive quarters and I plan to keep contributing further. As some of you may know, this summer I will be joining as the A-Team intern at Mozilla in San Francisco with @chmanchester as my mentor, and I am looking forward to do more exciting work here!


Jeff WaldenOffensive speech

Yesterday I discussed First Amendment forum doctrine as it related to Texas’s specialty plate program. Today I discuss the subject where everyone learns to stop worrying and love censorship: “offensive” speech.

Trollface
If you reach the end of this post without having been offended at least once, I apologize for the error. (But I am serious about all the arguments made here, trolling for offense notwithstanding.)

“Offensive” speech

Texas denied Texas SCV‘s design because “the design might be offensive to any member of the public” (emphases added). A mere possibility of offense, to a single person, allows Texas (if it wants, as a matter of even-handed objectivity and fairness) to tell the plate requester to pound sand.

What does the First Amendment say about “offensive” speech? Offensive speech particularly is strongly protected. The First Amendment lets you victim-blame soldiers at their own funerals. It lets you burn the American flag, notwithstanding that this “seriously offended” several witnesses. It lets you satirically attribute false quotations to someone implying that his first sexual experience occurred in an outhouse with his own mother…after kicking out the goat. It protects the right to wear an undoubtedly-offensive jacket bearing the words “Fuck the Draft”. It protects a student newspaper cover depicting police officers raping the Statue of Liberty. It lets you falsely claim to have received the highest and most respected of military awards, in some sense diminishing that award’s value. The list goes on and on.

An American flag held, burning, touching the ground
Freedom of speech (picture apparently by noplur, CC-BY-SA-2.0)

The First Amendment was designed to protect offensive and unpopular speech, full stop. “If there is a bedrock principle underlying the First Amendment, it is that the government may not prohibit the expression of an idea simply because society finds the idea itself offensive or disagreeable.” “The fact that protected speech may be offensive to some does not justify its suppression.” As the Court stated regarding a more restrictive forum than the public sphere where license plates reside, “The mere dissemination of ideas — no matter how offensive to good taste — may not be shut off in the name alone of ‘conventions of decency.'”

In short, Texas is horribly wrong to think it has authority to censor offensive specialty plate designs based on “levels of offensiveness” (which, we must remember, was Texas’s post hoc rationale, not anything in the law).

Could the Confederate flag be “fighting words”?

Is some offensive speech nonetheless unprotected? Yes, but it must be fighting words that “by their very utterance, inflict injury or tend to incite an immediate breach of the peace.” No one here argues Texas law is really trying to criminalize fighting words, because doing so would run into (at least) two major problems.

First, Texas concedes that Texas SCV could display a bumper sticker with identical design on vehicles. But it’s irrational to think moving a hypothetically-inciting message a few inches would change that aspect of its nature. If a plate design met that standard, then so would a bumper sticker, and both could be forbidden.

Second, Texas’s “might be offensive to any member of the public” rule goes far beyond fighting words. “Choose Life” or “Respect Choice” specialty plates (or even “All Lives Matter”, which is clearly offensive and you should check your privilege, you racist member of the patriarchy) would offend some people. But it’s inconceivable (not meant that way) these could ever be considered fighting words, rather than protected political statements.

If Texas were to argue its “offensiveness” standard addressed fighting words (and it doesn’t), the statute would be substantially overbroad in criminalizing too much speech and accordingly vulnerable to challenge. Even Texas isn’t stupid enough to try to argue this.

What about “hate speech”?

The First Amendment protects hate speech. You can burn a cross on your own property without intent to intimidate, and it’s protected speech. You can hold a Nazi parade on public streets, even if Holocaust survivors say that “seeing the swastika was like being physically attacked.” You can make various forms of pornography that “subordinat[e] women” in a variety of reprehensible ways, and laws against such will be struck down.

A Neo-Nazi rally on the US Capitol grounds
I hate Illinois Nazis. But this is, rightly, protected speech. (original by David, CC-BY-2.0)

And the recent American kerfuffles over various fraternities’ obnoxious racist speech pretty much all constitute protected speech, for which the perpetrators have been widely judged protected from punishment by public universities. (The universities can get away with whatever punishment they want if the perpetrators won’t stand up for themselves in court. [An action which doesn’t prevent a sincere apology and work toward rehabilitation.] Fortunately, not all universities [#notalluniversities] [#butmost] have forgotten their duties under the First Amendment and their obligations to protect academic freedom even when it hurts.)

The imagined “hate speech” exception to the First Amendment, and to freedom of speech more generally, doesn’t exist.

I see. So you’re a racist bigot standing in the way of progress and social justice?

No. There are plenty of horrible forms of expression that nonetheless are protected by the First Amendment, and should be protected. I don’t need to agree with anything anyone says in order to defend their right to say it. (But by all means continue with the insults! I appreciate the encouragement.)

But consider the alternative to allowing all this speech. Suppose we really did think it was okay to punish some of this speech. What neutral arbitrator is going to decide which categories of speech are okay and which are not okay? Who’s going to neutrally enforce such laws in a non-discriminatory manner? There is no neutral arbitrator: rather, the majority will. And history shows that when a majority has the power to censor, it’ll censor things according to its own interests, not according to some “neutral” perspective able to judge what speech is and isn’t permissible.

Indeed, when the standard is that “offensive” speech is prohibited, aren’t minority views precisely those most likely to be deemed offensive? Inoffensive views aren’t the ones that need protection. Rather, it’s the offensive views that need protection, because majorities often aren’t inclined to protect offensive views. Nobody’s going to complain when someone says “Stop Child Abuse”, but they might about a discussion of then-offensive homosexual marriage.

Offensiveness and correctness aren’t always in agreement

Protecting offensive speech is particularly important because because offensive views may also be right. Martin Luther opening a scholarly debate among theologians about corrupt Church practices in the Ninety-Five Theses offended Johann Tetzel and the Church. (Whether or not Luther or the Church was right, I think we can generally agree the world is a better place for that debate having taken place.) Galileo’s astronomical hypotheses offended many astronomers of the day, and notoriously the Church. John Quincy Adams’s speech in bringing petitions “about” slavery before Congress, where Southern congressmen, fearful of what petitions on the topic of slavery might say, had previously voted in a rule to immediately table any petition on the subject, offended those congressmen. Paul Robert Cohen offended many people in protesting a draft later ended, attributable to some small degree to his speech about it. Archie Bunker epitomized bigotry, yet without it All in the Family couldn’t have made the points it made. (Query whether they could make that series today without being shouted down by the congenitally-stupid.)

It’s easier to judge people who say offensive things

Additionally, offensive speech permits the observer to judge people for what they’ve said. Censorship that prohibits expression of particular views also makes it harder to determine whether a person nonetheless holds them.

(It’s not a First Amendment question, but as a matter of the broader concept of freedom of speech, and the expression and discussion of ideas, I wonder if sometimes we take stigmatization too far. That is, we take it to the point where most adherents to an odious idea adhere in silence, with no way to recognize such people when it’s most important to be able to do so — for personal physical safety, for example. The apparent absence of bigots, when everyone agrees they still exist at least sometimes, can all too easily trigger paranoia. The consequent imputation of widespread bigotry without actual proof, in an overly-broad manner, serves no useful purpose.)

Moving on…

I could write more about this, but I’m well past the length of any prior post and running out of time to write more. I leave readers with a recommendation for further reading: the brief of the Cato Institute and others (most notably including the humorist P. J. O’Rourke and former ACLU president Nadine Strossen) in this case. Its discussion and examples are truly disgusting, an awful piece of work. And this is precisely what makes it so great.

Next time, various observations on oral argument. I still have writing to do here, so this might slip a day or two. Not clear. But right now I’m still aiming for it to go live tomorrow.

Mike ConleyThings I’ve Learned This Week (April 20 – April 24, 2015)

Short one this week. I must not have learned much! 😀

If you’re using Sublime Text to hack on Firefox or Gecko, make sure it’s not indexing your objdir.

Sublime has this wicked cool feature that lets you quickly search for files within your project folders. On my MBP, the shortcut is Cmd-P. It’s probably something like Ctrl-P on Windows and Linux.

That feature is awesome, because when I need to get to a file, instead of searching the folder hierarchy, I just hit Cmd-P, jam in a few of the characters (they can even be out of order – Sublime does fuzzy matching), and then as soon as my desired file is the top entry, just hit Enter, and BLAM – opened file. It really saves time!

At least, it saves time in theory. I noticed that sometimes, I’d hit Cmd-P, and the UI to enter my search string would take ages to show up. I had no idea why.

Then I noticed that this slowness seemed to show up after I had done a build. My objdir resides beneath my srcdir (as is the defaults with a mozilla-central checkout), so I figured perhaps Sublime was trying to index all of those binaries and choking on them.

I went to Project > Edit Project, and added this to the configuration file that opened:

{
    "folders":
    [
        {
            "path": "/Users/mikeconley/Projects/mozilla-central",
      "folder_exclude_patterns": ["*.sublime-workspace", "obj-*"]
        }
    ]
}

I added the workspace thing too1, because I figure it’s unlikely I’ll ever want to open that thing.

Anyhow, after setting that, I restarted Sublime, and everything was crazy-fast. \o/

If you’re using Sublime, and your objdir is under your srcdir, maybe consider adding the same thing. Even if you’re not using Cmd-P, it’ll probably save your machine from needlessly burning cycles indexing stuff.


  1. That’s where Sublime holds my session state for my project. 

Mike ConleyThe Joy of Coding (Ep. 11): Cleaning up the View Source Patch

For this episode, Richard Milewski and I figured out the syncing issue I’d been having in Episode 9, so I had my head floating in the bottom right corner while I hacked. Now you can see what I do with my face while hacking, if that’s a thing you had been interested in.

I’ve also started mirroring the episodes to YouTube, if YouTube is your choice platform for video consumption.

So, like last week, I was under a bit of time pressure because of a meeting scheduled for 2:30PM (actually the meeting I was supposed to have the week before – it just got postponed), so that gave me 1.5 hours to move forward with the View Source work we’d started back in Episode 8.

I started the episode by explaining that the cache key stuff we’d figured out in Episode 9 was really important, and that a bug had been filed by the Necko team to get the issue fixed. At the time of the video, there was a patch up for review in that bug, and when we applied it, we were able to retrieve source code out of the network cache after POST requests! Success!

Now that we had verified that our technique was going to work, I spent the rest of the episode cleaning up the patches we’d written. I started by doing a brief self-code-review to smoke out any glaring problems, and then started to fix those problems.

We got a good chunk of the way before I had to cut off the camera.

I know back when I started working on this particular bug, I had said that I wanted to take you through right to the end on camera – but the truth of the matter is, the priority of the bug went up, and I was moving too slowly on it, since I was restricting myself to a few hours on Wednesdays. So unfortunately, after my meeting, I went back to hacking on the bug off-camera, and yesterday I put up a patch for review. Here’s the review request, if you’re interested in seeing where I got to!

I felt good about the continuity experiment, and I think I’ll try it again for the next few episodes – but I think I’ll choose a lower-priority bug; that way, I think it’s more likely that I can keep the work contained within the episodes.

How did you feel about the continuity between episodes? Did it help to engage you, or did it not matter? I’d love to hear your comments!

Episode Agenda

References

Bug 1025146 – [e10s] Never load the source off of the network when viewing sourceNotes

Jeff WaldenFirst Amendment forum doctrine

Yesterday I discussed government speech, and Texas’s arguments that specialty plates are government speech that individuals can’t compel Texas to make. Today I discuss the First Amendment forum doctrine.

A reminder: yesterday’s disclaimer still holds. In fact this post is probably most likely to contain errors of all the posts in this series. Caveat lector!

Forums

The Supreme Court classifies places (physical and metaphorical) where the First Amendment applies, and freedom of speech requirements hold, in a few different ways. These places are called forums.

Modern ruins of the Roman Forum
One of the original forums, the Roman Forum: per Wikipedia, the center of public life in ancient Rome, and a venue for many public speeches (Carla Tavares, CC-BY-SA-3.0

Not a forum

Some places aren’t forums. For example, most public television programs aren’t forums. In such places government may exclude particular viewpoints, topics, and content.

If Texas ran its specialty plate program as a contest, picking a few choices from a vast selection according to its own whims as occurs on public TV, it might not be a forum. (That might also make it government speech.) (Or maybe I’m grasping for a plausible example, and I’ve grasped wrongly. Hedge!) But Texas accepts basically anybody, so it’s a forum.

Traditional public forums

Some forums have always been open places of discussion: traditional public forums. These include public sidewalks and parks. Here, no restrictions on content or viewpoint are permitted (except a very small list). Government may impose reasonable content/viewpoint-neutral restrictions as to speech’s time, place, or manner. But in doing so it must further significant government interests, and restrictions must be narrowly written and leave open ample alternative channels for speech. For example: no use outside park hours, potentially you need a permit granted nondiscriminatorily, don’t exceed noise level limits, stay off grass being reseeded.

Specialty plate programs are a modern invention, obviously not a common-law forum historically open for discussion.

Nonpublic forums

Places that are forums that aren’t traditionally open, that government hasn’t opened to public expression, are nonpublic forums. Examples are courthouses and government buildings: generally, government property reserved for some particular use. Government can employ restrictions on content here, to further those particular interests. Otherwise, restrictions match those in traditional public forums (and particularly can’t restrict speech based on its viewpoint).

Texas invited everyone onto its license plates, opening them to public expression. So Texas’s specialty plate program is not a nonpublic forum.

Designated public forums (and limited public forums)

Finally we have locations not traditionally opened, that government has opened to some public expression: designated public forums. The restrictions applicable in traditional public forums are also applicable in designated public forums.

There’s also an additional sort of forum, sometimes viewed as a subset of the designed public forum, where the government limits speech to certain groups or topics: the limited public forum. A limited public forum may place restrictions on content, but it can’t restrict based on viewpoint.

The various forum definitions logically subdivide the set of all possible forums. (Although to be sure, there remains argument as to what subdivision the Court’s cases have actually recognized.) By process of elimination, Texas’s program must be a designated public forum, possibly a limited public forum.

Recap

Let’s check our work. License plate designs are not traditionally open to ideas, but Texas allows anyone to propose a custom plate design espousing practically any idea whatsoever. The ideas are legion: in-state and out-of-state colleges (even bitter rivals), college sports teams, and professional teams; corporations; non-profits; causes; war veteran status; the whimsical (“Rather Be Golfing”); and many others. The individual selects a specialty plate and displays it publicly. Moreover, Wooley v. Maynard indicates that individuals have First Amendment rights in what their license plates say. So Texas’s specialty plates are public expression, at the government’s invitation.

Texas’s specialty-plate program is a designated public forum, possibly a limited public forum. Hold the question as to which one — we’ll return to this later.

Tomorrow, a discussion of offensive speech.

Gervase MarkhamThree Forms of Diversity

I’ve been thinking about the concept of “diversity” recently, and have written some thoughts on the subject, called Three Forms of Diversity. (The title is a play on Three Forms of Unity.)

The piece does include, at the end, a section on the specific applicability of my analysis to the Mozilla community.

Comments, as always, are most welcome. :-)

Cameron KaiserAchievement unlocked: lack of any social or romantic activity PAYS OFF

*with two tests suppressed due to big endian

[4484|   0|   0|   0] 100% ==========================================>|1511.7s
PASSED ALL

Yaaaaaay! It's time to get jiggy with it! First, I'll scrub the changesets to remove PPCBC/JM entirely (except for the PowerPC-specific parallel type guards in Baseline inline caches, which are an important optimization); the only other vestiges that shall remain are a couple code snippets from Ben's original work in the new MacroAssembler and the G3/G4 software square root routine from David, which may be replaced with a call to the C library; I'm going to do some profiling there first. (G5 still uses its hardware square root instruction.) After that we'll sprinkle back in gcc 4.6 compatibility, the last branch we will support with this compiler, mix with IonPower, and then try to get the browser up. If I can get the minimp3 seek routine working, MP3 support will be exposed publicly too. We will also unveil our own bespoke browser front page to avoid issues with Electrolysis, which we don't support currently.

In addition, I have some lofty but possible performance goals for IonPower. For these goals, I've selected V8 as the metric because it's a medium-weight benchmark; SunSpider is probably not representative anymore of the bloatier code many sites (especially ad networks, bleah) now foist upon their users, and it is unlikely many Power Macs are braving the really heavy processing JavaScript apps for time reasons (or support reasons: games using WebGL, for example, won't run on TenFourFox anyway) which makes Kraken and Octane less representative as well. The aim is to get the optimized IonPower faster than PPCBC on V8 in Baseline-only mode, and faster than our old JaegerMonkey backend on V8 in full Baseline/Ion mode, which it now supports. The first is definitely achievable, and the second should come awfully close. That should translate into demonstrable real-world improvements on the vast majority of sites vintage Macs still frequent and will also make pdf.js finally viable on TenFourFox.

I treat every ESR release like it's our last, but by the time 38ESR ends, that will mean Power Macs will have had a fully supported branch of Firefox for 10 years after the last Power Mac rolled out of Cupertino. Now, that's support!

ETA for the first beta is 3-4 weeks, with rollout around 38.0.2 to coincide with the end of support for 31ESR. Localizers, stand by!

Armen ZambranoWhat Mozilla CI tools is and what it can do for you (aka mozci)

Mozci (Mozilla CI tools) is a python library, scripts and package which allows you to trigger jobs on treeherder.mozilla.org.
Not all jobs can be triggered but those that are run on Release Engineering's Buildbot setup. Most (if not all) Firefox desktop and Firefox for Android jobs can be triggered. I believe some B2G jobs can still be triggered.

NOTE: Most B2G jobs are not supported yet since they run on TaskCluster. Support for it will be given on this quarter.

Using it

Once you check out the code:
git clone https://github.com/armenzg/mozilla_ci_tools.git
python setup.py develop
you can run scripts like this one (click here for other scripts):
python scripts/trigger.py \
  --buildername "Rev5 MacOSX Yosemite 10.10 fx-team talos dromaeojs" \
  --rev e16054134e12 --times 10
which would trigger a specific job 10 times.

NOTE: This is independent if a build job exist to trigger the test job. mozci will trigger everything which is required to get you what you need.

One of the many other options is if you want to trigger the same job for the last X revisions, this would require you to use --back-revisions X.

There are many use cases and options listed in here.


A use case for developers

One use case which could be useful to developers (thanks @mike_conley!) is if you pushed to try and used this try syntax: "try: -b o -p win32 -u mochitests -t none". Unfortunately, you later determine that you really need this one: "try: -b o -p linux64,macosx64,win32 -u reftest,mochitests -t none".

In normal circumstances you would go and push again to the try server, however, with mozci (once someone implements this), we could simply pass the new syntax to a script (or with ./mach) and trigger everything that you need rather than having to push again and waster resources and your time!

If you have other use cases, please file an issue in here.

If you want to read about the definition of the project, vision, use cases or FAQ please visit the documentation.


Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Jeff WaldenGovernment speech and compelled speech

Yesterday I discussed specialty plate programs in lower courts and the parties’ arguments in Walker v. Texas Division, Sons of Confederate Veterans. Today I begin to analyze the questions in the case.

But first, a disclaimer.

Disclaimer

The following is my understanding of First Amendment law, gleaned from years of reading numerous free speech opinions, summaries, and analyses. I’m generally confident in this explanation, but I may well have made mistakes, or simply missed nuance present in the cases but not in the summaries I’ve read. Please point out mistakes in the comments.

Of course, I really have no business trying to explain First Amendment jurisprudence, if I want it explained correctly. First Amendment law is incredibly complex. My haphazard reading will miss things.

But I’m barging ahead anyway, for a few reasons. First, I want to talk about this. Second, it’s fun to talk about it! Third, you don’t learn unless you’re willing to look like a fool from time to time. Fourth, the law is not this recondite, bizarre arcana that only lawyers and judges can understand. It may require some work to correctly understand laws, terms of art, rules of statutory construction, and relevant past decisions in the common law. But any intelligent person can do it if they make the effort.

And fifth, nobody with any sense will unconditionally rely on this as authoritative, not when there are far better places to look for the finest in free Internet legal advice.

Government speech

The “recently minted” government speech doctrine occupies an uneasy place in the realm of speech. For when government speech occurs, non-governmental speech open to First Amendment challenge is reduced. There must be some government speech: otherwise we’d absurdly conclude that the government’s World War II war-bond propaganda must be accompanied by anti-bond propaganda. Government programs often have viewpoints suppressible only in the voting booth. But this mechanism is sluggish and imperfectly responsive, and government speech’s discretion can be abused. So it’s best to be careful anointing government speech.


This is your government. This is your government on beef. Any questions?

Certainly some license plates — the state’s default designs and designs ordered by the legislature — are government speech, even if they’re also individual speech under Wooley v. Maynard. In each case the government wholly chooses what it wishes to say, and that message is government speech. The individual’s choice to assist in conveying it, under Wooley, isn’t government speech.

Circularity

But Texas’s government-speech argument, applied beyond plates it designs itself, is laughable. The linchpin of Texas’s argument is that because they control the program, that makes it government speech they can control. This argument is completely circular! By starting from their control over the program’s speech, they’ve assumed their conclusion.

This doesn’t mean Texas is wrong. But their circular central government-speech argument can prove nothing. This logical flaw is blindingly obvious. Texas’s lawyers can’t have missed this. If they made this their lead argument, they’re scrambling.

Compelling Texas to speak?

Texas’s better argument is that vehicle licenses and plates are its program, implicating its right to speak or not speak under Wooley. But the First Amendment restrains government power, not individual power. And many courts (although so far not the Supreme Court) have held that government can be compelled to “speak” in accepting advertising in government-controlled places (public transit systems, for a common example). The problem is Texas voluntarily created a specialty plate program open to all for speech. No “compulsion” derives from a voluntary act.

Texas didn’t fully control the specialty plate program, but rather opened it to anyone with money. (As Chief Justice Roberts noted in oral argument: “They’re only doing this to get the money.”) It’s possible there’s government speech in Texas SCV‘s plate, perhaps the occasionally-proposed “hybrid” speech. But once Texas opens the program to all, it loses full control over what’s said.

How then do we consider specialty plate programs? What controls may Texas exercise? Now we must decide how to classify the specialty-plate program with respect to First Amendment-protected speech. What sort of forum for speech is Texas’s specialty-plate program?

Tomorrow, First Amendment forum doctrine.

Daniel Stenbergcurl on the NASDAQ tower

Apigee posted this lovely picture over at twitter. A curl command line on the NASDAQ tower.

curl-nasdaq-cropped

Armen ZambranoFirefox UI update testing

We currently trigger manually UI update tests for Firefox releases. There are automated headless update verification tests but they don't test the UI of Firefox.

The goal is to integrate this UI update testing as part of the Firefox releases.
This will require changes to firefox-ui-tests, buildbot scheduling changes, Marionette changes and other Mozbase packages. The ultimate goal is to speed up our turn around on releases.

The update testing code was recently ported from Mozmill to use Marionette to drive the testing.

I've already written some documentation on how to run the update verification using Release Engineering configuration files. You can use my tools repository until the code lands (update_testing is the branch to be used).

My deliverable is to ensure that the update testing works reliably on Release Engineering infrastructure and there is existing scheduling code for it.

You can read more about this project in bug 1148546.


Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Pierros PapadeasKPI Dashboard on reps.mozilla.org

Mozilla Reps as a program is full of activities. Reps around the world, do extraordinary  things everyday, promoting Mozilla’s mission and getting new contributors on board.

Moving forward, trying to identify how those activities align with top-tier initiatives, Mozilla Reps program wanted a way to visualize some Key Progress Indicators around the program.

We (the Participation Infrastructure team) sat down with the programmatic owners of Reps (Nuke & Rosana) and identified what numbers and metrics we would like to expose in a much more digestible way, so we can assess the progress of the program on many levels.

We identified 3 different KPIs:

  • Number of Reps (and growth rates)
  • Number of Events (and growth rates)
  • Number of Reports (and growth rates)

… and also 3 different filters you can apply on those numbers:

  • Country
  • Functional Area (of Mozilla)
  • Initiative (associated with Rep, Event or Report)

You can find the spec for this initial iteration here.

We decided to have the filters as drop-downs, applied on the whole page (in combination or one-by-one). Then for each KPI group we would have a time graph for the past 6 weeks (fixed for now) with a HUD of basic numbers and growth rates.

Screenshot from 2015-04-24 13:39:29Technology-wise, we tied the coding of this new dashboard to the delivery of a proper API for the Reps Portal (more info on that soon). The new API enabled us to easily create custom endpoints to calculate the numbers needed for our Reps KPI graphs (based on the existing Conversion Points). Nemo and Tasos did a fantastic work to deliver the new API and the custom endpoints, while making sure that this is not heavy on our DB.

Nikos then worked on the front-end using D3.js as the visualization library to create the graphs dynamically (each time you access the page or you filter using Country, Area or Initiative).

Screenshot from 2015-04-24 13:37:05The overall result is smooth and easily helps you assess progress of various Areas and Initiatives on specific Countries, for Reps, Events and Reports.

You can check out the dashboard here.

Next step would be to introduce a time-slider for customizing the time range you want to be displayed.

Chris LordWeb Navigation Transitions

Wow, so it’s been over a year since I last blogged. Lots has happened in that time, but I suppose that’s a subject for another post. I’d like to write a bit about something I’ve been working on for the last week or so. You may have seen Google’s proposal for navigation transitions, and if not, I suggest reading the spec and watching the demonstration. This is something that I’ve thought about for a while previously, but never put into words. After reading Google’s proposal, I fear that it’s quite complex both to implement and to author, so this pushed me both to document my idea, and to implement a proof-of-concept.

I think Google’s proposal is based on Android’s Activity Transitions, and due to Android UI’s very different display model, I don’t think this maps well to the web. Just my opinion though, and I’d be interested in hearing peoples’ thoughts. What follows is my alternative proposal. If you like, you can just jump straight to a demo, or view the source. Note that the demo currently only works in Gecko-based browsers – this is mostly because I suck, but also because other browsers have slightly inscrutable behaviour when it comes to adding stylesheets to a document. This is likely fixable, patches are most welcome.


 Navigation Transitions specification proposal

Abstract

An API will be suggested that will allow transitions to be performed between page navigations, requiring only CSS. It is intended for the API to be flexible enough to allow for animations on different pages to be performed in synchronisation, and for particular transition state to be selected on without it being necessary to interject with JavaScript.

Proposed API

Navigation transitions will be specified within a specialised stylesheet. These stylesheets will be included in the document as new link rel types. Transitions can be specified for entering and exiting the document. When the document is ready to transition, these stylesheets will be applied for the specified duration, after which they will stop applying.

Example syntax:

<link rel="transition-enter" duration="0.25s" href="URI" />
<link rel="transition-exit" duration="0.25s" href="URI" />

When navigating to a new page, the current page’s ‘transition-exit‘ stylesheet will be referenced, and the new page’s ‘transition-enter‘ stylesheet will be referenced.

When navigation is operating in a backwards direction, by the user pressing the back button in browser chrome, or when initiated from JavaScript via manipulation of the location or history objects, animations will be run in reverse. That is, the current page’s ‘transition-enter‘ stylesheet will be referenced, and animations will run in reverse, and the old page’s ‘transition-exit‘ stylesheet will be referenced, and those animations also run in reverse.

[Update]

Anne van Kesteren suggests that forcing this to be a separate stylesheet and putting the duration information in the tag is not desirable, and that it would be nicer to expose this as a media query, with the duration information available in an @-rule. Something like this:

@viewport {
  navigate-away-duration: 500ms;
}

@media (navigate-away) {
  ...
}

I think this would indeed be nicer, though I think the exact naming might need some work.

Transitioning

When a navigation is initiated, the old page will stay at its current position and the new page will be overlaid over the old page, but hidden. Once the new page has finished loading it will be unhidden, the old page’s ‘transition-exit‘ stylesheet will be applied and the new page’s ‘transition-enter’ stylesheet will be applied, for the specified durations of each stylesheet.

When navigating backwards, the CSS animations timeline will be reversed. This will have the effect of modifying the meaning of animation-direction like so:

Forwards          | Backwards
--------------------------------------
normal            | reverse
reverse           | normal
alternate         | alternate-reverse
alternate-reverse | alternate

and this will also alter the start time of the animation, depending on the declared total duration of the transition. For example, if a navigation stylesheet is declared to last 0.5s and an animation has a duration of 0.25s, when navigating backwards, that animation will effectively have an animation-delay of 0.25s and run in reverse. Similarly, if it already had an animation-delay of 0.1s, the animation-delay going backwards would become 0.15s, to reflect the time when the animation would have ended.

Layer ordering will also be reversed when navigating backwards, that is, the page being navigated from will appear on top of the page being navigated backwards to.

Signals

When a transition starts, a ‘navigation-transition-startNavigationTransitionEvent will be fired on the destination page. When this event is fired, the document will have had the applicable stylesheet applied and it will be visible, but will not yet have been painted on the screen since the stylesheet was applied. When the navigation transition duration is met, a ‘navigation-transition-end‘ will be fired on the destination page. These signals can be used, amongst other things, to tidy up state and to initialise state. They can also be used to modify the DOM before the transition begins, allowing for customising the transition based on request data.

JavaScript execution could potentially cause a navigation transition to run indefinitely, it is left to the user agent’s general purpose JavaScript hang detection to mitigate this circumstance.

Considerations and limitations

Navigation transitions will not be applied if the new page does not finish loading within 1.5 seconds of its first paint. This can be mitigated by pre-loading documents, or by the use of service workers.

Stylesheet application duration will be timed from the first render after the stylesheets are applied. This should either synchronise exactly with CSS animation/transition timing, or it should be longer, but it should never be shorter.

Authors should be aware that using transitions will temporarily increase the memory footprint of their application during transitions. This can be mitigated by clear separation of UI and data, and/or by using JavaScript to manipulate the document and state when navigating to avoid keeping unused resources alive.

Navigation transitions will only be applied if both the navigating document has an exit transition and the target document has an enter transition. Similarly, when navigating backwards, the navigating document must have an enter transition and the target document must have an exit transition. Both documents must be on the same origin, or transitions will not apply. The exception to these rules is the first document load of the navigator. In this case, the enter transition will apply if all prior considerations are met.

Default transitions

It is possible for the user agent to specify default transitions, so that navigation within a particular origin will always include navigation transitions unless they are explicitly disabled by that origin. This can be done by specifying navigation transition stylesheets with no href attribute, or that have an empty href attribute.

Note that specifying default transitions in all situations may not be desirable due to the differing loading characteristics of pages on the web at large.

It is suggested that default transition stylesheets may be specified by extending the iframe element with custom ‘default-transition-enter‘ and ‘default-transition-exit‘ attributes.

Examples

Simple slide between two pages:

[page-1.html]

<head>
  <link rel="transition-exit" duration="0.25s" href="page-1-exit.css" />
  <style>
    body {
      border: 0;
      height: 100%;
    }

    #bg {
      width: 100%;
      height: 100%;
      background-color: red;
    }
  </style>
</head>
<body>
  <div id="bg" onclick="window.location='page-2.html'"></div>
</body>

[page-1-exit.css]

#bg {
  animation-name: slide-left;
  animation-duration: 0.25s;
}

@keyframes slide-left {
  from {}
  to { transform: translateX(-100%); }
}

[page-2.html]

<head>
  <link rel="transition-enter" duration="0.25s" href="page-2-enter.css" />
  <style>
    body {
      border: 0;
      height: 100%;
    }

    #bg {
      width: 100%;
      height: 100%;
      background-color: green;
    }
  </style>
</head>
<body>
  <div id="bg" onclick="history.back()"></div>
</body>

[page-2-enter.css]

#bg {
  animation-name: slide-from-left;
  animation-duration: 0.25s;
}

@keyframes slide-from-left {
  from { transform: translateX(100%) }
  to {}
}


I believe that this proposal is easier to understand and use for simpler transitions than Google’s, however it becomes harder to express animations where one element is transitioning to a new position/size in a new page, and it’s also impossible to interleave contents between the two pages (as the pages will always draw separately, in the predefined order). I don’t believe this last limitation is a big issue, however, and I don’t think the cognitive load required to craft such a transition is considerably higher. In fact, you can see it demonstrated by visiting this link in a Gecko-based browser (recommended viewing in responsive design mode Ctrl+Shift+m).

I would love to hear peoples’ thoughts on this. Am I actually just totally wrong, and Google’s proposal is superior? Are there huge limitations in this proposal that I’ve not considered? Are there security implications I’ve not considered? It’s highly likely that parts of all of these are true and I’d love to hear why. You can view the source for the examples in your browser’s developer tools, but if you’d like a way to check it out more easily and suggest changes, you can also view the git source repository.

Cameron KaiserIonPower progress report

Remember: comparing the G5 optimized PPCBC Baseline-only compiler against the unoptimized test version of IonPower on V8!

% /Applications/TenFourFoxG5.app/Contents/MacOS/js --no-ion -f run.js
Richards: 203
DeltaBlue: 582
Crypto: 358
RayTrace: 584
EarleyBoyer: 595
RegExp: 616
Splay: 969
NavierStokes: 432
----
Score (version 7): 498

% ../../../../mozilla-36t/obj-ff-dbg/dist/bin/js -f run.js
Richards: 337
DeltaBlue: 948
Crypto: 1083
RayTrace: 913
EarleyBoyer: 350
RegExp: 259
Splay: 584
NavierStokes: 3262
----
Score (version 7): 695

I've got one failing test case left to go (the other is not expected to pass because it assumes a little-endian memory alignment)! We're almost to the TenFourFox 38 port!

The Rust Programming Language BlogRust Once, Run Everywhere

Rust’s quest for world domination was never destined to happen overnight, so Rust needs to be able to interoperate with the existing world just as easily as it talks to itself. For this reason, Rust makes it easy to communicate with C APIs without overhead, and to leverage its ownership system to provide much stronger safety guarantees for those APIs at the same time.

To communicate with other languages, Rust provides a foreign function interface (FFI). Following Rust’s design principles, the FFI provides a zero-cost abstraction where function calls between Rust and C have identical performance to C function calls. FFI bindings can also leverage language features such as ownership and borrowing to provide a safe interface that enforces protocols around pointers and other resources. These protocols usually appear only in the documentation for C APIs – at best – but Rust makes them explicit.

In this post we’ll explore how to encapsulate unsafe FFI calls to C in safe, zero-cost abstractions. Working with C is, however, just an example; we’ll also see how Rust can easily talk to languages like Python and Ruby just as seamlessly as with C.

Rust talking to C

Let’s start with a simple example of calling C code from Rust and then demonstrate that Rust imposes no additional overhead. Here’s a C program which will simply double all the input it’s given:

int double_input(int input) {
    return input * 2;
}

To call this from Rust, you might write a program like this:

extern crate libc;

extern {
    fn double_input(input: libc::c_int) -> libc::c_int;
}

fn main() {
    let input = 4;
    let output = unsafe { double_input(input) };
    println!("{} * 2 = {}", input, output);
}

And that’s it! You can try this out for yourself by checking out the code on GitHub and running cargo run from that directory. At the source level we can see that there’s no burden in calling an external function beyond stating its signature, and we’ll see soon that the generated code indeed has no overhead, either. There are, however, a few subtle aspects of this Rust program, so let’s cover each piece in detail.

First up we see extern crate libc. The libc crate provides many useful type definitions for FFI bindings when talking with C, and it makes it easy to ensure that both C and Rust agree on the types crossing the language boundary.

This leads us nicely into the next part of the program:

extern {
    fn double_input(input: libc::c_int) -> libc::c_int;
}

In Rust this is a declaration of an externally available function. You can think of this along the lines of a C header file. Here’s where the compiler learns about the inputs and outputs of the function, and you can see above that this matches our definition in C. Next up we have the main body of the program:

fn main() {
    let input = 4;
    let output = unsafe { double_input(input) };
    println!("{} * 2 = {}", input, output);
}

We see one of the crucial aspects of FFI in Rust here, the unsafe block. The compiler knows nothing about the implementation of double_input, so it must assume that memory unsafety could happen whenever you call a foreign function. The unsafe block is how the programmer takes responsibility for ensuring safety – you are promising that the actual call you make will not, in fact, violate memory safety, and thus that Rust’s basic guarantees are upheld. This may seem limiting, but Rust has just the right set of tools to allow consumers to not worry about unsafe (more on this in a moment).

Now that we’ve seen how to call a C function from Rust, let’s see if we can verify this claim of zero overhead. Almost all programming languages can call into C one way or another, but it often comes at a cost with runtime type conversions or perhaps some language-runtime juggling. To get a handle on what Rust is doing, let’s go straight to the assembly code of the above main function’s call to double_input:

mov    $0x4,%edi
callq  3bc30 <double_input>

And as before, that’s it! Here we can see that calling a C function from Rust involves precisely one call instruction after moving the arguments into place, exactly the same cost as it would be in C.

Safe Abstractions

Most features in Rust tie into its core concept of ownership, and the FFI is no exception. When binding a C library in Rust you not only have the benefit of zero overhead, but you are also able to make it safer than C can! Bindings can leverage the ownership and borrowing principles in Rust to codify comments typically found in a C header about how its API should be used.

For example, consider a C library for parsing a tarball. This library will expose functions to read the contents of each file in the tarball, probably something along the lines of:

// Gets the data for a file in the tarball at the given index, returning NULL if
// it does not exist. The `size` pointer is filled in with the size of the file
// if successful.
const char *tarball_file_data(tarball_t *tarball, unsigned index, size_t *size);

This function is implicitly making assumptions about how it can be used, however, by assuming that the char* pointer returned cannot outlive the input tarball. When bound in Rust, this API might look like this instead:

pub struct Tarball { raw: *mut tarball_t }

impl Tarball {
    pub fn file(&self, index: u32) -> Option<&[u8]> {
        unsafe {
            let mut size = 0;
            let data = tarball_file_data(self.raw, index as libc::c_uint,
                                         &mut size);
            if data.is_null() {
                None
            } else {
                Some(slice::from_raw_parts(data as *const u8, size as usize))
            }
        }
    }
}

Here the *mut tarball_t pointer is owned by a Tarball, which is responsible for any destruction and cleanup, so we already have rich knowledge about the lifetime of the tarball’s memory. Additionally, the file method returns a borrowed slice whose lifetime is implicitly connected to the lifetime of the source tarball itself (the &self argument). This is Rust’s way of indicating that the returned slice can only be used within the lifetime of the tarball, statically preventing dangling pointer bugs that are easy to make when working directly with C. (If you’re not familiar with this kind of borrowing in Rust, have a look at Yehuda Katz’s blog post on ownership.)

A key aspect of the Rust binding here is that it is a safe function, meaning that callers do not have to use unsafe blocks to invoke it! Although it has an unsafe implementation (due to calling an FFI function), the interface uses borrowing to guarantee that no memory unsafety can occur in any Rust code that uses it. That is, due to Rust’s static checking, it’s simply not possible to cause a segfault using the API on the Rust side. And don’t forget, all of this is coming at zero cost: the raw types in C are representable in Rust with no extra allocations or overhead.

Rust’s amazing community has already built some substantial safe bindings around existing C libraries, including OpenSSL, libgit2, libdispatch, libcurl, sdl2, Unix APIs, and libsodium. This list is also growing quite rapidly on crates.io, so your favorite C library may already be bound or will be bound soon!

C talking to Rust

Despite guaranteeing memory safety, Rust does not have a garbage collector or runtime, and one of the benefits of this is that Rust code can be called from C with no setup at all. This means that the zero overhead FFI not only applies when Rust calls into C, but also when C calls into Rust!

Let’s take the example above, but reverse the roles of each language. As before, all the code below is available on GitHub. First we’ll start off with our Rust code:

#[no_mangle]
pub extern fn double_input(input: i32) -> i32 {
    input * 2
}

As with the Rust code before, there’s not a whole lot here but there are some subtle aspects in play. First off, we’ve labeled our function definition with a #[no_mangle] attribute. This instructs the compiler to not mangle the symbol name for the function double_input. Rust employs name mangling similar to C++ to ensure that libraries do not clash with one another, and this attribute means that you don’t have to guess a symbol name like double_input::h485dee7f568bebafeaa from C.

Next we’ve got our function definition, and the most interesting part about this is the keyword extern. This is a specialized form of specifying the ABI for a function which enables the function to be compatible with a C function call.

Finally, if you take a look at the Cargo.toml you’ll see that this library is not compiled as a normal Rust library (rlib) but instead as a static archive which Rust calls a ‘staticlib’. This enables all the relevant Rust code to be linked statically into the C program we’re about to produce.

Now that we’ve got our Rust library squared away, let’s write our C program which will call Rust.

#include <stdint.h>
#include <stdio.h>

extern int32_t double_input(int32_t input);

int main() {
    int input = 4;
    int output = double_input(input);
    printf("%d * 2 = %d\n", input, output);
    return 0;
}

Here we can see that C, like Rust, needs to declare the double_input function that Rust defined. Other than that though everything is ready to go! If you run make from the directory on GitHub you’ll see these examples getting compiled and linked together and the final executable should run and print 4 * 2 = 8.

Rust’s lack of a garbage collector and runtime enables this seamless transition from C to Rust. The external C code does not need to perform any setup on Rust’s behalf, making the transition that much cheaper.

Beyond C

Up to now we’ve seen how FFI in Rust has zero overhead and how we can use Rust’s concept of ownership to write safe bindings to C libraries. If you’re not using C, however, you’re still in luck! These features of Rust enable it to also be called from Python, Ruby, JavaScript, and many more languages.

When writing code in these languages, you sometimes want to speed up some component that’s performance critical, but in the past this often required dropping all the way to C, and thereby giving up the memory safety, high-level abstractions, and ergonomics of these languages.

The fact that Rust can talk to easily with C, however, means that it is also viable for this sort of usage. One of Rust’s first production users, Skylight, was able to improve the performance and memory usage of their data collection agent almost instantly by just using Rust, and the Rust code is all published as a Ruby gem.

Moving from a language like Python and Ruby down to C to optimize performance is often quite difficult as it’s tough to ensure that the program won’t crash in a difficult-to-debug way. Rust, however, not only brings zero cost FFI, but also makes it possible to retain the same safety guarantees as the original source language. In the long run, this should make it much easier for programmers in these languages to drop down and do some systems programming to squeeze out critical performance when they need it.

FFI is just one of many tools in the toolbox of Rust, but it’s a key component to Rust’s adoption as it allows Rust to seamlessly integrate with existing code bases today. I’m personally quite excited to see the benefits of Rust reach as many projects as possible!

Emma IrwinMy year on Reps Council

It’s been one year! An incredible year of learning, leading and helping evolve the Mozilla Reps program as a council member. As my term ends I want to share my experiences with those considering this same path, but also as a way to lend to the greater narrative of Reps as a leadership platform.

I could write easily write 12 posts to cover the experience –  but  I thought this might be more helpful:

The 7 things I know for sure

(after 12 months on Reps Council)

1. Mozilla Reps Council Is a journey of learning and inspiration

When I decided to run for council last year, it was with an assumption my workload would  consist of mostly administrative tasks (although to be truthful that is a bulk of our work).  I also thought my previous leadership experiences would allow me to ‘help out’ as needed.

It turns out,  I had a lot to learn and improve on, especially during my two months as council chair. Here are some of the new and sharpened skills I am emerging with as a result:

  • Problem solving
  • Conflict Resolution/ Crisis Management
  • Communication
  • Strategy
  • Transparency
  • Project Planning
  • Task Management
  • Writing
  • Respecting Work-Life Balance
  • Debating Respectfully
  • Public Speaking
  • Facilitation
  • The art of saying ‘no’/when to step back
  • The art of ‘not dropping balls’ or knowing which balls will bounce back, and which will break
  • Being brave (aka stating opinions to leadership and with a nagging imposter syndrome)
  • Empathy
  • Planning for Diversity
  • Outreach
  • Teaching
  • Mentorship

2. 2015 is a (super) important year for Reps

Nurtured by the loving hands of 5 previous Reps councils, a strong mentorship structure and over 400 Reps and thousands of community members the Mozilla Reps program has come to an important milestone as a recognized body of leadership across Mozilla.  The  clearly articulated vision of Reps as a ‘launch pad for leadership’ has pushed us to be more  strategic in our goals.  And we are.  The next council together with mentors will be critical in executing these goals.

3. The voice of community is valued, and Mozilla is listening

In the past few months, we’ve worked with Mitchell Baker, Chris Beard, Mark Surman and David Slater, Mary-Ellen and others on everything from conflict resolution, to VP interview and on-boarding processes. Reps Council is on the Mozilla leadership page. The Mozilla Reps call has been attended by Firefox and Brand teams in need of feedback.  It’s not a coincidence, and it’s not casual – your voice matters.  Reps as leaders have the ear of the entire organization, because Reps are the voice of their extended community.

2015-04-23_1942

If you are one of those quiet people in our community with great ideas, passion and an interest in developing leadership skills – please consider running for council this year.

4. Mozilla Reps is  ever-evolving

View post on imgur.com

When I joined Reps Council, I had a lot of ideas about what would would ‘fix’.  And I laugh at myself for that bit of naivety  – ‘fixing’ is something we do to flaws, to errors and mistakes – but the Reps program is not a completed vision  – it’s a living organism, it’s alive with people, their ideas, inventions and actions.  How we evolve, while aligning with the needs of project goals, is a bit like changing the tire on a moving car .   If you are considering a run for council, it might help to envision ways you can evolve, improve and grow the program as it shifts, and in response to project & community vision for their own participation goals.

 5. Changing minds is hard / Outreach matters

I can’t write a list like this without acknowledging a my personal challenge of recognizing and trying to change ‘perception problems’.  It was strange to move from what had been a fairly easy transition between community, Rep and mentor to Reps council where almost suddenly –  I was regarded as part of a bureaucratic structure. I didn’t see or feel that from my fellow council members who work very hard on behalf of community –  and it’s been important to me that we change that perception through outreach.

Perceptions of our extended community have also been challenging – the idea that Reps is somehow isolated or a special  contributor group is contrary to the leadership platform we are really building.

Slowly we are changing minds, slowly outreach is making a difference – I am happy and optimistic about this.

 6.  Diversity Matters  Reps is an incredibly diverse community with diverse representation in many areas including age, geography and experience. Few other communities can compare .  But,  like much of the technology world we struggle with the representation of women in our council, and mentorship base.  To be truly reflective of our community, and our world – to have the benefit of all perspectives we need to encourage women leaders.  As I leave council, my hope is that we will continue to prioritize women in leadership roles.

7. Our community rocks  Brilliant, creative, energetic, passionate, motivated, friends and second family.  The heart of what we do, lies here.

To the Reps community, mentors, the Reps team, Mozilla leadership and community I thank you for this incredible opportunity to contribute and to grow.  I plan to pay it forward.

2015-04-23_1946

Feature Image Credit:  Fay Tandog

 

 

Air MozillaPrivacy Lab and Cryptoparty with guest speaker Melanie Ensign - How Security/Crypto Experts Can Communicate with Non-Technical Audiences

Privacy Lab and Cryptoparty with guest speaker Melanie Ensign - How Security/Crypto Experts Can Communicate with Non-Technical Audiences Our April Privacy Lab will include a speaker and an optional and free Cryptoparty, hosted by Wildbee (https://wildbee.org/cryptoparty.html). Our speaker will be Melanie Ensign. Melanie's...

L. David BaronThoughts on migrating to a secure Web

Brad Hill asked what I and other candidates in the TAG election think of Tim Berners-Lee's article Web Security - "HTTPS Everywhere" harmful. The question seems worth answering, and I don't think an answer fits within a tweet. So this is what I think, even though I feel the topic is a bit outside my area of expertise:

  • The current path of switching content on the Web to being accessed through secure connections generally involves making the content available via http URLs also available via https URLs, redirecting http URLs to https ones, and (hopefully, although not all that frequently in reality) using HSTS to ensure that the user's future attempts to access HTTP resources get converted to HTTPS without any insecure connection being made. This is a bit hacky, and hasn't solved the problem of the initial insecure connection, but it mostly works, and doesn't degrade the security of anything we have today (e.g., bookmarks or links to https URLs).

  • It's not clear to me what the problem that Tim is trying to solve is. I think some of it is concern over the semantic Web (e.g., his concern over the “identity of the resource”), although there may be other concerns there that I don't understand. I'd tend to prioritize the interests of the browseable Web (with users counted in the billions) and other uses of the Web that are widespread, over those of the semantic Web.

  • There are good reasons for the partitioning that browsers do between http and https:

    • Some of the partitioning prevents attacks directly (for example, sending a cookie that should be sent only to an https site to its http equivalent could allow an active attacker to steal the information in that cookie). Likewise for many other attacks involving the same-origin policy, where http and https are considered different origins.
    • Some of it (e.g., identifying https pages that load resources over http as insecure) is intended to prevent large classes of mistakes that would otherwise be widespread and drastically reduce the security of the Web. Circa 2000, a common Web developer complaint about browser security UI was that a site couldn't be considered secure if an image was loaded over HTTP. This might have been fine if the image was the company logo (and the attack under consideration was avoiding theft of money or credentials rather than avoiding monitoring), but isn't fine if the image is a graph of a bank account balance or if the image's URL has authentication information in it. (On the other hand, if it were a script rather than an image, an active attacker could compromise the entire page if the script could be loaded without authentication.) I think a similar rationale applies for not having mechanisms to do authentication without encryption (even though there are many cases where that would be fine).

    It's not clear to me how Tim's proposal of making http secure would address these issues (and keep everything else working at the same time). For example, is a secure-http page same-origin with insecure-http on the same host, or with https, or neither? They may well be solvable, but I don't see how to solve them off the top of my head, and I think they'd need to be solved before actually pursuing this approach.

  • One problem that I think is worth solving is that HTTPS as a user-presentable prefix has largely failed. Banks tell their customers to go to links like "bofa.com/activate" or "wellsfargo.com/activate". (The first one doesn't even work if the user adds "https://". I guess there's a chance that the experience of existing users could be fixed with HSTS, but that's not the case today.) They do this for a good reason; each additional character (especially the strange characters) is going to reduce the chance the user succeeds at the task.

    It's possible Tim's proposal might help solve this, although it's not clear to me how it could do so with an active man-in-the-middle attacker. (It could help against passive attackers, as could browsers trying https before trying http.) In the long term, maybe the Web will get to a point where typing such URLs tries https and doesn't try http, but I think we're a long way away from a browser being able to do that without losing a large percentage of its users.

I think I basically understand the current approach of migrating to secure connections by migrating to https, which seems to be working, although slowly. I'm hopeful that Let's Encrypt will help speed this up. It's possible that the approach Tim is suggesting could lead to a faster migration to secure connections on the Web, although I don't see enough in Tim's article to evaluate its security and feasibility.