Planet Mozilla Research

Today featured: the mystery of the overlapping node IDs -- which was causing yesterday's unsolved problem. Due to my get-around-the-borrow-checker hack, I was copying the session in a way that didn't actually copy all the state. So I was resetting the unique-ID counter to zero after doing the initial parsing, expansion, and injection of various primitives. This caused the obvious problems. Once I fixed that, I ran into the driver... not generating output when I called it from rustpkg on a library. That's bad. Haven't figured out why yet.

Unrelatedly, I've been trying to bisect a performance regression that I noticed in the compiler, though I'm not sure anyone else is seeing it (others have commented on a memory regression, but I'm noticing a time regression too). The first round of bisecting pointed to the commit from 5 days ago that updated the snapshot. So that suggests a performance problem that had been in stage2 and then snuck into stage1 when the snapshot got updated. So now I'm doing a second round of bisecting to figure out when the stage2 problem showed up, which is much slower.

And when both of those things are compiling, fixing warnings, which is almost done.

comments

May 24, 2013 02:29 AM

A frustrating day. On the plus side, I did solve yesterday's problem with the duplicate __test identifier. It turned out I was calling the driver functions wrong and it was literally synthesizing a test module twice. I was able to fix that bug.

Once I fixed that, I got this horrible resolve error:

<intrinsic>:58:37: 58:41 error: found value name used as a type: def_fn({crate: 1, node: 160381}, impure_fn)
<intrinsic>:58         fn visit_estr_uniq(&self) -> bool;

The error refers to the code for intrinsics (primops, basically) that rustc injects everywhere. It's complaining that bool is a function but is being used like a type. But... bool is a type! A primitive one, at that. I'd expect this to happen if there was a function somewhere called bool (maybe), but there isn't.

This was when I was trying to use rustpkg to build a test module. So next I decided to just compile a regular main module. So then I got:

rust: task failed at 'ty_fn_ret() called on non-fn type: &ty_nil', /Users/tjc/rust2/src/librustc/middle/ty.rs:2822

as an ICE in trans.

Given that all of this just comes from changing search paths so rustpkg can find libstd and not treat it as a remote-package-to-download, I'm... flummoxed.

Also, rustc as a whole seemed to get a lot slower sometime in the past two days. As in, even running three builds in different workspaces, at the same time, resulted in a longer than usual wait for any one of them to complete. And I wasn't running out of physical RAM.

Because of that, I spent some time while waiting for compiles to finish nominating bugs. I sorted github issues by least-recently-commented-on. As a result, all the open issues have been touched sometime in the past 2 months, and there are now 142 nominated bugs. I... don't think we'll get through all of them in the triage meeting tomorrow.

comments

May 23, 2013 03:13 AM

Still working on #5681, external crate inference. I resolved one of the remaining problems, which was how to not try to download-and-build std in directives like extern mod std; (where std is obviously already installed). But now, much to my dismay, I'm seeing some strange rustc behavior:

<core-macros>:1:0: 1:0 error: duplicate definition of type `__test`
<core-macros>:1 pub mod macros {
                ^
<core-macros>:1:0: 1:0 note: first definition of type __test here:
<core-macros>:1 pub mod macros {

This is only when compiling a file with a #[test] in it. What's weird is I haven't changed anything affecting how I interface with rustc. I'm not sure where the duplicate definitions of __test (the name that gets used for the automatically-created test module) are coming from, nor do I have any clear idea how to debug this.

Otherwise, today was bug triage and cleaning up warnings and stuff, in the moments when rustpkg was recompiling (of which there were many, since the compiler seems to have gotten quite a bit slower over the long weekend).

Better luck tomorrow, maybe...

comments

May 22, 2013 01:36 AM

This isn't Rust-related, but I wanted to report that I spent much of this evening developing a revised version of my Ionmonkey patch to implement generator syntax changes for Harmony, and finally submitted the patch for review, after a 9-month break since the last time I worked on this.

I want to emphasize that I only worked on the syntax changes and haven't touched any related semantics changes at all. Why did it take so long? Well, mostly because I spent most of the time not working on it. When faced with urgent work on Rust (inside my comfort zone) or JavaScript work (outside my comfort zone), it's pretty obvious what I'll pick! But also, it's just hard to work on a compiler for a language that I don't know (JavaScript) that's also implemented in a language I don't know well (C++).

Anyway, for the first time in 9 months I can actually relax and wait for a review, instead of feeling like I should work on this the next time I have a free moment and enough energy for work! I'll enjoy the feeling while it lasts.

comments

May 20, 2013 04:03 AM

I more-or-less finished external crate inference, at least in the local-crate case. I spent most of the time on that fighting with the borrow checker. To add the right directories into the session's list of additional library search path, I wrote some visitor code that looks at all the view_items in the crate. There are two types of view_items: extern mod directives and use directives. For this, I only care about extern mod directives.

The problem is that the visitor wants an @-closure, which can't use borrowed pointers, so accumulating a mutable vector of found extern mod thing seems... hard. I eventually gave up and worked around it with massive amounts of copying.

Once I get some tidy stuff cleaned up, and figure out the copy issue, I'll be ready to check that in and close #5681.

If I can finish #6407 (allowing extern mod directives to name remote packages) and #5683 (the test runner for rustpkg, which I've already started a rudimentary version of) tomorrow, then I'll be on-schedule so far. It might happen.

comments

May 17, 2013 01:10 AM

Mozilla is engaged in a broad, deep conversation about Internet privacy. We believe in putting users in control of their online experience, and we want a healthy, thriving web ecosystem — we do not see a contradiction. However, sometimes a crucial experiment is required to prove it.

To this end, we are testing a patch from Jonathan Mayer. Jonathan’s patch matches how Safari has worked for years, and does the following:

• Allows cookies from sites you have already visited.
• Blocks cookies from sites you have not visited yet.

The idea is that if you have not visited a site (including the one to which you are navigating currently) and it wants to put a cookie on your computer, the site is likely not one you have heard of or have any relationship with. But this is only likely, not always true. Two problems arise:

False positives. For example, say you visit a site named foo.com, which embeds cookie-setting content from a site named foocdn.com. With the patch, Firefox sets cookies from foo.com because you visited it, yet blocks cookies from foocdn.com because you never visited foocdn.com directly, even though there is actually just one company behind both sites.

False negatives. Meanwhile, in the other direction, just because you visit a site once does not mean you are ok with it tracking you all over the Internet on unrelated sites, forever more. Suppose you click on an ad by accident, for example. Or a site you trust directly starts setting third-party cookies you do not want.

Our challenge is to find a way to address these sorts of cases. We are looking for more granularity than deciding automatically and exclusively based upon whether you visit a site or not, although that is often a good place to start the decision process.

We plan to ship an evolution of the patch “on” by default, but we want to make refinements first. To make sure we get this right we need more data. Our next engineering task is to add privacy-preserving code to measure how the patch affects real websites. We will also ask some of our Aurora and Beta users to opt-in to a study with deeper data collection.

There are many conflicting claims about how this patch will affect the Internet. Why debate in theory what we can measure in practice? We are going to find out more and adjust course as needed. This is the essence of the release test cycle.

On Tuesday we did two things:

1. The patch has progressed to the Beta release channel for Firefox 22, but it is not “on” by default there. This allows more people to test the patch via Firefox’s “preferences” (AKA “options”) user interface, and avoids an abrupt change for site owners while we work on handling the hard cases.
2. The patch remains in the Aurora channel for Firefox, where it is “on” by default. This gives the patch better ongoing test coverage and facilitates A/B testing.

We have heard important feedback from concerned site owners. We are always committed to user privacy, and remain committed to shipping a version of the patch that is “on” by default. We are mindful that this is an important change; we always knew it would take a little longer than most patches as we put it through its paces.

For those who read this as Mozilla softening our stance on protecting privacy and putting users first, in a word: no. False positives break sites that users intentionally visit. (Fortunately, we haven’t seen too many such problems, but greater testing scale is needed.) False negatives enable tracking where it is not wanted. The patch as-is needs more work.

We look forward to continued dialog with colleagues, contributors, fans, and detractors. We will update all of you within six weeks so you can understand our thinking and how we will proceed. Comments welcome.

/be

P.S. Cookies (name history) were originally intended to be ephemeral (Windows 3.1 had so little usable memory with its 64K memory segments that Netscape’s founders had no choice). At first, they held only session state that could be recovered from the server by logging in again.

(Remind me to tell the story some day of Montulli’s aborted “twinkies” idea from the Netscape 2 era. UPDATE: Lou has published a new blog post about cookies.)

How far we have come in the amazing, living system that is the Web! No one planned for what actually happened, but with more work on the cookie policy in Firefox and (I hope) other browsers, I believe that we can evolve to a better space.

May 16, 2013 07:11 AM

The remote-package-ID patch landed, so I'm on to teaching rustpkg how to find external crates.

Currently, if you include an extern mod directive at the top of a Rust file, like:

extern mod foo;

supposing foo lives in some arbitrary (non-system) directory like /home/tjc/quux/lib, you have to pass in a -L flag to rustc, like: -L /home/tjc/quux. That's so it knows where to find the foo library to link with.

But in rustpkg we're trying to make it possible for files to self-describe, and get away from compiler flags as much as possible. So rustpkg will, itself, scan files in the crate it's trying to build for extern mods and try to build their dependencies automatically. So far, it only looks in the same workspace that the package it's building is in, since it only knows about one workspace (the current directory). But once I implement RUST_PATH, that'll be a lot more powerful.

Even more so, the plan is to change the syntax of extern mod directives so they don't have to just name identifiers. So you'll be able to write:

extern mod foo = "github.com/catamorphism/foo";

Eventually, this will cause rustpkg to try to fetch the foo package from my repository on github and install it. The package can be referred to in the code with the short name foo, since the entire URL fragment isn't a valid Rust identifier.

So far, I've implemented the part where rustpkg scans for extern mods and builds the named packages and dependencies if possible. I'm working on getting it to pass the right directories to the rustc driver so as to actually find the resulting libraries.

comments

May 16, 2013 02:33 AM

Back once again from another (shorter) absence -- I had to go down to Olympia to say goodbye to my 11-year-old rabbit, who'd had a stroke and needed to be euthanized. This spring, I'm telling you...

This morning, Ron Garcia came over to the office from UBC to visit us and hear about Rust. It's always good to get a bit of external validation, and it seems like he and our team have at least some research interests in common.

After lunch, I got down to hacking, having previously gotten somewhat derailed by adding a few more tests to #6418, the URL-like package-ID pull request. I hit the "computing fictitious type" ICE that a lot of other people had hit, and so I figured it was time for me to fix it, since I knew it probably related to my recent changes to do with derived errors. Turned out the ICE has to do with erroneous matches on patterns of the form (foo, bar) where the scrutinee doesn't have a tuple type -- this is a common error because of, among other reasons, argument patterns where it should really be &(foo, bar). So I fixed that, but that was all over the weekend.

Today, that patch having landed, I went back to finishing #6418. Partly this required resolving merge conflicts, due to the intervening make-vectors-not-implicitly-copyable change which made PkgIds non-implicitly copyable. Partly, I fixed the ugly hack I was using where output file names would be of the form foo-0-1 instead of foo-0.1, where 0.1 is the version number. I was doing that so that rustc wouldn't drop the portion after the last dot in order to make the output .o filename. But it turns out the correct output file to pass is just foo, since for libraries, rustc pulls the version number out of the metadata (not the filename) anyway.

And so I can get this checked in before I leave the office tonight, I hope (which is a bit of a race against time, since all the bathrooms in the office are out of order).

comments

May 15, 2013 01:03 AM

I finished yesterday's work, in the "tests pass" sense, so now rustpkg can recognize package IDs that resemble URL fragments. For example, I can write:

rustpkg build github.com/catamorphism/test-pkg

and rustpkg will fetch the test-pkg repository from github, then build a local cached copy of it.

We also had a useful meeting today to discuss what Servo needs from rustpkg, resulting in ten or so new github issues. (The rustpkg metabug collects them all.) One thing we talked about will actually obviate the need for the tedious string-munging that I talked about in yesterday's post. Specifically, we'll change the syntax of the extern mod directive -- which says "link this external crate with this program" -- to take a string rather than an identifier. So instead of:

extern mod foo;

we'll allow:

extern mod "foo";

or

extern mod "github.com/catamorphism/foo";

But in the latter case, how will I refer to foo in my code, since the package ID here is not an identifier? The Rust compiler will either guess the package's short name from the package ID's file stem if it's a valid identifier; in other cases, it will support this syntax too:

extern mod foo = "github.com/catamorphism/foo-bar";

that gives a package with arbitrary characters in its package ID a valid Rust identifier.

Not rocket science, but incremental progress towards the dream of writing the Rust build system in Rust.

comments

May 11, 2013 01:41 AM

After taking three days off to fly to Europe, attend a funeral, and fly back, then some time to catch up on the inevitable email and bug-triage backlog, I'm back to working on remote package IDs for rustpkg.

On the plane over the weekend, I was working on getting rustpkg to accept package IDs that resemble URLs. For example: rustpkg build github.com/catamorphism/test-pkg. So, not really a URL, but a URL fragment. rustpkg should do the obvious thing and use git to clone the remote repo into a local directory, then build the local cached copy as usual.

You'd think this wouldn't be hard, but I ran into issues because internally, rustpkg normalizes hyphens into underscores so that all package IDs can be legal Rust identifiers. (Rust identifiers can't contain dashes, but can contain underscores.) But for a remote package, you still need to keep track of the original package ID since that dictates the URL you need to fetch from. I solved this by adding two fields, a local path and a remote path, into rustpkg's internal representation of package IDs, but I'm still not sure this is the right solution. It beats calling a normalize function in an ad hoc way everywhere, as I was doing before, though.

Then the second thing is being consistent about which build artifacts contain the version number as part of their name (for example, the library name does, but the installed executable doesn't...) though I'm not sure why this wasn't coming up before.

And, doubtless, more than that. Details, details...

comments

May 10, 2013 12:28 AM

Among the build systems peer, I am very much a newcomer. Despite working with Thunderbird for over 5 years, I've only grown to understand the comm-central build system in gory detail in the past year. Most of my work before then was in trying to get random projects working; understanding it more thoroughly is a result of attempting to eliminate the need for comm-central to maintain its own build system. The goal of moving our build system description to a series of moz.build files has made this task much more urgent.

At a high level, the core of the comm-central build system is not a single build system but rather three copies of the same build system. In simple terms, there's a matrix on two accesses: which repository does the configuration of code (whose config.status invokes it), and which repository does the build (whose rules.mk is used). Most code is configured and built by mozilla-central. That comm-central code which is linked into libxul is configured by mozilla-central but built by comm-central. tier_app is configured and built by comm-central. This matrix of possibilities causes interesting bugs—like the bustage caused by the XPIDLSRCS migration, or issues I'm facing working with xpcshell manifests—but it is probably possible to make all code get configured by mozilla-central and eliminate several issues for once and all.

With that in mind, here is a step-by-step look at how the amorphous monster that is the comm-central build system works:

python client.py checkout

And comm-central starts with a step that is unknown in mozilla-central. Back when everyone was in CVS, the process of building started with "check out client.mk from the server, set up your mozconfig, and then run make -f client.mk checkout." The checkout would download exactly the directories needed to build the program you were trying to build. When mozilla-central moved to Mercurial, the only external projects in the tree that Firefox used were NSPR and NSS, both of which were set up to pull from a specific revision. The decision was made to import NSPR and NSS as snapshots on a regular basis, so there was no need for the everyday user to use this feature. Thunderbird, on the other hand, pulled in the LDAP code externally, as well as mozilla-central, while SeaMonkey also pulls in the DOM inspector, Venkman, and Chatzilla as extensions. Importing a snapshot was not a tenable option for mozilla-central, as it updates at an aggressive rate, so the functionality of checkout was ported to comm-central in a replacement python fashion.

./configure [comm-central]

The general purpose of configure is to discover the build system and enable or disable components based on options specified by the user. This results in a long list of variables which is read in by the build system. Recently, I changed the script to eliminate the need to port changes from mozilla-central. Instead, this script reads in a few variables and tweaks them slightly to produce a modified command line to call mozilla-central's configure...

./configure [mozilla-central]

... which does all the hard work. There are hooks in the configure script here to run a few extra commands for comm-central's need (primarily adding a few variables and configuring LDAP). This is done by running a bit of m4 over another file and invoking that as a shell script; the m4 is largely to make it look and feel "like" autoconf. At the end of the line, this dumps out all of the variables to a file called config.status; how these get configured in the script is not interesting.

./config.status [mozilla/comm-central]

But config.status is. At this point, we enter the mozbuild world and things become very insane; failure to appreciate what goes on here is a very good way to cause extended bustage for comm-central. The mozbuild code essentially starts at a directory and exhaustively walks it to build a map of all the code. One of the tasks of comm-central's configure is to alert mozbuild to the fact that some of our files use a different build system. We, however, also carefully hide some of our files from mozbuild, so we run another copy of config.status again to add in some more files (tier_app, in short). This results in our code having two different notions of how our build system is split, and was never designed that way. Originally, mozilla-central had no knowledge of the existence of comm-central, but some changes made in the Firefox 4 timeframe suddenly required Thunderbird and SeaMonkey to link all of the mailnews code into libxul, which forced this contorted process to come about.

make

Now that all of the Makefiles have bee generated, building can begin. The first directory entered is the top of comm-central, which proceeds to immediately make all of mozilla-central. How mozilla-central builds itself is perhaps an interesting discussion, but not for this article. The important part is that partway through building, mozilla-central will be told to make ../mailnews (or one of the other few directories). Under recursive make, the only way to "tell" which build system is being used is by the directory that the $(DEPTH) variable is pointing to, since $(DEPTH)/config/config.mk and $(DEPTH)/config/rules/mk are the files included to get the necessary rules. Since mozbuild was informed very early on that comm-central is special, the variables it points to in comm-central are different from those in mozilla-central—and thus comm-central's files are all invariably built with the comm-central build system despite being built from mozilla-central.

However, this is not true of all files. Some of the files, like the chat directory are never mentioned to mozilla-central. Thus, after the comm-central top-level build completes building mozilla-central, it proceeds to do a full build under what it thinks is the complete build system. It is here that later hacks to get things like xpcshell tests working correctly are done. Glossed over in this discussion is the fun process of tiers and other dependency voodoo tricks for a recursive make.

The future

With all of the changes going on, this guide is going to become obsolete quickly. I'm experimenting with eliminating one of our three build system clones by making all comm-central code get configured by mozilla-central, so that mozbuild gets a truly global view of what's going on—which would help not break comm-central for things like eliminating the master xpcshell manifest, or compiling IDLs in parallel. The long-term goal, of course, is to eliminate the ersatz comm-central build system altogether, although the final setup of how that build system works out is still not fully clear, as I'm still in the phase of "get it working when I symlink everything everywhere."

May 08, 2013 01:46 AM

This morning, Mozilla and OTOY made an announcement:

Mozilla and OTOY deliver the power of native PC applications to the Web, unveil next generation JavaScript video codec for movies and cloud gaming

What this means:

ORBX.js, a downloadable HD codec written in JS and WebGL. The advantages are many. On the good-for-the-open-web side: no encumbered-format burden on web browsers, they are just IP-blind runtimes. Technical wins start with the ability to evolve and improve the codec over time, instead of taking ten years to specify and burn it into silicon.

After these come more wins: 25% better compression than H.264 for competitive quality, adaptive bit-rate while streaming, integer and (soon) floating point coding, better color depth, better intra-frame coding, a more parallelizable design — the list goes on.

The GPU cloud has your back. Think of the amazing 3D games that we have on PCs, consoles, and handheld devices thanks to the GPU. Now think of hundreds of GPUs in the cloud, working for you to over-detail, ray/path-trace in realtime, encode video, do arbitrary (GPGPU) computation.

Or consider high-powered tools from Autodesk, Adobe, and others for 3D modeling and rendering:

Native apps from any popular OS, in the GPU cloud and on your browser. Yes, both: this is not just remote desktop tech, or X11 reborn via JS. Many local/remote hybrid computation schemes are at hand today, e.g. a game can do near-field computing in the browser on a beefy client while offloading lower LOD work to the GPU cloud.

OTOY’s CEO Jules Urbach demo’ed an entire Mac OS X desktop running in a cloud VM sandbox, rendering via ORBX.js to Firefox, but also showed a Windows homescreen running on his Mac — and the system tray, start menu, and app icons were all local HTML5/JS (apps were a mix ranging from mostly local to fully remoted, each in its own cloud sandbox).

Valve’s Steam was one such app:

Watermarking, not DRM. This could be huge. OTOY’s GPU cloud approach enables individually watermarking every intra-frame, and according to some of its Hollywood supporters including Ari Emanuel, this may be enough to eliminate the need for DRM.

We shall see; I am hopeful. This kind of per-user watermarking has been prohibitively expensive, but OTOY estimates the cost at pennies per movie with their approach.

Oculus Rift, Lightfield displays, Holodecks, and beyond. OTOY works with Paul Debevec of USC’s Institute for Creative Technologies. This is Tony Stark stuff, coming at us super-fast and soon to be delivered via JS, WebGL, and ORBX.js running in the browser.

I was thrilled to be included in today’s event, hosted at Autodesk‘s fabulous San Francisco offices. I gave a demo of Epic Games Unreal Engine 3 (Unreal Tournament, “Sanctuary” level) running via Emscripten and asm.js at full frame-rate in Firefox Aurora, and spoke about how JS will continue to evolve “low-road” as well as “high-road” APIs and features to exploit parallel hardware.

As Jeff Kowalski, Autodesk’s CTO, pointed out, the benefits go beyond major cost reduction in CGI and similar processing work, to increase collaboration and innovation radically, by relieving creative people from having to sit at big workstations. The GPU cloud means many alternative ideas, camera angles, etc., can be tried without waiting hours for each rendering. Even from the beach, via your 4G-connected tablet. Teams around the world can collaborate closely as timezones permit, across the web.

We will continue to collaborate with OTOY; I’ll post updates on this topic. It’s hot, and moving very quickly. Kudos to OTOY for their brilliant innovations, and especially for porting them to JS and WebGL so quickly!

When we at Mozilla say the Web is the platform, we are not bluffing.

/be

P.S. Always bet on JS!

P.P.S. Hat tip to Andreas Gal for seeing far, with Broadway.js.

May 04, 2013 01:03 AM

Life does not, apparently, go on as fast as one hopes; yesterday I was too exhausted and depressed to work and took a day off. To which I credit me being semi-functional today.

In any case, today I learned about the magic of rustc's filesearch module. What seems like eons ago but probably only a week ago, I set up some wacky symbolic link inside my build directory to get rustpkg to be able to find the core library. After pulling a new LLVM version, I had to blow away my build directory, as one often does. And thus, the rustpkg tests no longer ran because once again, it couldn't find core. And of course, the same thing happened on the bots, causing tests to fail, because my hack was a truly terrible hack (that only worked on my local machine).

rustc infers the sysroot -- which is the root directory that it uses for searching for standard libraries -- from the name of the executable you're running. Keep in mind that rustc is just-a-library so there's more than one possible executable name. Most executables sit under build/x86_64-apple-darwin/stage2/bin/whatever in our build system (not even talking about installed builds here, just in-place builds). The stages refer to the stages of compiler bootstrapping, but for testing we usually run stage 2. And of course, the host and But it happens that the test executable containing all the #[test] functions lives in build/x86_64-apple-darwin/tests/ and so the parent directory getting inferred was totally wrong. The answer is basically for me to just add in some terrible hacks to override the sysroot from my test functions, but it took me a while to get to that.

They didn't tell me in grad school that there'd be days like these...

comments

May 03, 2013 01:28 AM

The past couple days, I've been unsurprisingly not very focused on work; you'll know why if you read my most recent blog post that doesn't have the "research" tag.

But life does go on. Having landed the file searching patch that I talked about in my previous Rust post, I went on to working on rustpkg install. Like a good little software developer, I wrote the tests first, making the actual implementation pretty painless. Right now my tests are failing because apparently, I'm creating directories with the wrong permissions. Although lsing the directories, they look fine, so the Rust permission-checking code I wrote was probably wrong.

That (and a triage meeting) was about all I could manage today.

comments

April 26, 2013 01:49 AM

Today's work was on bringing the reality of rustpkg up to the point described in my doc pull request, at least up to the point of the directory and file structure of packages and having rustpkg look for package source files in a src directory inside a workspace. Also, now it uses a one build directory per workspace for putting object files and other temporary files in, instead of one per package. A workspace is just a container for packages; right now, always a directory on your local filesystem, but in the future, also on github or some other remote server containing version control repositories.

Now most of the test scenarios that I checked in build, and I've crossed off one more task on the schedule. Yay crossing things off. This pull request contains the work I'm talking about.

comments

April 23, 2013 01:29 AM

With rustpkg I'm trying as much as I can to stick to a tests-first/docs-first approach (I suppose strictly speaking, tests-first/docs-second), so this is my pull request for today: adding a draft of a manual (without much actual content, but starting to write things down) and rearranging the test scenarios again to have the directory structure that we agreed on.

I figure when I don't know exactly what I'm doing, writing things down as I go along (rather than waiting until the end to write the docs) helps me see if there's an area I'm still not clear on. I don't know why it's so hard for me to remember that that's the case for me :-)

comments

April 19, 2013 12:13 AM

Today's work on rustpkg is all in my pending pull request. I wrote out a schedule containing all the significant pieces-of-work remaining on rustpkg that Graydon and I could think of right now, and estimated times for each task (like all software engineers I'm terribly bad at estimating time for things). I'm hoping having everything laid out in a linear fashion will help me stay on-track, though.

I'm also glad to have spent some time writing down hypothetical-rust-package-commands-that-should-work (in a text file in the pull request), even though we can't do automatic testing yet. It's good to know what you're doing before you do it...

As an end-of-the-day small task, I made the typechecker able to survive more errors; generally the typechecker should be able to emit a type error and then keep going (since most people like to see as many errors as possible so they can try to fix them in one go, rather than one at a time) but there are a few cases here and there where it's tricky to set up the right data in various environments for the typechecker to keep going. So there's two less of those cases now.

comments

April 18, 2013 02:46 AM

I'd given feedback on a rustc pull request saying "try to get rid of that copy" in a pull request, but when asked how, I looked at the code and realized it wasn't easy to rewrite without a copy. So that was an hour gone!

Then, back to rustpkg; I started testing it on the test cases I wrote yesterday (just manually, for now, though writing a test runner needs to happen soon) and noticed that I wasn't naming the output executables or libraries correctly. So I added code to synthesize the link crate attributes with the inferred name and version, so you don't have to write it explicitly in the package file. Now, 2 out of 3 of my test cases work.

And finally, to round the day off, removing a few more bad copies.

Next:

• Error out in the case where no lib.rs, main.rs, bench.rs, or test.rs fine is found
  
• Split tests into pass and fail subdirectories (for tests expected to build correctly, and tests expected to fail)
  
• Add a test case to test that a package without a lib, main, bench, or test file fails.
  
• Submit a new pull request.
  

comments

April 17, 2013 12:28 AM

I landed this commit from the end of last week, cleaning up tests and error handling in rustpkg, and I submitted this pull request that allows rustpkg to build executables.

When talking to Graydon about how to do regression tests for rustpkg, I realized I'd been doin' it wrong, at least a bit -- the new policy for rustpkg is that you're going to have to name your crate files main.rs if you want to get an executable and lib.rs if you want to get a library (both can coexist), with additional options for building tests and benchmarks. I'd been allowing an additional option, which is naming your main crate foo.rs for a package named foo, and inferring whether or not it was a library based on the presence or absence of a main function. This was all wrong, so I'll have to undo that work tomorrow. I'm trying to tell myself I'm "exhibiting flexibility and willingness to admit mistakes" rather than "spending all my time writing code that I then delete".

For an easy end-of-the-day task, I decided I'd try to kill off a few bad copies, but that involved rebasing a branch, which involved rebuilding LLVM, which is evidently going to involve deleting my build/ directory and starting from scratch. Which, amusingly, emphasizes the need for a functional rustpkg!

comments

April 16, 2013 01:37 AM

My pull request from yesterday landed, so I'm on to further cleanup. First, I submitted another pull request to use a condition for error handling instead of making a function return an Option. Conditions are supposed to be for rare errors; this particular case was the situation where creating the build directory in the package directory to put the build output in fails. I figured that would be pretty rare. I hadn't really written code with conditions before, but they're about as simple as it gets. Sort of like exceptions, but simpler and implemented in terms of Rust macros.

For my next trick, I wanted to make sure that rustpkg could compile a package that doesn't have a package script. For realism, I decided to see which of the Servo submodules had a relatively simple Makefile, meaning that rustpkg should be able to build it as-is; I picked rust-core-foundation, and am working on getting it to build now.

comments

April 13, 2013 12:52 AM

So I finally submitted a pull request with my initial work on rustpkg. I'm embarrassed at how little this patch actually does; I've tested it with one command, build, on one package, my fork of rust-sdl. The difference now is that as per #5679, rustpkg doesn't require a package to explicitly declare its ID and version number; instead, it just looks for a directory with the name you give it. So if you write rustpkg build rust-sdl, rustpkg will expect the current directory to contain a directory called rust-sdl that contains the package. Likewise, if it was rustpkg build rust-sdl-0.1, it would infer the version to be 0.1 (though I don't think I actually implemented that part yet).

I still have to figure out how to write unit tests for this. Also, rather than trying to do everything at once, I made several of the commands other than build fail with "not yet implemented". rustpkg wasn't in a great state anyway, so I don't think this will make anyone too unhappy. I think it's better to explicitly fail than quietly do something not-quite-right.

Hopefully I can proceed faster now that I'm past the awkward initial "what am I doing?" stage of the project.

comments

April 12, 2013 01:19 AM

My titles are un-creative lately.

So remember when I said I'd gotten rust-sdl building with rustpkg? I really... didn't. Or else, I broke a whole lot of code while rebasing. Either way, I spent today and yesterday (when not paying attention to the work week proceedings) getting it to work again. And, just now, I finally did, after learning a valuable lesson about Unix command line argument processing that doubtless I knew in 2004 but didn't know yesterday.

Next, continue cleaning up code so I'm not embarrassed by "oops, how did that get in there?" moments when I submit a pull request.

comments

April 11, 2013 01:34 AM

When running final tests for my latest patch queue, I discovered that someone has apparently added a new color to the repertoire: a hot pink. So I now present to you a snapshot of TBPL that uses all the colors except gray (running), gray (pending), and black (I've never seen this one):

April 10, 2013 09:00 PM

Today was the beginning of the research work week here in Vancouver, and I don't know about you, but I can't follow a discussion for 8 hours straight -- even with a break for lunch in the middle. So I tried to listen as well as I could to the parts that overlapped more with my knowledge, and during the rest, hack on stuff and listen with one ear.

I started cleaning up my rustpkg changes in preparation for submitting a pull request. Part of that was adding more documentation and such, part was removing bits of code that aren't needed now that we're using package IDs and not requiring a package script. I'm not sure whether to break existing code that uses rustpkg (not that there's a lot of it yet) or try to be backwards-compatible.

After that, it'll be submitting a pull request for the changes I made to rust-sdl, making the clean command work (since it's annoying to have to manually delete files every time to make sure my changes still worked), and making rust-http-client build as my second test project. And then, of course, making all the other commands work. That's just the beginning.

Figuring out how to write unit tests for rustpkg will be interesting as well. Necessary, but I'm not sure how.

comments

April 09, 2013 12:43 AM

This post is admittedly long overdue, but I kept wanting to delay this post until I actually had patches up for review. But I have the tendency to want to post nothing until I verify that the entire pipeline consisting of over 5,000 lines of changes is fully correct and documented. However, I'm now confident in all but roughly three of my major changes, so patch documentation and redistribution is (hopefully) all that remains before I start saturating all of the reviewers in Thunderbird with this code. An ironic thing to note is that these changes are actually largely a sidetrack from my original goal: I wanted to land my charset-conversion patch, but I first thought it would be helpful to test with nsIMimeConverter using the new method, which required me to implement header parsing, which is best tested with nsIMsgHeaderParser, which turns out to have needed very major changes.

As you might have gathered, I am getting ready to land a major set of changes. This set of changes is being tracked in bugs 790855, 842632, and 858337. These patches are implementing structured header parsing and emission, as well as RFC 2047 decoding and encoding. My goal still remains to land all of these changes by Thunderbird 24, reviewers permitting.

The first part of JSMime landed back in Thunderbird 21, so anyone using the betas is already using part of it. One of the small auxiliary interfaces (nsIMimeHeaders) was switched over to the JS implementation instead of libmime's implementation, as well as the ad-hoc ones used in our test suites. The currently pending changes would use JSMime for the other auxiliary interfaces, nsIMimeConverter (which does RFC 2047 processing) and nsIMsgHeaderParser (which does structured processing of the addressing headers). The changes to the latter are very much API-breaking, requiring me to audit and fix every single callsite in all of comm-central. On the plus side, thanks to my changes, I know I will incidentally be fixing several bugs such as quoting issues in the compose windows, a valgrind error in nsSmtpProtocol.cpp, or the space-in-2047-encoded-words issue.

It's not all the changes, although being able to outright remove 2000 lines of libmime is certainly a welcome change. The brunt of libmime remains the code that is related to the processing of email body parts into the final email display method, which is the next target of my patches and which I originally intended to fix before I got sidetracked. Getting sidetracked isn't altogether a bad thing, since, for the first time, it lets me identify things that can be done in parallel with this work.

A useful change I've identified that is even more invasive than everything else to date would be to alter our view of how message headers work. Right now, we tend to retrieve headers (from, say, nsIMsgDBHdr) as strings, where the consumer will use a standard API to reparse them before acting on their contents. A saner solution is to move the structured parsing into the retrieval APIs, by making an msgIStructuredHeaders interface, retrievable from nsIMsgDBHdr and nsIMsgCompFields from which you can manipulate headers in their structured form instead of their string from. It's even more useful on nsIMsgCompFields, where keeping things in structured form as long as possible is desirable (I particularly want to kill nsIMsgCompFields.splitRecipients as an API).

Another useful change is that our underlying parsing code can properly handle groups, which means we can start using groups to handle mailing lists in our code instead of…the mess we have now. The current implementation sticks mailing lists as individual addresses to be expanded by code in the middle of the compose sequence, which is fragile and suboptimal.

The last useful independent change I can think of is rewriting the addressing widget in the compose frontend to store things internally in a structured form instead of the MIME header kludge it currently uses; this kind of work could also be shared with the similar annoying mailing list editing UI.

As for myself, I will be working on the body part conversion process. I haven't yet finalized the API that extensions will get to use here, as I need to do a lot of playing around with the current implementation to see how flexible it is. The last two entry points into libmime, the stream converter and Gloda, will first be controlled by preference, so that I can land partially-working versions before I land everything that is necessary. My goal is to land a functionality-complete implementation by Thunderbird 31 (i.e., the next ESR branch after 24), so that I can remove the old implementation in Thunderbird 32, but that timescale may still be too aggressive.

April 07, 2013 09:47 PM

I said yesterday that I got LLVM to build without a Makefile, but I didn't, really. The Android-related message that I thought was a warning was actually an error, but it took me most of the day to figure out for sure. The cause of the problem? There was code in our Rust wrapper for the LLVM library that was unconditionally initializing the command-line arguments on every call to LLVMRustWriteOutputFile. Since rustpkg may call this function twice -- once to write the executable for the package script, and one to write the executable or library (library, in this case) we're compiling, the error resulted. Cryptic for several hours, to crystal clear within a minute. This wasn't a problem in previous iterations of rustpkg since the change to add some Android-related flags was made recently.

Fixing it was easy, since apparently in LLVM, the flags that say whether a particular command-line flag is toggled are global state, and I submitted a pull request.

But I'm still kicking myself that it took me hours to track this down. Software engineering, folks.

Next, cleaning up the changes I made to rustpkg and rust-sdl (in the latter case, getting it to build with incoming Rust) so I can actually submit pull requests!

comments

April 06, 2013 12:32 AM

Rust 0.6 happened!

I set aside the trans bug I was working on before, just like I said I would do, because trans badly needs the Hatchet of Refactoring, but now is not the time.

Instead, I went back to working on rustpkg; the work that needs to be done is now better outlined (by Graydon). In the meantime, though, I'd wanted to make rustpkg able to build an actual, existing package and infer the package name and version from the path name. I succeeded in doing this today, with rust-sdl, except that I got some warnings at the end about the Android back-end that I need to investigate. The notable thing is that rust-sdl previously had a Makefile that renamed a symbol in the existing SDL library before building; I ported that logic into a package script -- a Rust file -- instead, so now it builds without a Makefile.

Of course, one of the things rustpkg needs to do is build packages that don't have a pkg.rs file -- in the current trunk version, it always requires one -- so perhaps I didn't pick the best example package to start with.

comments

April 05, 2013 06:33 AM

[air.mozilla.org video]
[slideshare.net link]

Disrupt any enterprise that requires new clothes.

— Thoreau (abridged) adjusted for Mozilla by @lawnsea.

I gave a brief talk last night at the Mozilla Research Party (first of a series), which happened to fall on the virtual (public, post-Easter-holiday) celebration of Mozilla’s 15th anniversary.

I was a last minute substitution for Andreas Gal, fellow mad scientist co-founder at Mozilla Research, so I added one slide at his expense. (This talk was cut down and updated lightly from one I gave at MSR Paris in 2011.) Thanks to Andreas for letting me use two of his facebook pics to show a sartorial pilgrim’s progress. Thanks also to Dave Herman and all the Mozilla Researchers.

Mozilla is 15. JavaScript is nearly 18. I am old. Lately I mostly just make rain and name things: Servo (now with Samsung on board) and asm.js. Doesn’t make up for not getting to name JS.

(Self-deprecating jokes aside, Dave Herman has been my naming-buddy, to good effect for Servo [MST3K lives on in our hearts, and will provide further names] and asm.js.)

Color commentary on the first set of slides:

I note that calling software an “art” (true by Knuth’s sensible definition) should not relieve us from advancing computer science, but remain skeptical that software in the large can be other than a somewhat messy, social, human activity and artifact. But I could be wrong!

RAH‘s Waldo featured pantograph-based manipulators — technology that scaled over perhaps not quite ten orders of magnitude, if I recall correctly (my father collected copies of the golden age Astounding pulps as a teenager in the late 1940s).

No waldoes operating over this scale in reality yet, but per Dijkstra, our software has been up to the challenge for decades.

I like Ken‘s quote. It is deeply true of any given source file in an evolving codebase and society of coders. I added “you could almost say that code rusts.”

Here I would like to thank my co-founder and partner in Mozilla, Mitchell Baker. Mitchell and I have balanced each other out over the years, one of us yin to the other’s yang, in ways that are hard to put in writing. I can’t prove it, but I believe that until the modern Firefox era, if either of us had bailed on Mozilla, Mozilla would not be around now.

A near-total rewrite (SpiderMonkey and NSPR were conserved) is usually a big mistake when you already have a product in market. A paradox: this mistake hurt Netscape but helped Mozilla.

I lamented the way the Design Patterns book was waved around in the early Gecko (Raptor) days. Too much abstraction can be worse than too little. We took years digging out and deCOMtaminating Gecko.

As Peter Norvig argued, design patterns are bug reports against your programming language.

Still, the big gamble paid off for Mozilla, but it took a few more years.

Who remembers Netscape 6? At the time some few managers with more ego than sense argued that “the team needs us to ship” as if morale would fall if we held off till Mozilla 1.0. (I think they feared that an AOL axe would fall.) The rank and file were crying “Nooo!!!!!”

AOL kept decapitating VPs of the Netscape division until morale improved.

2001: Another year, another VP beheading, but this one triggered a layoff used as a pretext to eliminate Mitchell’s position. The new VP expected no mo’ Mitchell, and was flummoxed to find that on the next week’s project community-wide conference call, there was Mitchell, wrangling lizards! Open source roles are not determined solely or necessarily by employment.

At least (at some price) we did level the playing field and manage our way through a series of rapid-release-like milestones (“the trains will run more or less on time!”) to Mozilla 1.0.

Mozilla 1.0 didn’t suck.

It was funny to start as a pure open source project, where jwz argued only those with a compiler and skill to use it should have a binary, and progress to the point where Mozilla’s “test builds” were more popular than Netscape’s product releases. An important clue, meaningful in conjunction with nascent “mozilla/browser” focus on just the browser instead of a big 90′s-style app-suite.

A lot of credit for the $2M from AOL to fund the Mozilla Foundation goes to Mitch Kapor. Thanks again, Mitch! This funding was crucial to get us to the launch-pad for Firefox 1.0.

We made a bit more money by running a Technical Advisory Board or TAB, which mostly took advice from Enterprise companies, which we proceeded to (mostly) ignore. The last TAB meeting was the biggest, and the one where Sergey Brin showed up representing Google.

Due to a back injury, Sergey stood a lot. This tended to intimidate some of the other TAB members, who were pretty clearly wondering “What’s going on? Should I stand too?” An accidental executive power move that I sometimes still employ.

Jump to today: here is Rust beating GCC on an n-body solver. Safety, speed, and concurrency are a good way to go through college!

As you can see, Mozilla Research is of modest size, yet laser-focused on the Web, and appropriately ambitious. We have more awesome projects coming, along with lots of industrial partners and a great research internship program. Join us!

/be

April 04, 2013 05:32 AM

The release work continues apace, with just 3 open issues left! I worked on documentation a bit and then when it seemed like there was nothing blocking the release that I could easily finish, went back to one of the bugs that got deferred, #4850. I had done some work on this last week but got over-ambitious in trying to restructure trans, when the urgent thing was to fix the bug.

Once I started over, with the knowledge I gleaned from last week's debugging, it was pretty obvious that the issue was that trans code was assuming that the self argument to a trait reference is always passed by reference. But Rust allows traits to be declared with their self argument passed by value; depending on what type self actually is (and this is usually, but not always, known at compile time, since Rust compiles polymorphism by monomorphization), that means either moving it into the callee, or copying it. So there was some code assuming self was passed by reference, other code assuming it was passed by value, so in some cases with by-value self, the result was generated code that segfaulted.

It seems like the trans code could be better structured as far as how many pieces of code know things about self. But I'm trying to make minimal changes right now, and am just patching up those places as they appear to take into account that the self argument to a method might not always be a pointer. However, I'm worried that the fact that the size of the self argument depends on the monomorphized code for the method is going to cause problems. It hasn't yet, but I'm afraid it will, and that this will mean bigger restructuring is needed for trans.

I'm setting this aside in any case once we tag the release, though, and moving on to focus on rustpkg for a while, so I don't get totally lost in bug-fixing.

comments

April 02, 2013 01:56 AM

This week, a number of Mozillians attended the annual Game Developers Conference in San Francisco to demonstrate how the Web is a competitive platform for gaming and game development.

We’ve worked very hard over the past couple of months on technologies used to speed up the Web for game development, including asm.js, a JavaScript optimization technology and pure-JS-subset target language for compilers; Emscripten, a C++ to JavaScript compiler; and OdinMonkey, the SpiderMonkey JavaScript engine’s compiler for asm.js source.

On Wednesday, we were happy to announce that we’ve teamed up with Epic Games to combine these technologies to bring Epic’s Unreal Engine 3 to the Web.

This is significant because it demonstrates to everyone that game developers and publishers can now take advantage of the reach and scale of the Web without the additional user acquisition friction and even higher costs (infinite on iOS ) associated with third-party plugins. These technology advancements also mean improved performance and the ability to port lots of games to the Web on desktops, laptops, and mobile devices. (Note well that asm.js code runs pretty fast on modern browsers, and super-fast when optimized as by OdinMonkey.)

We’ve also developed a new demo for Banana Bread that includes multi-player support and integrates parts of WebRTC for the data channel.

Big thanks to everyone on the gaming team and supporters throughout the Mozilla project for all the hard work to make this possible. I must single out Martin Best, Dave Herman, Vladimir Vukićević (WebGL creator!), Luke Wagner, and Alon Zakai (AKA @kripken).

I was personally thrilled to meet Tim Sweeney, CEO and founder of Epic Games, for breakfast before GDC Thursday. I’d been aware of Tim since his invited talk at POPL 2006. Tim and I were on very much the same page on many things, including JS’s ascendance as a safe target assembly-like language for C++ via asm.js and beyond.

/be

P.S. I’d like to thank Luke Wagner for taking on module ownership of SpiderMonkey, as Dave Mandelin has handed the baton on.

P.P.S. Hot rumor: WebGL in IE11? Source, judge for yourself. I have hopes.

UPDATE: withinwindows.com says WebGL is coming, disabled by default so far, and (look for “UPDATE3″ run together at the original source) with a different shader language, IESL based on HLSL, from GLSL ES.

2nd UPDATE: From “UPDATE4″ at François Remy’s blog post, GLSL is the default and you set a registry flag to get HLSL. Cool!

P.P.P.S. asm.js support in V8 bug on file.

March 29, 2013 10:22 PM

Down to 35 open issues for the release. I'm still working on #4439 and trying to untangle the mysteries of owned self. Since it started looking like we're not releasing 0.6 this week, I decided (quite probably unwisely) to rewrite a lot of the trans code to use the Datum type, to make the error easier to track down. The older trans code passes around raw LLVM values, whereas some of the newer code uses a Datum, which is an LLVM value bundled with a Rust type t and an annotation as to whether this value is a value of type t, or a pointer to t. Modernizing things is hard and tedious and I don't really know whether it's more or less time-consuming than actually tracking down the bug without modernizing the code. But that's how it goes.

I also added a span (location in source code that tells you the line and column number of a location in some source file) field to the Datum type, which I don't know if reviewers will object to when I try to check it in, but it sure makes debugging easier!

I feel like I should be doing something more urgent for the release, but I don't know what it is.

comments

March 29, 2013 03:29 AM

We're down to 46 open issues for the release! I spent most of my time today working on #4439, which may be behind a few other open bugs. Bugs that cause the compiler to generate seg-faulting code are bad, so I figured I would see if I could fix it before the release. Turns out trait methods that take their self by-move... are implemented strangely. As usual, many bits of undocumented code in trans (the Rust-to-LLVM pass) combine to create confusion.

I thought the best way to go forward would be to convert the data structure representing the self object in trans to use a Datum -- a type that wraps up an LLVM value with data about whether it's meant to be passed by reference or by value -- instead of a raw value. I still think it is, but getting everything consistent means debugging many LLVM assertion failures.

comments

March 27, 2013 01:24 AM

Today I thought I would focus on closing remaining release blockers; we're trying to punt most of the bugs marked 0.6 to some point in the future, as I mentioned last week. I didn't have a super-productive time, but we are down to 72 open issues milestoned for 0.6. I think I isolated the mysterious bit of code -- in trans::base::copy_arg_to_allocas -- that's causing by-value self not to work (as manifested in segfaults or bad frees), for example in #4850 and #4439. But I don't know exactly why the code is there or how to fix it.

Otherwise, I tried fixing a few bugs only to realize the work had already been done and the bug just needed to be closed.

comments

March 26, 2013 01:30 AM

Today, landed the patch I've been working on for a month to make the typechecker's handling of the error and _|_ types more sensible. I've already written about this; I'm just glad to have it done now, and know that the Rust typechecker won't complain about derived errors anymore! I think this is important for usability; I know from experience that too much noise (error messages that don't matter) can be demoralizing when you're trying to get your code compiling.

Except... then I went back and ran the test for this issue and still got a derived error. Oh no! It turned out I didn't edit the code in the pattern-match-checking part of the typechecker, and it was still calling span_fatal (a function that, as you might guess, unconditionally prints out an error message and then kills the whole compiler) rather than report_type_error (a function I wrote that checks for errors involving error types and suppresses them). So I started fixing that, but ran out of time.

In the meantime, I also finished triaging my allotted quota of randomly selected 0.6-milestone bugs, with the goal of getting everything either closed or un-milestoned unless it's SUPER IMPORTANT; the plan is to try to release 0.6 next week. We're going to shift towards priorities rather than milestones in the bug tracker, since it's become clear that labelling bugs with milestones doesn't help that much when doing time-based releases.

Tomorrow, finishing the pattern-checking stuff, doing my quota of doc work for the release, and going back to rustpkg.

comments

March 22, 2013 02:46 AM

I got my week off to a good start by restoring a "new" laptop from my TimeMachine backup, for the second time in about ten days. (After last week's 193 million memory errors.)

After the restore (which took about an hour, fortuitously coinciding with the weekly all-hands meeting) I started my three Rust workspaces building, and after some strategic sledgehammer (removing-the-build-directory) applications, that actually succeeded. So I went back to alternating between work on rustpkg and the typechecker refactor, as build times permitted.

I actually made some progress on rustpkg: last week I'd started trying to port an example of a custom Makefile into a package script instead. The example package I'm using, rust-sdl, currently has a Makefile with this rule (which only fires when building on Macs):

$(SDLXMAIN): $(SDL_PREFIX)/libSDLmain.a
        $(CP) $< $@
        $(AR) -x $@ SDLMain.o || $(RM) -f $@
        $(LD) -r SDLMain.o -o SDLXMain.o -alias _main _SDLX_main \
             -unexported_symbol main || $(RM) -f $@
        $(MV) SDLXMain.o SDLMain.o || $(RM) -f $@
        $(CHMOD) u+w $@ || $(RM) -f $@
        $(AR) -r $@ SDLMain.o || $(RM) -f $@

It's not for me to ask why; anyway, being fairly simple, this made a good example for me. I transliterated it into Rust like so:

const SDL_PREFIX : &static/str = "/usr/local/lib";

#[pkg_do(post_build)]
fn post_build() {
 // todo: delete new_sdl_lib_name if any commands fail
    let existing_sdl_lib_name = path(SDL_PREFIX).push("libSDLmain.a");
    let new_sdl_lib_name = path(SDL_PREFIX).push("libSDLXmain.a");
    let sdl_obj_file = ~"SDLMain.o";
    let sdlx_obj_file = ~"SDLXmain.o";
    copy_file(existing_sdl_lib_name, new_sdl_lib_name);
    archive_file(Extract, new_sdl_lib_name, sdl_obj_file);
    load_file(sdl_obj_file, sdlx_obj_file, ~[(~"_main", ~"_SDLX_main")],
              ~[(~"_main")]);
    move_file(sdlx_obj_file, sdl_obj_file);
    change_file_permissions((User, Writable), new_sdl_lib_name);
    archive_file(Extract, sdl_obj_file);
}

This is still sketchy and I haven't actually implemented most of the functions above yet. But the basic idea for what we're trying to do in rustpkg is that as much functionality as possible should be in the Rust build system; you shouldn't need to write a Makefile even to do something fancy.

So I spent that part of today trying to make this work. It doesn't yet, but I did get up to the point where rustpkg fails with a resolve error -- when calling out to compile the code in the package script -- because the functions I didn't implement yet aren't defined. Most of that time was spent defining a new version of rustpkg's PackageScript structure that has a PkgId field instead of a name field and a version field. PkgIds just bundle together the name and version of the package, both of which now will be inferred from the directory name (or version control info if present) if not declared. (The existing version of rustpkg that's up on github requires you to declare your package's name and version explicitly.)

The typechecker situation is sadder; I fixed my bad merges and ran tests, only to find that one of the run-fail tests was failing with a nondeterministic error:

• Assertion failed: ((Ty->isIntegerTy() || Ty->isPointerTy()) && "Invalid cast"), \
      function CreatePointerCast, \
      file /Users/tchevalier/rust3/src/llvm/lib/VMCore/Instructions.cpp, \
       line 2385.
  

• error: internal compiler error: Asked to compute contents of fictitious type

• rustc(29409,0x107cca000) malloc: *** error for object 0x7fdd9ba19f80: pointer \
      being freed was not allocated
  *** set a breakpoint in malloc_error_break to debug
  

• Assertion failed: (box->td != NULL), function free, \
     file /Users/tchevalier/rust3/src/rt/boxed_region.cpp, line 75.

These are all different error messages that I got on different runs of the same test. Because this behavior looked so similar to what was happening on the other, broken laptop (though with a different test), I thought maybe this one was borked too. What are the chances of that? It's like in the joke that goes "I always bring a bomb with me when I fly, because what are the chances of there being two bombs on the airplane?" Running valgrind, though, assuaged that fear, since it actually worked (and reported a metric crap-ton of invalid reads), as opposed to declaring its inability to execute rustc (like on the broken laptop). The errors suggest some sort of memory corruption having to do with types, which isn't totally surprising since rustc represents types in a totally unsafe way, and it would be easy to get something wrong. It's still puzzling me that I can get so many tests to work with only one failing if I introduced a bug like that, though. Usually, these kinds of failures are catastrophic.

Anyway, progress, in that I closed the day with a laptop that isn't obviously broken, unlike last Friday...

comments

March 19, 2013 12:57 AM

I spent too much of today dealing with an apparent hardware failure on -- irritatingly enough -- the laptop I'd borrowed from IT to replace the laptop I was using for the past almost-a-year, which just before I left for Vancouver decided it didn't have a wifi card anymore. Most of yesterday I was trying to debug an issue where I was getting non-deterministic segfaulting and also occasional not-seeing-the-bug-I-was-looking for. Graydon suggested that I run memtest86, and so I did... and it reported 193 million errors before I decided that was enough. So then I sshed into a buildbot that Graydon decommissioned temporarily, and moved my working directories onto it.

After which, I found the bug! I'm not happy with my fix, though. So you may recall that a while ago, I introduced a ty_err type to represent a value that can't be typed and allow the typechecker to typecheck other parts of the program before finally reporting all the errors. The fix I was working on was to never represent a type like () -> ty_err -- a function type where the result type is erroneous -- because this should just normalize to ty_err; a type with a component that was erroneous is erroneous itself. The problem is that a type might look like () -> α, where α is a type variable, and if α gets unified with ty_err at some later point in time, the way the Rust typechecker works makes it hard to see how that information gets propagated back to anyone who might have a copy of () -> α sitting around.

My solution was just to call a function that resolves all type variables in the type at the point where we've just checked the body of a closure, and write the resolved type back in for the closure. But this is unsatisfactory to me because I haven't really thought about whether this is the only point in the typechecker where that needs to happen.

The whole approach is probably wrong and misguided in some way I don't quite understand, and then I decided it was a good idea to rebase my branch for some reason, which resulted in conflicts and quite possibly a bad merge by me. I think that means it's time to leave for the weekend.

comments

March 16, 2013 01:00 AM

I succeeded at moving to Vancouver! A bit of culture shock. (Feel free to come up with a joke about moving out of a mutable reference or something.) I'm only here for less than three months, so I'm trying to absorb as much of Graydon's knowledge as possible in the meantime.

Continuing to work on the typechecker refactoring (yes, still); running up against very frustrating sporadic segfaults that I'm pretty sure are not to do with my changes, and that's about the most frustrating thing that can happen.

I also started working on rustpkg to finish up the work that our valiant volunteer Zack started. So far, trying to get rustpkg to build rust-sdl. Since rust-sdl requires some custom build logic -- currently implemented in a Makefile -- to rename the library file on Macs, I figured that might be a good starting point for porting that into build logic implemented in Rust that rustpkg can run. Or not; I still don't really know what I'm doing yet.

Figured I'd prioritize resuming regular posting ahead of making informative postings ;-)

comments

March 15, 2013 01:24 AM

Last week started with a bang, with Mozilla’s Firefox OS launch at Mobile World Congress 2013. We announced that Firefox OS had won the support of 18 carriers, four device manufacturers, and a major chipset company (Qualcomm) at a barn-burner of a press conference in Barcelona on Sunday night.

Pictures (the first shows the room less than half-full, pre-press-conference):

Founders and executives:

The Mozilla crew who helped at the press conference (others were in Barcelona doing equally important pre-MWC work):

Here is our stunning, always-crowded booth in Hall 8 (“App Planet”):

Two days later, the excitement ramped even higher with news of a fifth device maker, Sony.

Our success in attracting support from these partners is due in part to our ability to innovate and standardize the heretofore-missing APIs needed to build fully-capable smartphones purely from web standards. Therefore I will give an update below to my post from last MWC with details on API standardization progress over the last year.

I want not only to identify the good work being done, but also to reinforce that:

• We are collaboratively building these APIs to fill real gaps in the standards.
• We actively work to get them standardized across all operating systems and browsers, as part of Mozilla’s mission.
• We have made terrific progress in the last year thanks to Firefox OS, and many of the APIs work in Firefox for Android too.
• Coopetition for web developers has already caused Samsung to patch WebKit to support some of these APIs.
• We will continue to update API specs and code as new devices and sensors come to market.

The missing APIs must be added to the web platform in order to enable the billions of new mobile users who will be coming online in the next few years to have affordable web-based phones, tablets, and apps. Emerging market consumers and developers generally cannot afford increasingly higher-end, native-app-advantaged smartphones from the two bigs.

The main reason that many developers historically have turned to native OS-specific app platforms and tools is because that was the only way to work with the different hardware elements and proprietary features (e.g., battery status, dialer, SMS, payments system) present on mobile devices. Those developers who bravely soldiered on, trying to use HTML5, were forced to envelop their code in compiled native app wrappers such as Phonegap generates.

There was no technical reason for OS proprietors to disadvantage their web stacks, lock developers into their native stacks, or make their store an exclusive trust anchor. Yet the incumbents did hold the web back by withholding APIs.

Just as the first browsers’ creators exposed file pickers and alert dialogs to work with the underlying OS, we are finally filling in the web platform to include the missing smartphone APIs. This levels the web vs. native playing field. And by following our usual best practices of working in the open and submitting all of our work to the standardization process, we can be confident that the missing functionality will become available on all platforms eventually.

When these APIs are available on all web rendering engines — even desktops and laptops, where the APIs make sense — then developers can be sure that their code will run interoperably, no matter what the underlying operating system. Fragmentation, if any, could arise at the OS level, e.g., due to not updating the OS’s web engine (as happened in Android 2.3).

Barring OS-specific update failures, developers can be sure because they have consolidated cross-OS, cross-browser market power. In today’s competitive browser markets, web developers rule the roost; browser and web engine purveyors seek to win their favor. Developers will not bother with up-and-coming operating systems that deviate more than epsilon from the standards.

The example set by our good friends at Samsung in patching WebKit based on the new specifications validates our approach. It means that if Tizen devices become numerous, it is highly likely that mobile web applications that work on Firefox OS will work on Tizen.

Likwise, we are working with Microsoft on Pointer Events, since touch-enabled web apps must today unify mouse and touch events, and the previous W3C multi-touch work foundered on patent assertions.

The orders-of-magnitude increase in Javascript performance across all engines in the last few years has been stunning. This boost helps to address another web vs. native “gap”. And just around the corner is asm.js — more on that in a future post!

Other performance enhancements include continued work on off main thread compositing and GL layers. Yes, it’s true: the modern mobile-focused web rendering model includes implicit threading in the form of the Compositor thread, and the GPU’s parallel hardware.

As always, standards in progress are subject to change, but they require prototype implementations and user-testing (“user” meaning both developer and consumer). Mozilla remains committed to playing fairly by not forging de-facto standards out of prototypes, rather proposing before disposing and in the end tracking whatever is standardized.

(This is why we are vendor-prefixing the less certain among our new APIs — but I think we should aim to unprefix quickly once we have emerging consensus among implementors.)

Here’s what we’ve achieved in the last year and what we’re working on:

• Battery Status is now a W3C Candidate Recommendation (CR), and there was discussion recently about moving it to Proposed Recommendation (the last step before a REC, the status of an officially endorsed W3C recommended standard, indicating readiness for deployment within its problem domain).
• WebTelephony. Mozillians: Ben Turner, Jonas Sicking, Philipp von Weitershausen. This is being worked on by Intel and Telefónica folks as the Telephony API in the SysApps Working Group.
• WebSMS. Mozillians: Mounir Lamouri, Jonas Sicking. Being worked on as the Messaging API in the SysApps WG.
• Pointer Lock: this API is largely stable and implemented in multiple browsers. Mozilla is participating in the W3C Pointer Events working group along with Microsoft, Google, Opera, jQuery, and others. Mozillian: Matt Brubeck, together with Microsoft’s Jacob Rossi.
• Open WebApps has been proposed to W3C. We seem to have reached consensus to use the Mozilla proposal as a “First Public Working Draft” (FPWD) in the SysApps WG with Samsung as co-editor.
• Alarm API is now a W3C working draft. Intel is helping edit, a great example of collaboration.
• Web Activities, a simplified version of Web Intents. We focused on a simpler subset of Web Intents, which has proven to be a good idea. The feature set of Web Activities is working well, while Web Intents is currently going through a restart.
• Push Notifications. This API is still in an early state. We have some prototypes but not yet shipping code. Getting this right is hard given that we want to create something secure, scaleable, and highly available.
• WebFM API. We have an initial draft, but the W3C SysApps WG has decided to focus on a smaller set of specs first to build common ground. This API was deemed to be lower priority and so will have to wait a bit before it goes through standardization.
• WebPayment, implemented by Fernando Jiménez Moreno of Telefónica. There is a W3C Community Group entirely focused on payments, and now a W3C Task Force. We will work in these groups to make a flexible API for payments part of the web standards.
• Ambient light sensor, Mozillian Doug Turner is serving as editor.
• Proximity sensor, note Doug Turner in acknowledgements.
• UPDATE: Contacts Manager API, Eduardo Fullea and Jose M. Cantera of Telefónica are editing.

Developers interested in getting to know the new APIs better should look at this Mozilla Hacks post by Robert Nyman, which provides guidance on how to use them along with code examples.

So, Mozilla is leading the effort to complete the web platform on mobile devices. Our work is bearing fruit, as shown by the industry support for Firefox OS and the W3C’s efforts to standardize the new APIs. Even Andy Rubin had some kind words for us (thanks, Andy!).

The main prize for me remains the picture of smiling “mobile web developers” (read: “web developers”) that I can see just around the corner. Based on our MWC experience this year, the Web is in good shape to win that prize.

/be

March 07, 2013 12:01 AM

I fixed the tests that weren't working yesterday in the typechecker refactoring branch, and I'm running the test suite again -- a few compile-fail tests have failed already, but those should be fun to figure out. (So far I see a few ICEs due to the fact that we're continuing typechecking in more places now, rather than returning early because of seeing an error type.)

My post from yesterday got a few more comments than usual! I'm always glad to get comments on my Rust posts, especially since it can be hard for people who don't use Dreamwidth regularly to comment on my blog, so I appreciate any effort towards that end. Here's how to comment if you don't have a Dreamwidth account.

Also along those lines, I'm well aware that my Rust posts swing wildly back and forth between explaining basic terms and assuming you know about things that you'd only know about if you'd read some obscure module in rustc::middle -- so if you're curious about anything, please don't assume you "should" know what it means and please do ask questions!

comments

February 27, 2013 01:31 AM

Almost finished with the big typechecker refactor. Or the typechecker refactor of moderate size, anyway. Getting libcore to compile with the refactored typechecker was painful, but once I got there, I got all the way through building stage1 and most of the run-pass tests! The first round of tests that failed were:

failures:
    [run-pass] /Users/tchevalier/rust3/src/test/run-pass/unreachable-code-1.rs
    [run-pass] /Users/tchevalier/rust3/src/test/run-pass/unreachable-code.rs
    [run-pass] /Users/tchevalier/rust3/src/test/run-pass/weird-exprs.rs

There was something satisfying about that. Little weird-exprs tests doing their job.

As for why the unreachable-code tests failed: trans (the part of rustc that translates Rust to LLVM) tries to use type information to avoid generating code for unreachable Rust expressions. This is all pretty ad hoc, but since we have the _|_ type in the type system, why not use it?

The expression in this case looks like return + 1. In the old typechecker, if you looked up the type for this instance of 1, it would have been int or uint. But return always returns and arguments to + in Rust are always evaluated left to right, so we know that the 1 is unreachable here. So the new typechecker says 1 has type _|_ in this context.

That caused the code to translate int literals to complain, since it's not expecting an int to have type _|_. Why would anyone write code like this, you ask? They probably wouldn't, but macros could.)

So I'll just make trans more consistent in when it avoids generating code for unreachable expressions. Arguably it might be better to make trans not approximate what code is unreachable and just treat _|_-typed things as if they are reachable and generate code for them, but that's a change for another day.

comments

February 26, 2013 04:17 AM

I'm mostly done with the typechecker refactoring I talked about in my last post, though when I say I'm done I mostly mean I'm done getting it to compile. Tests passing is another matter! Plus, with a change like this, there's the nagging fear that (since we don't know what our test coverage is like) some subtle bug will slip through the cracks. I guess that's what code review is for.

In the meantime, bug triage! I spent my bug triage time today just looking at new bugs, although I'd been hoping to come up with a slightly shorter list of critical traits-related bugs for 0.6.

In the meantime, though, I'm going to prioritize these five trait-inheritance-related bugs:

• Check that for each impl of a trait, impls of the supertraits exist
  
• ICEs with traits that inherit generic traits -- also this one, which I hope is closely related
  
• Several examples of incorrect typechecking
  
• Something bad happens with trait inheritance and type synonyms of traits
  

Altogether, there are 40 open bugs labeled "traits" with the 0.6 milestone. I hope some of these are duplicates. I'd like to fix as many as possible, but I don't know how hard it will be.

There's also my pending work on 4678 -- refactoring how "self" and type parameters get handled -- which I really don't know how to make progress on; I have to talk to Niko about it when it's not late at night in his time zone. I'm thinking that maybe if a trait A has n type parameters -- including self -- then a *reference* to T (like in fn f<T: A>) should only require n - 1 type parameters. That seems weird, though, since the point is "treat self like any other type parameter".

comments

February 22, 2013 02:13 AM

I fixed #4736, which was easy enough, but that led to an adventure in refactoring the Rust typechecker.

Right now, most functions in the typechecker associate zero or more node IDs with types as a side effect (in a hash table that gets passed around), and return a bool. Every node in the abstract syntax tree (AST) has a node ID that identifies it uniquely.

The bool return value denotes whether or not the expression diverges. You can think of what that means like this: there's a primitive called fail, and fail or anything that definitively invokes fail diverges. (It's a little more complicated now because fail is a macro, but never you mind.) That information gets used to warn about statements that are statically known to be unreachable.

At the same time, one of the types that exists in the internal representation of types is ty_error, which represents an erroneous value. Normally a typechecker would abort at the first error it sees, but we want to report as many errors as possible during one compilation session, so we need a name for the type that erroneous subexpressions have. We need it because otherwise we end up reporting derived errors -- errors that are because ty_error doesn't match some other expected type. But if we saw a ty_error, we know we already reported an error about it, so we don't need to report lots more errors and overwhelm the programmer.

#4736 was because there was code that needed to execute to fill in certain data structures in error cases -- solely so we can keep typechecking and try to report more errors -- but it was getting skipped, reflecting an assumption that a certain error was fatal when it wasn't. At the same time, there were some missing checks for ty_error, so derived errors were getting reported.

But then I started thinking that I wanted to make the handling of ty_error more principled, so I changed the bool return value to a three-valued type that I called Outcome. An Outcome is either a ProperType (no errors), a BotType (saw something that diverges, which is not necessarily an error) or a TypeError (saw a type error). So in addition to the types that get written to tables as a side effect, typechecker functions now return an Outcome. This works, and is hopefully going to make it less likely for more derived errors to get reported accidentally.

But then I thought: why do we need a separate return flag in the first place? No function should ever return a type with a sub-component that is ty_error -- in that case, it should just return ty_error. For ty_bot, it's a little more complicated: for example, the expression true || fail!() doesn't always diverge (in fact, it never does) even though one of its components does. Nonetheless, I think if an expression isn't known to diverge, we don't need to know if any of its subcomponents are known to diverge.

The reason why the flag needs to be there right now is that the typing rules (as manifested in the typechecker) are non-compositional. We need the flag because we need to track information about an expression that isn't reflected in its type. But I think that's a bad code smell.

So I'm going to take a stab at making the rules compositional, which means that typechecker functions will return unit instead of bool or Outcome. This shouldn't be hard, because the internal representation of types already has a flag that says whether it contains ty_error -- so I can just make the constructors smart and return ty_error for anything whose representation would contain ty_error (like ~[ty_error]). It's a little harder to handle ty_bot correctly, but it shouldn't be too much harder.

comments

February 20, 2013 05:56 AM

I've submitted a proposal to the Open Source Bridge conference to give a tutorial talk on Rust. I can still edit the proposal between now and March 2, so I'd welcome any comments or suggestions about it!

comments

February 17, 2013 01:36 AM

If you do anything with web development, you are probably well aware that Opera recently announced that it was ditching its Presto layout engine and switching to Webkit. The reception of the blogosphere to this announcement has been decidedly mixed, but I am disheartened by the news for a very simple reason. The loss of one of the largest competitors in the mobile market risks entrenching a monoculture in web browsing.

Now, many people have attempted to argue against this risk by one of three arguments: that Webkit already is a monoculture on mobile browsing; that Webkit is open-source, so it can't be a "bad" monoculture; or that Webkit won't stagnant, so it can't be a "bad" monoculture. The first argument is rather specious, since it presumes that once a monoculture exists it is pointless to try to end it—walk through history, it's easier to cite examples that were broken than ones that weren't. The other two arguments are more dangerous, though, because they presume that a monoculture is bad only because of who is in charge of it, not because it is a monoculture.

The real reason why monocultures are bad are not because the people in control do bad things with it. It's because their implementations—particularly including their bugs—becomes the standards instead of the specifications themselves. And to be able to try to crack into that market, you have to be able to reverse engineer bugs. Reverse engineering bugs, even in open-source code, is far from trivial. Perhaps it's clearer to look at the problems of monoculture by case study.

In the web world, the most well-known monoculture is that of IE 6, which persisted as the sole major browser for about 4 years. One long-lasting ramification of IE is the necessity of all layout engines to support a document.all construct while pretending that they do not actually support it. This is a clear negative feature of monocultures: new things that get implemented become mandatory specifications, independent of the actual quality of their implementation. Now, some fanboys might proclaim that everything Microsoft does is inherently evil and that this is a bad example, but I will point out later known bad-behaviors of Webkit later.

What about open source monocultures? Perhaps the best example here is GCC, which was effectively the only important C compiler for Linux until about two years ago, when clang become self-hosting. This is probably the closest example I have to a mooted Webkit monoculture: a program that no one wants to write from scratch and that is maintained by necessity by a diverse group of contributors. So surely there are no aftereffects from compatibility problems for Clang, right? Well, to be able to compile code on Linux, Clang has to pretty much mimic GCC, down to command-line compatibility and implementing (almost) all of GCC's extensions to C. This also implies that you have to match various compiler intrinsics (such as those for atomic operations) exactly as GCC does: when GCC first implemented proper atomic operations for C++11, Clang was forced to change its syntax for intrinsic atomic operations to match as well.

The problem of implementations becoming the de facto standard becomes brutally clear when a radically different implementation is necessary and backwards compatibility cannot be sacrificed. IE 6's hasLayout bug is a good example here: Microsoft thought it easier to bundle an old version of the layout engine in their newest web browser to support old-compatibility webpages than to try to adaptively support it. It is much easier to justify sacking backwards compatibility in a vibrant polyculture: if a website works in only one layout engine when there are four major ones, then it is a good sign that the website is broken and needs to be fixed.

All of these may seem academic, theoretical objections, but I will point out that Webkit has already shown troubling signs that do not portend to it being a "good" monoculture. The decision to never retire old Webkit prefixes is nothing short of an arrogant practice, and clearly shows that backwards compatibility (even for nominally non-production features) will be difficult to sacrifice. The feature that became CSS gradients underwent cosmetic changes that made things easier for authors that wouldn't have happened in a monoculture layout engine world. Chrome explicitly went against the CSS specification (although they are trying to change it) in one feature with the rationale that is necessary for better scrolling performance in Webkit's architecture—which neither Gecko nor Presto seem to require. So a Webkit monoculture is not a good thing.

February 16, 2013 04:39 AM

If you are a regular user of DXR, you may have noticed that the website today looks rather different from what you are used to. This is because it has finally been updated to a version dating back to mid-January, which means it also includes many of the changes developed over the course of the last summer, previously visible only on the development snapshot (which didn't update mozilla-central versions). In the coming months, I also plan to expand the repertoire of indexed repositories from one to two by including comm-central. Other changes that are currently being developed include a documentation output feature for DXR as well as an indexer that will grok our JS half of the codebase.

February 15, 2013 06:37 PM

[This is an extended essay on the news out of Norway yesterday. See the closing for encouragement toward Opera and its fans, whatever the open source projects they choose to join, from me on behalf of Mozilla. /be]

Founder Flashback

I wrote about the founding of HTML5 in June, 2004, without dropping that acronym, mentioning only “Opera and others” as partners, because Apple was shy. Fragments of memory:

• @t was then working for Microsoft (he’s at Mozilla now, it’s great to have him) and sitting in front of me during the second day of the workshop.
• Vested interests touted XForms as inevitable (the UK Insurance Industry had standardized on it!).
• Browser vendors were criticized as everything from uncooperative to backward in resisting the XML Utopia.
• JavaScript took its lumps, including from Bert (whom I met years later at SXSW and got on fine with).

Hixie (pre-Google, already working on Web Forms 2) and Håkon of Opera joined David Hyatt of Apple, Mozilla’s @davidbaron and me at a San Jose pub afterward. There I uncorked and said something like “screw it, let’s do HTML5!”

We intended to do the work in the WHAT-WG. Håkon reminded us that the only likely path to Royalty-Free patent covenants remained the W3C, so we should aim to land specs there. He also correctly foresaw that Microsoft would not join our legal non-entity, rather would prefer the W3C.

So we drank a toast to HTML5.

Fast-Forward to 2013

In early January, I heard from an old friend, who wrote “Some major shift is happening at Opera, senior developers are being laid off, there’s a structural change to the company….” Other sources soon confirmed what became news yesterday: Opera was dropping Presto for WebKit (the chromium flavor).

One of my sources wrote about Mozilla and why it matters to him:

Smart people, and people I know and like, at that. Interesting work going on in [programming languages and operating systems]. Probably the last relatively independent web technology player (not something I cared about historically, but after leaving the Apple iOS fiefdom for the Google Android fiefdom, I find I’m just as locked in and feel just as monitored).

I hear this kind of comment more often lately. I think it’s signal, not noise.

Why Mozilla Matters

The January signal reminded me of mail I wrote to a colleague about “How Mozilla is different”:

1. We innovate early, often, and in the open: both standards and source, draft spec and prototype implementation, playing fair and revising both to achieve consensus.
2. Our global community cuts across search, device, social network businesses and agendas, and wants the end-to-end and intertwingled mix-and-match properties of the Web, even if people can’t always describe these technical properties precisely, or at all. What they want most is to not be “owned”. They seek user sovereignty.
3. We restored browser competition by innovating ahead of standards in user-winning ways. We’re doing it again, and with allies pulling other platforms up: Samsung is patching WebKit to have the B2G-pioneered device APIs.
4. Because we are relatively free of agenda, we can partner, federate, integrate horizontally instead of striving to capture users in yet another all-purpose vertical silo.
5. We advance the vision of you owning your own experience and data, attached to your identity, with whatever services you choose integrated. Examples include the Social API and the new Firefox-sync project that outsources storage service via DropBox, Box, etc.

The last point is hugely important. It’s why we have had such good bizdev and partnering with Firefox OS. But it is also why we have a strongly trusted brand and community engagement: people expect us to be an independent voice, to fight the hard but necessary fight, to be willing to stand apart some times.

I see these as critical distinguishing factors. No competing outfit has all of these qualities.

Thoughts on WebKit

I read webkit-dev; some very smart people post there. Yet the full WebKit story is more complex and interesting than shallow “one WebKit” triumphalist commentary suggests.

First, there’s not just one WebKit. Web Developers dealing with Android 2.3 have learned this the hard way. But let’s generously assume that time (years of it) heals all fragments or allows them to become equivalent.

WebKit has eight build systems, which cause perverse behavior. Peel back the covers and you’ll find tension about past code forks: V8 vs. Apple’s JavaScriptCore (“Nitro”), iOS (in a secret repository at apple.com for years, finally landing), various graphics back ends, several network stacks, two multi-process models (Chrome’s, stealth-developed for two years; and then Apple’s “WebKit2″ framework). Behind the technical clashes lie deep business conflicts.

When tensions run high the code sharing is not great. Kudos to key leaders from the competing companies for remaining diplomatic and trying to collaborate better.

Don’t get me wrong. I am not here to criticize WebKit. It’s not my community, and it has much to offer the world on top of its benefits to several big companies — starting with providing a high-quality open-source implementation of web standards alongside Mozilla’s. But it is not the promised land (neither is Mozilla).

If you are interested in more, see Eric Seidel’s WebKit wishes post for a heartfelt essay from someone who used to work at Apple and now works (long time on Chrome) at Google.

My point is that there’s no “One WebKit”. Distribution power matters (more below near the end of this essay). Conflicting business agendas hurt. And sharing is always hard. There are already many cooks.

In software, it’s easier in the short run to copy design than code, and easier to copy code than share it. In this spirit the open-source JS engines (two for WebKit) have learned from one another. Mozilla uses JSC’s YARR RegExp JIT, and we aim to do more such copying or sharing over time.

Web Engine Trade-offs

Some of you may still be asking: beyond copying design or sharing pieces of code, why won’t Mozilla switch to WebKit?

Answering this at the level of execution economics seems to entail listing excuses for not doing something “right”. I reject that implied premise. More below, but at all three levels of vision, strategy, and certainly execution, Mozilla has good reasons to keep evolving Gecko, and even to research a mostly-new engine.

But to answer the question “why not switch” directly: the switching costs for us, in terms of pure code work (never mind loss of standards body and community leverage), are way too high for “switching to WebKit” on any feasible, keep-your-users, current-product timeline.

XUL is one part of it, and a big part, but not the only large technical cost. And losing XUL means our users lose the benefits of the rich, broad and deep Firefox Add-ons ecosystem.

In other words, desktop Firefox cannot be a quickly rebadged chromium and still be Firefox. It needs XUL add-ons, the Awesome Bar, our own privacy and security UI and infrastructure, and many deep-platform pieces not in chromium code.

Opera, as a pure business without all of XUL, the Mozilla mission, and our community, has far lower technical switching costs. Especially with Opera desktop share so low, and Opera Mini as a transcoding proxy that “lowers” full web content and so tends to isolate content authors from “site is broken, blame the browser” user-retention problems that afflict Opera’s Presto-on-the-desktop engine and browser.

Nevertheless, my sources testify that the technical switching costs for Opera are non-trivial, in spite of being lower relative to Mozilla’s exorbitant (multi-year, product-breaking) costs. This shows that the pure “business case” prevailed: Opera will save engineering headcount and be even more of a follower (at least at first) in the standards bodies.

The Big Picture

At the Mozilla mission level, monoculture remains a problem that we must fight. The web needs multiple implementations of its evolving standards to keep them interoperable.

Hyatt said pretty much the last sentence (“the web needs more implementations of its evolving standards to keep them interoperable “) to me in 2008 when he and I talked about the costs of switching to WebKit, technical and non-technical. That statement remains true, especially as WebKit’s bones grow old while the two or three biggest companies sharing it struggle to evolve it quickly.

True, some already say “bring on the monoculture”, imagining a single dominant power will distribute and evergreen one ideal WebKit. Such people may not have lived under monopoly rule in the past. Or they don’t see far enough ahead, and figure “après moi, le déluge” and “in the long run, we are all dead.” These folks should read John Lilly’s cautionary tumblr.

@andreasgal notes that the W3C tries not to make a standard without two or more interoperating prototype implementations, and that this now favors independent Gecko and WebKit, since the only other major engine for WebKit to pair with is Microsoft’s Trident. We shall see, but of course sometimes everyone cooperates, and politics makes strange bedfellows.

I expect more web engines in the next ten years, not fewer, given hardware trends and the power wall problem. In this light, Mozilla is investing not only in Gecko now, we are also researching Servo, which focuses on the high-degree parallel (both multicore CPU, and the massively parallel GPU) hardware architectures that are coming fast.

If we at Mozilla ever were to lose the standards body leverage needed to uphold our mission, then I would wonder how many people would choose to work for or with us. If Servo also lacked research partners and good prospects due to changing technology trends, we would have more such retention troubles. In such a situation, I would be left questioning why I’m at Mozilla and whether Mozilla matters.

But I don’t question those things. Mozilla is not Opera. If we were a more conventional business, without enough desktop browser-market share, we would probably have to do what Opera has done. But we’re not just a business, and our desktop share seems to be holding or possibly rising — due in part to the short-term wins we have been able to build on Gecko.

Future Web Engines

So realistically, to switch to WebKit while not dropping out of the game for years, we would have to start a parallel stealth effort porting to WebKit. But we do not have the hacker community (paid and volunteer) for that. Nowhere near close, given XUL and the other dependencies I mentioned, including the FFOS ones. (Plus, we don’t do “stealth”.)

The truth is that Gecko has been good for us, when we invest in it appropriately. (There’s no free lunch with any engine, given the continuously evolving Web.) We could not have done Firefox OS or the modern Firefox for Android without Gecko. These projects continue to drive Gecko evolution.

And again, don’t forget Servo. The multicore/GPU future is not going to favor either WebKit or Gecko especially. The various companies investing in these engines, including us but of course Apple, Google, and others, will need to multi-thread as well as process-isolate their engines to scale better on “sea of processors” future hardware.

There’s more to it than threads: due to Amdahl’s Law both threads and so-called “Data Parallelism”, aka SIMD, are needed at fine grain in all the stages of the engine, not just in image and audio/video decoding. But threads in C++ mean more hard bugs and security exploits than otherwise.

I learned at SGI, which dived into the deep end of the memory-unsafe SMP kernel pool in the late ’80s, to never say never. Apple and Google can and probably will multi-thread and even SIMD-parallelize more of their code, but it will take them a while, and there will be productivity and safety hits. Servo looks like a good bet to me technically because it is safer by design, as the main implementation language, Rust, focuses on safety as well as concurrency.

Other approaches to massively parallel hardware are conceivable, and the future hardware is not fully designed yet, so I think we should encourage web engine research.

Strategic Imperatives

Beyond these considerations, we perceive a new strategic imperative, both from where we sit and from many of our partners: the world needs a new, cleanish-slate, safer/parallel, open-source web engine project. Not just for multicore devices, but also — for motivating reasons that vary among us and our partners — to avoid depending on an engine that an incredibly well-funded and lock-in-prone competitor dominates, namely WebKit.

Such a safer/parallel engine or engines should then be distributed and feed back on web standards, pushing the standards in more declarative and implicitly parallelizable directions.

Indeed more is at stake than just switching costs, standards progress, and our mission or values.

One pure business angle not mentioned above is that using an engine like WebKit (or what is analogous in context, e.g., Trident wrappers in China) reduces your product to competing on distribution alone. Front end innovations are not generally sticky; they’re in any event easy to clone. Consider Dolphin on Android. Or observe how Maxthon dropped from 20+% market share to below 1% in a year or so in China. Observe also the Qihoo/360 browser, with not much front end innovation, zooming from 0 to 20% in the same time frame.

Deep platform innovations can be sticky, and in our experience, they move the web forward for everyone. JS performance is one example. Extending the web to include device APIs held back for privileged native app stacks is another. So long as these platform innovations are standards-based and standardized, with multiple and open source implementations interoperating, and the user owns vendor-independent assets, then “quality of implementation and innovation” stickiness is not objectionable in my view.

Back to Mozilla. We don’t have the distribution power on desktop of our competitors, yet we are doing well and poised to do even better. This is an amazing testimony to our users and the value to them of our mission and the products we build to uphold it. In addition to Firefox on the desktop, I believe that Firefox OS and Firefox for Android (especially as a web app runtime) will gain significant distribution on mobile over the next few years.

Take Heart

So take heart and persevere. It is sad to lose one of the few remaining independent web platforms, Presto, created by our co-founder of HTML5 and the unencumbered <video> element. I hope that Opera will keep fighting its good fight within WebKit. Opera fans are always welcome in Mozilla’s community, at all levels of contribution (standards, hacking, engagement).

We don’t know the future, but as Sarah Connor carved, “No fate but what we make.” Whatever happens, Mozilla endures.

/be

Other voices:

• Christian Heilmann
• Robert O’Callahan

February 15, 2013 07:35 AM

Today I more or less only worked on #4678, and I finally got stage1 to compile! Of course, it doesn't work yet. So far, I've changed the ty param list in a trait item to be a non-empty list, so that it's obvious which param is the self type parameter. That's the point of the issue, making the Self type in a trait just like any other type parameter (because that's what it is, and this gets rid of some needless code). But, I haven't updated resolve yet to resolve the Self type to the first ty param. At least my build's failing in the way I expect it to.

And, unbelievably, I'm not done with one-tuples either; I got the compiler to compile and the whole test suite to pass except... the one test case that I added for one-tuples, which fails to parse in the exact same way it did in the first place. sadtrombone.wav

comments

February 14, 2013 02:20 AM

Over the weekend I'd been banging on #4183 without much insight. I tried to say everything I knew (which wasn't much) in comments on the issue; this morning I talked to Niko on IRC and as a result of what he said, I decided that #4678 should be fixed first.

The issue in 4183 had to do with the self type in a supertrait bound not getting the type parameter substitutions from the subtrait applied to it (in the code example there, the self type should be T but instead it was getting treated as unsubstituted self, causing an error message about T not matching self). #4678 says we should refactor the code so that the self type is no longer special, but rather, treated (internally to the compiler) as just another type parameter. That's not all that needs to be done -- also, we need to carry around each set of parameter substitutions in a chain of supertrait/subtrait relationships, otherwise T would still be unbound when checking the supertrait bound FuzzyEq<T>. But since #4678 needs to be done anyway, I think it'll be easier to do it first.

I couldn't quite stomach taking that on, though, so instead I went back to my branch where I was trying to finish removing macro code that generates code with structural records. There was a last bit of code in the pipes compiler did this, so I took a stab at removing it, which requires more advanced macrology than I've ever used in Rust before (which is to say, any macrology at all). I ran into a problem with, of all things, one-tuples. That's right, a tuple with only one field. Why would you ever want that? Well, there's a saying that whenever you ask that question in PL, the answer is macros, and in this case, it's macros. It doesn't seem like we parse tuple patterns with a single field properly, so I made a separate branch to just isolate that issue.

And that was Monday.

comments

February 12, 2013 04:02 AM

Today, I overslept and missed a research talk by Tucker Taft, who was visiting today, on his language ParaSail. Oops!

...But fortunately, Tucker stayed around long enough to chat with the Rust team, and then was kind enough to give me a rehash of part of the talk one-on-one. ParaSail sounds like a pretty interesting language, with some things in common with Rust but more of a slant towards implicit parallelism rather than explicit. The compiler infers opportunities for parallelism in most cases, but the language also includes annotations that a particular loop in the code must be executable in parallel (meaning that if there are any ordering dependencies, it's a compile-time error). That's a nice idea, and reminds me a little of work I wanted to do in grad school (but didn't get to) on performance annotations.

ParaSail also has an ambitious precondition/postcondition language that's as powerful as the programming language itself (because it's the same language), which backs off to dynamic checks in cases where it's not possible to check a particular precondition within a reasonable amount of time statically. It'll be interesting to see if this eventually turns into a compiler that drops you into a theorem prover in cases where it can't prove your assertions automatically.

And as a result, didn't get much of anything done on Rust today, but I guess that's what the weekend is for!

comments

February 09, 2013 06:07 AM

Triaged some bugs. It's now the case that I've reviewed open bugs 1-2000 in the past two weeks. That sector of the issue tracker is in pretty good order! Lots of the bugs that are still open are enhancements, the others mostly look pretty challenging. So it's not like we have a lot of old bugs we've rejected for no reason.

Following a FIXME, I tried to do what I thought would be a simple refactoring in resolve, and messed up trait method type parameters thoroughly. The usual.

Removed some more structural records, now that the Feb. 4 snapshot is finally live! But, it looks like I need yet another snapshot to remove the last of the structural records from the pipes macros. What's more, I don't understand the pipes compiler well enough to fix that, though I've been staring at it for quite some time today.

And that was today.

comments

February 08, 2013 04:12 AM

Snapshot: still not quite done yet! Rarrr. Suddenly, an obscure problem appeared as the result of the combination of my snapshot and another commit, probably involving borrow checking. That commit got backed out, Niko's trying to figure out what the problem is, and the results are still building.

In the meantime I'm trying to make tuple-like struct constructors work cross-crate, which involves diving back into metadata reading and parsing. Which is my favorite part of the compiler. Oh, no, not "favorite", the other one...

Slow day. I also learned that if I want to get to the office on time for the weekly 9 AM Rust meeting, I should really leave home *more* than an hour before.

comments

February 06, 2013 01:53 AM

I'm glad people got something out of my previous post! I would like to write more posts like that, but they take *a long time* to write. It's more important to me to make sure I post regularly about work than to post the highest-quality content than I can, so I usually stick with this "what did I do today?" format. Still, I'll try to occasionally write a longer post to mix it up (and you can always ask me a question about Rust, here or on Twitter; it just might inspire me).

The snapshot I tried to build over the weekend failed to build on Mac and Linux, and I'm glad I decided to just set that aside for the rest of the weekend, because I never would have figured out the problem on my own. The problem relates to two fun aspects of Rust: the compiler is self-hosting, and on Mac and Linux, we cross-compile so as to generate binaries for both 32-bit and 64-bit machines. That's why the build only failed on Mac and Linux.

Often when we're trying to remove a feature from the language (I was -- structural records), we'll add #[cfg(stage0)] attributes -- same for 1, 2, and/or 3 -- so as to have different versions of the same piece of code that gets used in different compiler "stages". Stage 0 is the snapshot, stage 1 is the result of building the compiler from a new source code tree using the snapshot, stage 2 is the same as stage 1 except building with the stage1 compiler (on the same source code tree) instead of the snapshot, and so on. (So it's *not* actually turtles all the way down.) Also, macros and staging interact in "fun" ways and so it's particularly hard to remove a feature when macros generate code that uses it. I was trying to change the Buffer type, which the pipes compiler (implemented as a syntax extension) uses, and I accidentally made the "old" version active in stage0 and stage1, and the new one only in stage2. What threw me off was that the FreeBSD and Windows snapshots built anyway -- but it turns out our build system interprets stages in some weird ways when we're cross-compiling.

Figuring that out was the painful part (though Graydon helped a lot) and then it was just a matter of following the type errors. Just another day of compiler janitor work (and I wouldn't want it any other way). Removing deprecated features is one of those things they don't (usually) teach you about in grad school.

As soon as the snapshot finishes building, I think that should allow me to banish structural records completely from the core library! (Which is only the first step to removing them completely; the compiler still uses them.) I actually submitted a pull request earlier that removes the last structural records from the standard library, which doesn't require the snapshot. Our shiny new build automation robot will, I hope, merge it in due course.

comments

February 05, 2013 03:25 AM

On Twitter, mcclure111 asked -- and I'm wildly paraphrasing here, since apparently it's impossible to search for a tweet that's more than a week old:

"Research languages seem to focus on only one question. Rust focuses on ownership types. I'm not sure if they're useful or not. They 'deform the code' and I don't know whether it's worth it."

It amuses me to see people looking at Rust and concluding that it focuses on ownership types, because when I started working on Rust, less than two years ago, it didn't have ownership types at all. When I started on the Rust team as an intern, I thought that Rust was centrally about investigating the usefulness of typestate. (In Rust, typestate was a system of assertions that were visible to the typechecker but relied on dynamic checks for soundness. We removed it long ago.)

But later, the team decided to get rid of typestate, and add ownership types. I think the decision to add ownership types happened during the three months in between the end of my internship and the start of my full-time employment at Mozilla, and I didn't pay a lot of attention while I was on leave. But, my understanding is that the decision got made not because anybody said "I really want to do research on ownership types", but because all of the other options had been exhausted. Way, way back in the day, Rust had neither automatic GC nor ownership types. It only had automatic reference counting (that the compiler manages for you), which necessitated a cycle collector that never quite got finished. But ref-counting everything just added too much overheard. Everyone figured that GCing everything would also add too much overheard. So that left what's known in the programing languages literature as "regions", though we've come to call them "lifetimes" instead.

Also, calling Rust a research language is funny to me because -- as its name reflects -- we've tried hard to avoid incorporating new technology into it. We haven't always succeeded at failing to be novel, but we have a rule of thumb of not including any ideas in the language that are new as of the past ten years of programming language research. The field of programming language is full of old technology that hasn't been put to use in solving problems that it's exactly suited for. The goals with Rust were to avoid reinventing wheels, and see what the past had to teach us. I can't blame anyone for thinking Rust is a research language, though, since it is being developed by Mozilla Research.

But anyway, back to ownership types. Sometimes, people new to Rust say that having three different sorts of pointers is complicated, and do we really need to have that? There was a recent thread on the rust-dev mailing list about whether Rust's syntax is "intimidating" for newcomers, and a large part of that discussion was about the syntax for ownership types. Part of the source of the original confusion had to do with explicit type declarations that weren't really necessary, that were only present to work around a bug in the compiler (that got fixed later). It's hard to say much more without an example, so here is one.

( Cut for code, and lengthy discussion )

comments

February 03, 2013 08:28 PM

We're down to ten pending pull requests, which is the lowest it's been in a while!

It's hard to know what to write today, since it seems like I spent most of the day merging pull requests. Or trying to. Merging a pull request: if it looks OK from reading it on github, and it merges cleanly, I just click merge. If I'm worried, I pull it as a new branch and run the testsuite on my machine. Several times today, a test failed, which means someone didn't run the full testsuite (understandable since it can take an hour...) or else there was a conflict between their code and another recent change. I decided today I really wanted to clear out pull requests, so I would fix things myself instead of asking the submitter to fix the problem and submit a new pull request. I am particularly proud of merging this library fix, for which I had to read the comments several times just to figure out what was needed. (I blame sleep deprivation, not the commenters.)

Also, I worked a little bit on this bug relating to our syntactic sugar for for loops. The syntactic sugar is very nice, but in this case it leads to a confusing error message that takes some thought as to how to get straight (due to the baroqueness of our for construct).

comments

January 24, 2013 03:59 AM

The original purpose of DXR, back when the "D" in its name wasn't a historical artifact, was to provide a replacement for MXR that grokked Mozilla's source code a lot better. In the intervening years, DXR has become a lot better at being an MXR replacement, so I think that perhaps it is worth thinking about ways that DXR can start going above and beyond MXR—beyond just letting you searched for things like "derived" and "calls" relationships.

The #ifdef problem

In discussing DXR at the 2011 LLVM Developers' Conference, perhaps the most common question I had was asking what it did about the #ifdef problem: how does it handle code present in the source files but excluded via conditional compilation constructs? The answer then, as it is now, was "nothing:" at present, it pretends that code not compiled doesn't really exist beyond some weak attempts to lex it for the purposes of syntax highlighting. One item that has been very low priority for several years was an idea to fix this issue by essentially building the code in all of its variations and merging the resulting database to produce a more complete picture of the code. I don't think it's a hard problem at all, but rather just an engineering concern that needs a lot of little details to be worked out, which makes it impractical to implement while the codebase is undergoing flux.

Documentation

Documentation is an intractable unsolved problem that makes me wonder why I bring it up here…oh wait, it's not. Still, from the poor quality of most documentation tools out there when it comes to grokking very large codebases (Doxygen, I'm looking at you), it's a wonder that no one has built a better one. Clang added a feature that lets it associate comments to AST elements, which means that DXR has all the information it needs to be able to build documentation from our in-tree documentation. With complete knowledge of the codebase and a C++ parser that won't get confused by macros, we have all the information we need to be able to make good documentation, and we also have a very good place to list all of this documentation.

Indexing dynamic languages

Here is where things get really hard. A language like Java or C# is very easy to index: every variable is statically typed and named, and fully-qualified names are generally sufficient for global uniqueness. C-based languages lose the last bit, since nothing enforces global uniqueness of type names. C++ templates are effectively another programming language that relies on duck-typing. However, that typing is still static and can probably be solved with some clever naming and UI; dynamic languages like JavaScript or Python make accurately finding the types of variables difficult to impossible.

Assigning static types to dynamic typing is a task I've given some thought to. The advantage in a tool like DXR is that we can afford to be marginally less accurate in typing in trade for precision. An example of such an inaccuracy would be ignoring what happens with JavaScript's eval function. Inaccuracies here could be thought of as inaccuracies resulting from a type-unsafe language (much like any C-based callgraph information is almost necessarily inaccurate due to problems inherent to pointer alias analysis). The actual underlying algorithms for recovering types appear known and documented in academic literature, so I don't think that actually doing this is theoretically hard. On the other hand, those are very famous last words…

January 21, 2013 05:57 PM

It's good to be able to go home on a Friday night having submitted two pull requests: one for #3979 and one for #3860.

I've been working on these two minor bugs for way too long. I guess #3979 isn't all *that* minor, though. It was an aspect of the interaction between default methods and supertraits that was left incomplete -- fixing it required some thought. I wasn't sure at first what was already implemented. It turned out supertraits weren't transitive, plus there was no way to figure out in the compiler what the impl is of a given trait for a given type. It was really easy to make the coherence checking pass spit out a table mapping trait IDs to tables mapping types to impls -- the rest wasn't too bad (except for a hairy bit of code in ty that computes the transitive supertraits for a given list of trait bounds).

I didn't spend as much time on #3860, which involved borrowck and trans having divergent notions of which nodes in the AST introduced new scopes. My first solution was to change trans to introduce a new scope for each statement, because that seemed easier. Unfortunately, this meant creating lots of extra basic blocks and it was a performance loss. So it looked like it would make more sense to change borrowck, which is a static analysis so that's not going to directly mess up performance. My fix is kind of awkward (and best understood by looking at the code), but it works.

I still have more time to make up by working over the weekend, so there's no rest for the weary.

comments

January 19, 2013 05:20 AM

A short blog post to let everyone in the Mozilla community know the latest news.

As the “About Brendan” sidebar says, I’m a founder of Mozilla, and since 2005 I have had the title of CTO. That job has entailed work on technical strategy, Web standards, influencing/partnering, Mozilla Research, public speaking, and talent-scouting. (Oh, and I still code.)

I’m happy to announce that as CTO I will now also be managing our product and platform engineering teams. This is a privilege and an obligation I take seriously. I look forward to working even more with community members including Mozilla’s engineering staff to advance the Open Web on all platforms, especially the mobile ones rising to overshadow desktop computers.

Mozilla is many things: mission-based organization, community of hackers and supporters, social movement representing the people using the Web above all other agendas. To fulfill our mission, we must serve users with excellent products. Successful products require great design and engineering, and also (increasingly) partnering.

We can’t fulfill our mission without winning products. We can’t have winning products without focus on the technology trends, including the rise of mobile, and also of multicore and GPU processing, and yet more laser-like focus on serving the people who use our products.

The watch-word for me is therefore “focus”. Competition and partnering have focused Mozilla in the last few years, and all to the good in my view. We are on the right track with Firefox OS, Firefox for Android, Firefox on the desktop, and our other initiatives (two to name here: Persona and Web Maker). We need to continue, with even sharper focus on what’s in front of us.

In particular, on mobile this means not just great user interfaces and fast, smooth performance. It also means, as I said to @stshank, expanding up the stack to fight proprietary lock-in that diminishes developer and user experience. We did it with Firefox in 2004, we can do it again with Firefox OS, Firefox for Android, Persona and beyond.

I remain the buck-stopper for Mozilla module ownership and other technical delegated decision-making disputes. Mitchell remains governance conflict buck-stopper and Chief Lizard-Wrangler.

/be

January 17, 2013 02:02 AM

This was one of those "do work, realize own work is misguided, undo work, collect paycheck" days. When I was working on #3979, I made the mistake of trying to clean up some bad copies as part of the same change. I made the mistake of thinking this would make it "simpler" to finish my change. Instead, I got the code to compile and got a completely mysterious type error when building libcore with stage 1.

So now I'm pulling out all my bad-copy fixes so I can just finish the fix for #3979, commit it, and then get to the bad copies. You'd think I'd have learned in third grade to only work on one thing at a time, but I didn't go to third grade.

I also finished my fix for #3860, and tested it, but have to do some cleanup before I submit a pull request. #3860 was a bug having to do with borrowck and trans (codegen) being inconsistent with each other, so fixing either one could address the problem. trans looked easier to change, so I changed it, but my change resulted in inserting lots more LLVM basic blocks, which turned out to be a performance regression. Changing borrowck really makes more sense since it's just refactoring a static analysis pass and shouldn't have any runtime effect, but I was afraid to touch the borrowck code before. I put aside my fear and implemented a mildly ugly hack (keeping a separate table for trans to use that gives borrowed pointers less-precise lifetimes, meaning trans doesn't have to assume a given statement has a scope attached to it), which worked. The part I have to clean up is making borrowck compute the table of which node IDs refer to statements -- rather than having region inference do it and pass it to borrowck -- which is totally straightforward.

comments

January 15, 2013 02:11 AM

While doing bug triage yesterday, I got sucked into the black hole of fixing bad copies. Actually, it wasn't that hard once I got the hang of it, but there was some fumbling involved.

To give some more context, Rust has three different kinds of pointers: ~, pronounced "owned", @, pronounced "managed", and &, pronounced borrowed. So ~int is an owned pointer to a machine int; @int is a managed pointer to a machine int; and ∫ is a borrowed pointer to a machine int. @ pointers are garbage-collected and can be freely copied, while ~ pointers are guaranteed by the type system to always have a single "owner", meaning they can be freed at the end of the scope associated with the owner. Owned pointers can be "borrowed" as long as the span of time of the borrow can be statically guaranteed to be a sub-interval of the owner's lifetime.

Pushing so much of the memory model into the type system lets you program in Rust in a way that lets you relax and know that the typechecker will likely catch mistakes involving undesired copying. By declaring data as type ~T for any type T, you're saying you don't want it to be copied (unless you really mean to). But when you don't care whether your pointers get copied or not, you can use @T and copy to your heart's content.

Strings and vectors also come in three flavors, just like the ones for pointers.

I think this is all totally neato, but it was added to Rust fairly late in the language's history. So in the Rust compiler, there's lots of awkward code that copies stuff around for no good reason, except the historical reasons that for a while, there was only a ~str type (and in truth, @str still isn't well-supported); and also, references (borrowed pointers) weren't first-class in the past. Now that these restrictions have been lifted, it should be straightforward to update the code to pass non-copyable types by reference and to use @-vectors and @-strings for references that we don't mind copying. Still, it's easy to make one change and find that if you propagate changes naïvely, you're changing one of the most basic data types in the compiler (I found this out when inadvertently, a consequence of a change I made was changing idents to @strs. I backed that out pretty quick).

I finally got to a set of changes that still has a few copies, but less than where I started. Hopefully I've learned something from this adventure and will be able to progress more quickly later on.

comments

January 12, 2013 05:14 AM

Congratulations to Marcos Caceres, Yehuda Katz, Alex Russell, and Anne van Kesteren on the news of their election to the W3C Technical Architecture Group (TAG).

This is great news: four out of the five reformers won.

Back-story: in late 2010, TBL invited me to join the TAG. I declined with thanks for two reasons:

1. I was (at the time, especially) over-committed to standards (JS, mostly) and Mozilla work (e.g., setting up Mozilla Research).
2. The TAG was not engaged “toothfully” (my word) on Web problems faced by developers, rather it seemed focused on XML and Semantic Web matters, and therefore I would be odd-TAG-member-out.

Having both (a) more time and (b, the point of this story) three or four kindred-spirit candidates would have changed my mind. I hope my decline-with-thanks message helped in some small way to bring about today’s good news.

Kudos to the reformers for running, to the W3C Advisory Committee representatives who voted in the reformers, and to the W3C and TBL for being open to reform.

/be

January 11, 2013 07:54 AM

Today's triumph: using git bisect (for the first time ever!) to find out why the build was broken, then fixing it. I didn't fix the underlying bug, but at least the build won't be on fire now. I hope.

Otherwise: bug triage. It's a thing.

comments

January 11, 2013 06:05 AM

I'm very excited to announce that I've started a new blog for writing about the research and research-related program activities that have taken over this journal in the last few years. Also, I'm excited to have a domain that's less embarrassing than rockstargirl.org.

I hesitate a bit in calling it a "research blog" -- that sounds so bloodless. Rather, I want it to be a blog about my experiences as a researcher, which necessarily include research but also include a lot of other things. If you've been reading my research posts here, I hope you'll consider reading there, too.

January 10, 2013 11:55 PM

I finished one pull request, which fixes 3563 and got most of the way along to finishing another, that one for issue 3979.

Both issues involve default methods, which are a new feature that's not quite baked yet (so much so that for 0.5, we turned it off by default). #3563 had a couple of test cases attached; in the simplest one, calling a non-existent method on self inside a default method would cause an internal compiler error during typechecking. This turns out to be because the typechecker assumes that anything with the type self is the value self (a special name that refers to the self object inside either an implementation of a trait, or a default method in a trait). The simplest fix for this I could see was to treat self as if it has dynamic scope, just for typechecking purposes. That is, instead of throwing away the self ID from the enclosing scope when typechecking a nested closure, keep the same one, so closures inside default methods can refer to self. This seems a little bit dirty (as if it might enable out-of-scope uses of self), but I *think* it's okay because in the Rust compiler, name resolution is a separate pass from typechecking, and happens first. So if you tried to do something fishy with self, resolve would catch it before typechecking ever happened.

Issue 3979 showed that you couldn't call a supertrait method inside a default method. After quite a bit of fumbling, I fixed the test case shown in the issue report, but while working on it I noted to myself that I should check at least two more cases (supertraits with type parameters, and the case where you define the traits in a separate crate from the impls of them). Once I wrote those two test cases and ran them, it turned out neither of them works. So I can't submit this one quite yet (in good conscience).

comments

January 08, 2013 04:04 AM

Continued from part 1!

As June came to an end, I went to back-to-back Mountain Goats concerts -- John Darnielle solo shows, really -- on the 27th and 28th, bringing Alex with me for the first (it was his first Mountain Goats show) and my friend Emily for the second (her first, as well). Because I lived in Mountain View, I spent twice as much time getting to and from these shows on public transit than I spent actually at the shows, but it was worth it, and I was really happy to get to give Alex and Emily a proper introduction to a band I've loved for a long time. Neither show quite left my head for months. (LMA has the first night's recording, which happened to be the night that JD played not one but two songs about professional wrestling.)

July arrived, and Ryan and I made a final push to finish up our POPL submission on time, with the help of many friends and colleagues who read drafts of it for us. Ken Shan spotted a couple of particularly subtle bugs.

Just before the deadline, a friend of Ryan's pointed us to some related work we hadn't seen yet: a very recent tech report from the group working on the Bloom language at Berkeley. Then, it turned out that my Mozilla colleague Niko had a friend who had worked on Bloom. Niko put me in touch with his friend, I started talking to the Bloom folks, various enthusiastic emails were exchanged, and I ended up being invited to go visit Berkeley and give a talk in August.

With the POPL deadline behind me, I took two weeks off from Mozilla to attend OPLSS in Eugene, Oregon, an annual two-week-long summer school that draws PL folks from around the world. The lectures were intense, mind-bending, and entertaining, and I loved hanging out with the people at OPLSS.

But it was at around this point in the summer that I began to feel pulled apart at the seams, because there were so many things I had to do and they all seemed critically urgent. Although our paper was now done, I had returned to trying to fix the remaining bug in our determinism proof, so that Ryan and I could put up the tech report that was supposed to accompany the paper. I felt guilty for not doing the OPLSS homework -- after attending the lectures all day, whatever energy I had left was going toward working on the proof instead. On the other hand, I felt equally guilty for being at OPLSS at all instead of back at Mozilla hacking on traits, which had gone rather quickly in my mind from being a somewhat exotic, nice-to-have feature to an essential feature that the language sorely needed. (Indeed, Niko's type inference code was filled with plaintive comments about how the implementation could have been so much cleaner, if only we had traits.) And on top of everything else, I had to prepare for my upcoming talk at Berkeley.

After a few days at OPLSS, with all my various research and work obligations at war for my attention, I was a stressed-out wreck. Alex listened to my woes on the phone with characteristic patience and understanding. Another thing that helped was getting out of my dreary, isolating OPLSS dorm room (my roommate had been a no-show, so it was just me) and running on the beautiful network of trails around Eugene. Nevertheless, with all the other stuff I had to do, I began to fall behind on the 20-mile-a-week habit I would have had to keep up in order to make my 1000-mile goal for the year.

I began to make progress on the proof again; then, at the start of the second week of OPLSS, I got a big emotional boost: Ryan and I got official notification that our grant had been funded! This was pretty huge news for me: it meant that my plans for doing a Ph.D. with Ryan were starting to shape up in the way that I had hoped.

By the end of OPLSS, I had managed to isolate and finally really understand the renaming bug. I still hadn't managed to fix it properly, though, so Ryan and I decided to post an incomplete version of our tech report, and I went back to Mountain View with a vow to go back and fix it once my internship was over.

In August, back at Mozilla, I worked furiously to get at least some part of my traits proposal implemented before my internship ended. I had to give my final presentation on traits in my second-to-last week at Mozilla, before I had anything actually working yet; in it, I overoptimistically predicted that traits would be done by the 0.4 release. (Much of the work ended up being deferred to the 0.5 release, which just came out in December, and many bugs still remain to be addressed.)

But finally, on Monday of my last week, I successfully compiled and ran the first-ever Rust program with a Haskell-style default method. It was just a toy program, but I was pretty excited. I also felt heartened by all the enthusiasm I was seeing for traits, both on the team and in the Rust community beyond the core team. (As I said in my presentation, "I'm an academic -- I'm not used to working on things that people actually want.") For that matter, it was rather exciting that a Rust community beyond the core team could now actually be said to exist; that hadn't been the case back in 2011!

A few days before we had to head back to Indiana, I went to give my talk at Berkeley, where I had a great time chatting with Neil Conway, Joe Hellerstein and the rest of the Bloom and Daedalus folks. Their work on Bloom^L, which is an extension of Bloom, had turned out to be directly related to our work: what Ryan and I were doing for single-assignment parallel languages -- namely, parameterizing them by a user-specified lattice to give them a more flexible and general notion of monotonicity -- Bloom^L had already done for Bloom. Their goal was guaranteeing eventual consistency in distributed databases, rather than deterministic parallelism, but the math was largely the same.

Moreover -- and I might never have realized this, had we not come across the Bloom work -- the ideas about monotonicity that Ryan and I had been kicking around had already been in play in the distributed systems community for a while. The novel thing that I hope to be able to offer is a language-based account of those ideas, as part of a language-based approach to deterministic parallelism. But just because something looks like a language problem to me doesn't mean it's okay to succumb to PL myopia! We need to pay attention to what the distributed systems people are doing; it turns out they're working on really hard, interesting problems.

With our internships done, Alex and I flew back home to Indiana, again with cats in tow, and I spent a much-needed week doing absolutely nothing while Alex began teaching NLP to undergrads. On August 27th, we happily celebrated our first wedding anniversary.

The highlight of my September was a trip to Denmark, where I attended ICFP in Copenhagen, then spent a few days adventuring with Chris. While ICFP was going on, Ryan and I received the reviews for our POPL submission and wrote our response to the reviews. I had been nervous that the reviewers would spot the remaining hole in our proof, but my fear turned out to be unfounded -- none of the reviewers took issue with the technical development. Rather, their complaints were that we needed to motivate the paper better and provide more real-world examples of why a new approach to deterministic parallelism was called for, which were both fair points.

Considering that it was my first time submitting a paper to POPL, I was actually quite happy with the outcome: the reviewers found our idea to be interesting and potentially useful, pending more examples. But since it looked like our paper wasn't going to be accepted to POPL, we started thinking about resubmitting it to ESOP. I also began talking to people about potential applications for our work, and Matt Might suggested using LVars for parallelizing CFAs, so I began thinking a bit about how that might work. (I didn't get particularly far with the idea, but as it turns out, two of my favorite colleagues, Will Byrd and Andy Keep, are just about to join Matt's research group as postdocs, and there's a possibility that we'll be collaborating on it in the future, which would be very cool.)

At the end of September, Alex and I went to Florida for a few days to be part of his sister Natalie's wedding, and happily welcomed her new husband, Bryan, into our family.

On October first, Ryan and I were notified that our paper didn't get into POPL, so we spent the first half of the month readying our ESOP submission. Since the ESOP page limit is shorter, we had to condense and cut a lot of the material, which was difficult, but I think ultimately improved the paper. Aaron Turon helped by commenting extensively on a draft. It was also during this time that I finally fixed the renaming bug, after some conversations with Amr and a few emails exchanged with Neel Krishnaswami (who had offered to help when we'd talked at ICFP). We posted an updated version of our tech report with the finished proof.

With the determinism proof finally complete, I went back to thinking about the frame-rule-like property we'd had to show along the way. Something about it was nagging at me. In my experience, people used frame rules to show that, if a program only used certain resources when it ran, then the resources it didn't use remained the same before and after the run. That was a useful property, but it seemed like only half the story. A frame rule could also show that if a program only used certain resources and produced a given result, then running it again with additional resources present would produce the same result.

For us, it was that second guarantee that was particularly important, because "the same result" was what determinism was all about! When I asked about it on cstheory.stackexchange, both Aaron and Neel had illuminating responses, and I realized that there was more to the frame rule than I'd thought. Conveniently, Larry Moss had just asked me to give a talk at the IU logic seminar in November, so I used the classic "agree to give a talk about something as a way of motivating yourself to learn about it" trick and told Larry that I would give a talk about frame properties, adding a paper that Aaron had recommended to my reading list.

Once we'd sent our paper off to ESOP, Ryan suggested that while we were waiting to hear back, I could help out with some monad-par hacking. He asked me to do what he thought of as a fairly minor task, but when I tried to dig into it, I was utterly lost. Since I had barely written any Haskell in my life up to that point, I decided to back up and work on actually learning the language first, which was a bit embarrassing, since Ryan had assumed that I was already pretty fluent with Haskell.

By this point in the year, I was so far behind on my 1000-mile running goal that I more or less gave up on it entirely, although I continued to accrue points on Fitocracy for my bike commutes to campus. Alex, though, continued to run regularly as preparation for another marathon he had coming up in November (he actually did a total of three marathons in 2012). He also became enamored of these things on a weekend trip we made to Atlanta to attend yet another wedding.

I crossed one minor academic cheevo off my list in October: the Bloom^L paper by Neil Conway et al. was published at SOCC, marking the first time that my work on LVars has been cited by someone else. Finally, in October Larry Moss and Ken Shan both agreed to join Ryan and Amr as members of my dissertation committee, bringing it up to the required total of four members.

As November began, my efforts to get better at Haskell limped along. I had frustrating interactions where I would say that I was struggling with Haskell, and then another grad student would proceed to start 'splaining basic type theory to me. The types weren't the hard part; the purity, the laziness, and the parallelism were the hard part! Eventually, I begged off of working on the monad-par project so that I could get back to my own stuff. I also spent some time organizing events for CS Club, especially the second annual CS Club "So you want to go to grad school?" panel.

Over our Thanksgiving break, Alex and I made a road trip to Iowa to visit my parents and my grandpa Joe, my mom's dad. We spent a few days traipsing around Cedar Rapids, where I lived for a year growing up, and Grinnell, where I went to undergrad.

After Thanksgiving, I gave my logic seminar talk about separation logic and frame properties. While preparing for it, I chatted with Aaron a bit more, and he suggested that I take a look at a paper on "views" that was about to come out at POPL; he suspected there might be an interesting relationship between views and LVars. I began doing some more background reading on concurrent separation logic so that I'd be better able to make sense of the views paper.

Finally, at the end of the month, we got the reviews back from our ESOP submission and wrote our response. The reviewers had liked many things about the paper, but, just as with POPL, they wanted to see more convincing examples of what LVars were good for.

In December, while looking through some old emails, I realized that back in October, Neel Krishnaswami had independently recommended that I read the same paper on views that Aaron had pointed me to. (Dun-dun-DUNNNNNNNN.) I emailed Aaron and Neel to see if they were interested in working with me to investigate a connection between views and LVars, and they responded enthusiastically, inviting me to visit Saarbruecken right after POPL so that we could spend some time working together. There are a couple of other LVar-semantics ideas we're interested in pursuing, as well.

Needless to say, I'm incredibly excited about collaborating with Aaron and Neel. For one thing, I've been wanting to do something involving logical relations and LVars right from the start; for another, the fact that a frame property just fell out in the process of doing the LVar determinism proof suggests the possibility of some kind of separation logic for deterministic parallelism. (Also, I've never been to Germany, and I'm quite looking forward to visiting, especially since Ezgi Cicek alerted me to the existence of Flammkuchen.)

In mid-December, Ryan and I got a very nice rejection letter from the ESOP committee, in which they reiterated that they liked our idea but that they wanted more convincing examples. Although I'm of course disappointed that we haven't been able to publish our work yet, I think that the two rounds of reviewing we've been through have put us in a good place to move forward with it. The technical development is now quite solid, the idea is novel as far as we know, and people seem to want to be enthusiastic about it. But it's clear that we need to motivate the work better. Everyone knows that nondeterminism can be nasty, but why would someone want to use our thing instead of just using a single-assignment language? What kinds of programs are expressible (or conveniently expressed) with a monotonic-multiple-assignment language that would be impossible (or inconvenient) to express with a single-assignment language or some other existing deterministic parallel model? Those are the questions I think we need to address next time.

The semester wrapped up, concluding with Andy Keep's dissertation defense, and Alex and I skipped town for the holidays, going first to New York for an end-of-the-world party that some friends were throwing, and then to Tallahassee to spend a few days with his family, as has become traditional. All this was great fun, except that just as we got to NYC, I caught a terrible cold that destroyed my voice and just wouldn't go away. When we got back to Bloomington, Alex convinced me to see a professional, who prescribed four different medications. Believe it or not, this is the first time in my adult life that I remember having to take prescription drugs of any kind, and I was a little reluctant to do so, but they seem to have done the job. Still, it's only now, after nine days of drugs, that my throat is getting back to normal.

Finally, I submitted a talk proposal about the LVars work for the POPL student short talk session, and it was accepted, so it looks like I'll be giving a short presentation at POPL in addition to the, uh, even shorter one that I might be giving at PLMW. I have a bit of work to do to prepare for both of those before I head off to POPL (in Rome!) in a couple of weeks.

To sum up, I'm happy with how my 2012 went. I made a lot of progress on research this year. Getting our grant proposal accepted was a big win. I'm proud of having submitted my first paper to POPL, having finally gotten the LVar determinism proof done, and having given various talks and successfully reached out to various people about research collaborations. At Mozilla, I contributed to every release of Rust, and I helped bring traits into the language. In my personal life, my marriage continued to go wonderfully. Alex is still the best. The paper rejections were unfortunate, but that's part of academic life; there's not much I can really complain about.

In 2013, I'd like to do another marathon; I'm tempted to go back to Toronto and try that course again in May. Research-milestone-wise, I want to get our first LVar paper published and get another paper or two underway, and I'm hoping to propose my thesis relatively soon -- sometime this summer, perhaps. Alex is planning to propose this year, too, so he and I are even more or less synchronized, which is awesome. Although we decided we don't want to do any more internships before we graduate, I want to keep contributing to Rust as a volunteer when I can, and to do what I can to help other people contribute. And that's about it for me; thanks for reading! How about you?

January 07, 2013 02:03 AM

Today was largely spent trying to install a Windows VM on my Mac. Dual booting seemed like it would be more pain, but now I'm not sure. I wanted to be able to debug Windows issues locally, at long last, because they happen fairly often and saying it was someone else's problem would only take me so far. Mozilla has a license for VMWare, but I ended up not understanding what versions we have licenses for (or whatever) and installing VirtualBox instead, which was free. That was somewhat painless. Then I installed Windows 8 inside the VM, which was surprisingly easy as well (once I told VirtualBox that I was installing a 64-bit Windows). Unfortunately, once I installed MinGW and MSys (which we need in order to build Rust, at least at the moment), things fell apart. Typing "ls" or "mkdir whatever" at the shell prompt resulted, apparently, in a hang (ls doesn't normally take minutes in a directory with just one entry). Maybe MinGW and Windows 8 don't cooperate well? The answer is that it's the weekend.

Otherwise, I tried to work on some bugs involving default methods, which more or less result from default methods just not being finished. They are no more finished as of today than they were yesterday, but maybe Monday.

comments

January 05, 2013 06:15 AM

So it turns out I have a lot to say about how my 2012 went. Part 2 to follow!

In January, I began a research assistant position with my advisor, Ryan, marking the first time in grad school that I've had no major responsibilities other than research (that is, no classes and no teaching). In December, Ryan and I had submitted an NSF grant proposal about lattice-based deterministic parallel computation, so now we began fleshing out the ideas we'd described in the proposal, putting together a semantics for a small language with lattice-based data structures that we called LVars.

Alex and I started training for the 2012 Toronto Marathon, and I opened a Fitocracy account on January 1 with ambitious plans to log 1,000 miles by year's end.

I spent some time helping organize InWIC, a conference for women in computing in Indiana; the main thing I did was work together with my friends Will and Ian at Studio Cypher to design a cooperative puzzle game for the InWIC attendees to play.

At the end of the month, I went on a delightful road trip to POPL in Philadelphia with a whole pile of folks from IU and CMU. Highlights included playing board games in the hotel lobby and being too scared to say hello to Simon Peyton Jones in the elevator (later I got over it and we had a lovely chat).

In February, Ryan and I continued working on our project. As our LVar-based language began to solidify, I started working on a determinism proof for it, taking the determinism proof for the Featherweight CnC language in this paper as a starting point and frequently soliciting Amr's advice. It had actually been the Featherweight CnC proof that had gotten me started thinking about the relationship between monotonicity and determinism in the first place, so it was interesting to go back to that proof and try to adapt it to our setting.

It began to sink in that I was headed toward doing a Ph.D. with Ryan on deterministic parallelism, which was pretty exciting. Deterministic parallelism was still rather unfamiliar territory for me, but now that I had a big proof to work on, I was in my element. Amr also agreed to join my nascent thesis committee.

As March began, I was getting close -- or so I thought! -- to finishing the proof, and we considered submitting something to ICFP, but with the proof still not done, we decided we'd better wait and shoot for POPL in July instead.

During IU's spring break in mid-March, Alex and I went to Boston for a few days, where I spent some time working with Amal on the still-unfinished multi-language parametricity project I was doing with her. While there, I told Amal about what I was working on with Ryan, and when I showed her one of the properties I'd had to prove to make our determinism proof go through, she pointed out that it looked sort of like a frame rule. At the time, I didn't really understand what that meant, but the idea stuck with me.

While I was in Boston, Amr found a bug in my proof that had to do with the names of memory locations and the circumstances under which they could be renamed. It wasn't a showstopper, but it would continue to plague us, in one guise or another, for several months.

When I got back to Bloomington, after a couple of meetings in which Ryan, Amr, and I all found ourselves evaluating expressions by hand on Amr's whiteboard, we realized that we really needed a runnable model of our language, so I started hacking a Redex model together. I had a lot of fun bringing the paper semantics to life. On March 30th, which was my 30th birthday, I celebrated by giving a whiteboard talk at PL-wonks about what Ryan, Amr, and I had been doing, followed by a Redex demo.

The next day, Alex and I ran the IU Mini Marathon, which doubled as part of my Toronto marathon training.

In April, I had the honor of visiting my undergrad school, Grinnell College, as part of the Alumni Scholars Program. It was a lot of fun: I chatted with my former professors, met some current students, and gave a talk about the work I did in 2011 implementing parts of the (former, prototype-based) Rust object system.

Once back at home, I spent much of the rest of April trying to finish a particularly hairy case of the diamond lemma (the lemma that did the heavy lifting of our determinism proof) that had been thwarting us for the previous two months. I finally got it at the end of the month -- the solution required a tweak to the aforementioned frame-rule-like property that seemed obvious in retrospect -- and the only remaining known problem with the proof was the location-renaming bug that Amr had pointed out. By now, the proof had grown to 40 pages long, and we'd started drafting our POPL submission.

In May, the semester ended, and Alex and I headed to Toronto for the marathon (following a last-minute trip to Chicago to get my passport renewed). Toronto was great! We got to hang out with our friends Ryan and Janice, who live around the corner from Honest Ed's, and I caught up with my old friend Steven from the Act IV days. And I successfully completed my seventh marathon. Given the course's negative change in elevation, it should have been an easy PR, but I missed my PR by five minutes due to a series of unfortunate gastrointestinal events.

After Toronto, Alex and I headed to California (with cats in tow) for our second summers of working on Google Translate and Rust, respectively. It was great to return to Mozilla, where I pretty much already knew everyone on the Rust team. Most impressively, all of the summer 2011 Rust interns -- Tim Chevalier, Eric Holk, Paul Stansifer, Michael Sullivan, and me -- were back for summer 2012, either as returning interns or, in Tim's case, as full-time employees.

For me, returning to Mozilla for a second summer was something of an indulgence. I had reached a point in my Ph.D. when I really should have been concentrating on research, rather than going off and doing yet another internship, and I suspect that some of the other interns were in a similar position. But the opportunity to work on a project as cool as Rust, with a team as good as the Rust team, was too tempting for any of us to pass up, it seems.

Near the end of May, Ryan and I got the incredibly exciting news that the grant proposal we'd submitted in December had been recommended for funding. Being recommended for funding usually means that a grant will be funded, but we kept quiet about it because it wasn't yet a sure thing. So began an extremely squirmy two months of compulsive email-checking and NSF-website-page-reloading.

In June, I managed to keep up my running, aided by the perfect Mountain View weather and the fact that our apartment was right next to the Stevens Creek Trail. For Alex's part, he ran almost every day, sometimes as part of his commute.

At work, I had to spend a fair amount of time re-familiarizing myself with Rust; the compiler had matured a lot since I'd left the previous August, and the language had sprouted a mysterious new region system, a typeclass system of sorts, and a few other features that were new to me. After some amount of flailing with the type inference engine (which had been overhauled (...again)), I finally landed integer-literal suffix inference in the compiler in mid-June, in time for the 0.3 release.

Once suffix inference was done, I needed a new project. The team had been mulling over the idea of adding traits to Rust for some time; as far as I know, Patrick was the first to bring up the idea of traits, in his object system proposal from November 2011. After reading a bit and asking some questions, I realized that what everyone had been calling traits were essentially Haskell-style typeclasses with default methods -- in the terminology of the original traits paper, "provided" methods were implementations, while "required" methods corresponded to signatures -- and that Rust's existing ifaces were just underpowered typeclasses that could be extended to traits (whereas traits and interfaces had been distinct notions in the original proposal). The idea of combining traits and interfaces was well received, and I spent a fair amount of time putting a proposal together, with help from Patrick and Dave, who had already done some thinking about separate compilation and so on. Adding default methods to an interface, it turns out, has surprisingly deep semantic ramifications -- the whole thing's...polarity?...flips, or something. I don't have a good explanation at hand, but something about it made me go "whoa" and sort of stagger back from the whiteboard.

Meanwhile, I was continuing to work on Ryan's and my POPL submission in my off hours. We still had the location-renaming bug in the proof to contend with, but Ryan suggested that I put the proof aside for a while and focus on writing instead, since the POPL deadline was quickly approaching...

Continued in part 2!

January 04, 2013 02:19 AM

For Rust 0.5, which was released two weeks ago, we had a meeting a couple of months ago to set priorities. I can't find the wiki page that resulted from that meeting (I did find a note consisting of our suggested candidate goals, not the list of goals we actually came up with). But I do remember the two issues in the issue tracker that were supposed to be my priorities: pattern reform and labeled break and continue.

Back in November (or something), I decided that the best way to address pattern reform would be to make `let` and `match` share the same codegen path. I thought this would be easy, but it wasn't. Over a month later, I was still working on it. I gave up in frustration when we were getting close to the 0.5 release and I decided to work on blockers instead. So that meant I didn't finish pattern reform.

I sort of finished labeled break and continue, but there's still a bug open on labeled break not working from inside a `for` loop. I don't consider this a huge failure on my part. We implemented labeled break and continue for a very specific reason, which doesn't seem to require labeled break from inside a `for` anyway. From a "no arbitrary restrictions" language-design point of view, it would be good if it worked, but I don't see it as urgent. (The reason why it's hard is that Rust effectively implements `for` loops by desugaring into a call to a higher-order function. So a `break` out of a `for` loop actually jumps to a different stack frame -- hence, involves a `return` -- as opposed to a `break` out of a `while` or `loop` (infinite loop), which is really just a `goto`. The back-end has special cases to handle the unlabeled-`break` case, but I didn't understand the code well enough to extend it to labeled `break`s.) (Also, don't take the word "desugaring" there too literally, since the Rust compiler -- surprisingly for me as an ex-GHC-hacker -- preserves the AST in almost the same shape all the way down to codegen, keeping many higher-level constructs intact.)

Going back to pattern reform, one of the things that was hard about the refactoring I was doing was the need to account for where "cleanups" get introduced. At least right now, Rust uses a combination of automatic reference counting, and "owned" and "borrowed" pointers that don't require reference counting, to manage memory. There is a GC, but it's a last resort. So the codegen has a lot of very fiddly code that inserts "cleanups" that free up memory when the refcount of a shared pointer goes to zero or when an owned pointer's lifetime ends. If you get this code wrong, you get memory leaks, dangling pointers, or both, as I discovered many times. The codegen pass contains any number of undocumented invariants about where the cleanups have to go and when it's safe to cancel them or leave them out. Every time I thought I understood how it worked, I would discover one more case where I didn't understand. There's been some talk of getting rid of reference counting, but in the meantime I'm kind of disappointed in myself that I wasn't more aggressive about pursuing answers (either by reading code or trying to ask questions of the people who wrote it).

I've been wanting to spend some of my time extensively documenting trans (and the rest of the Rust compiler), but that's hard to do without understanding the code in the first place! Lack of documentation shuts out new contributors, especially new contributors who don't feel socially comfortable just jumping into IRC and asking questions. If *I* often feel uncomfortable asking questions about code, as someone who's been working on the Rust project for almost two years and knows everyone else on the core team, what would it be like for an outsider, especially one who isn't a middle-class white English-speaking guy like most of us on the core team?

So that's a summary of my last few months. For the new year, at least as of yet, my calendar seems to be wide open!

comments

January 02, 2013 09:12 PM

On January 2, 2012 I started work at Mozilla as a seasonal employee working on the Rust team. In March, I was offered and accepted a permanent position, and as of today I've been full-time (which is to say, not an intern -- sometimes "full-time" gets contrasted with "contractor/seasonal", in which case my one-year anniversary wouldn't be for 3 more months) for one year. Post-master's-degree, I've had five different full-time jobs including my current one, and three internships including Mozilla, but I've never stayed at any of them for more than ten months other than Mozilla. I guess that says I like it here?

This year was full of more heartbreak, fury, grief, and difficulty than I would have ever expected to experience in conjunction with a job. On the other hand, working with the Rust team has been as much of a pleasure and a joy as I can imagine any compiler engineering job being. There's a change that I'm hoping to make happen early this year that I'm hoping will help me contribute more fully, but in the meantime, I'm just taking a moment to remind myself that I made it. Especially after how I got pushed out of grad school, it's been a relief to work with people who support me. Thanks to Dave, Graydon, Patrick, Brian, Niko, Alon, Donovan, Andrew, Jesse, Lukas, Christie, and everyone I'm forgetting about for the laughs/lunch companionship/collaboration/help/advice/IRC conversations/etc. Also thanks to the summer interns -- Lindsey, Paul, Eric, Brian, Sully, Ben, Margaret, Stephen, Elliott, and (again) everyone else I'm forgetting -- for bringing needed enlivenment to the office :-D

comments

January 02, 2013 08:44 PM

Today was my first day back at work after having surgery and taking a short vacation (it went great, thanks!) I didn't get much done (weekend work, here I come). But here's what I did try to work on today:

Inheritance doesn't work right in default methods, as the bug title says, and I actually fixed the typechecker bug that was reported here. Woe is me, though, I ran into a place where trans was incomplete in re: generating code for inherited self-method calls. I'm still trying to understand the code well enough (which isn't very documented) to finish the incomplete parts. An old story. I'm about to get kicked out of a coffee shop, so no more details beyond that :-D

comments

December 27, 2012 04:02 AM

I haven't written anything under this tag in a long while. For most of that time, I've been head-down working on the let/match refactoring in trans. I feel like I've learned a ton about trans in the meantime that I will be able to translate into comments that will help future trans hackers (pun possibly intended). But I also feel embarrassed about not having a lot of visible work to show for the past month or month-and-a-half.

The middle of this week, though, I decided to set that aside since we have the 0.5 release coming up soon, and I didn't think I was going to finish #3235 in time. Instead, I decided to just try to fix as many release blockers as possible. I'm particularly satisfied to have fixed #3121 just now (well, it's not committed yet, still waiting for tests to run). I spent a good chunk of time trying to isolate this bug when it was newer, and I gave up because I didn't understand trans::alt well enough. But thanks to that time I've spent trying to refactor, when I looked at it again today, I was able to find the buggy code in just a few minutes. The bug is even simple enough to explain in terms of code (I think):

fn enter_default(bcx: block, dm: DefMap, m: &[@Match/&r],
                 col: uint, val: ValueRef)
    -> ~[@Match/&r]
{
    do enter_match(bcx, dm, m, col, val) |p| {
        match p.node {
          ast::pat_wild | ast::pat_rec(_, _) | ast::pat_tup(_) |
          ast::pat_struct(*) => Some(~[]),
          ast::pat_ident(_, _, None) if pat_is_binding(dm, p) => Some(~[]),
          _ => None
        }
    }
}

This is the code that takes a list of Matchs (structures representing patterns that may match) and narrows down that list to patterns that are irrefutable: that match any value of the right type. Examples are _, the wildcard pattern that matches anything, and x if x is a free variable (which just binds any value to the name x. enter_match is a more general function -- the details aren't important -- but the point is that it takes a higher-order function that classifies patterns. Here, the match on p says that the following kinds of patterns are defaults: wildcards (_); record, struct, and tuple patterns (that is, matches on constructors for product types), and variable patterns. The higher-order function returns Some(~[]) (again, the reason why doesn't matter) for defaults, and None for non-defaults.

That's all fine and good, but in Rust we have pattern guards; to use the example from #3121:

match *m {
      to_go(_) => { }
      for_here(_) if cond => {}
      for_here(hamburger) => {}
      for_here(fries(s)) => {}
      for_here(shake) => {}
    }

Normally, for_here(_) would match any value of type order (see the bug report for the type definitions), since the _ nested pattern matches anything. But the pattern guard -- written if cond -- means this pattern only matches if cond is true. So the _ here isn't really a default, because a guard makes the entire pattern that it's attached to refutable.

Once I realized this, the fix was quite simple. And happily, I got to close several other issues that turned out to be instances of the very same bug: #2869, #3257, and #3895.

There's no greater moral here, no fancy theory; just another mundane, prosaic day in the life of a compiler writer. The real issue was that pattern guards were added on after the fact, and probably not tested all that thoroughly -- but I certainly can't place any blame for that, since I generally don't put as much effort as I should into test coverage either.

ETA: Well, the fix wasn't as simple as that after all, sadly. Don't want to have an "I am so smart" moment here. :P Still working on it.

comments

December 08, 2012 07:08 AM

When writing custom passes for a compiler, it's often a good idea to try running them on real-world programs to assess things like scalability and correctness. Taking a large project and seeing your pass work (and provide useful results!) is an exhilarating feeling. On the other hand, trying to feed in your compiler options into build systems is a good route to an insane asylum.

I complained some time ago about autoconf 2.13 failing because it assumes implicit int. People rightly pointed out that newer versions of autoconf don't assume that anymore. But new versions still come with their own cornucopias of pain. Libtool, for example, believes that the best route to linking a program is to delete every compiler flag from the command line except those it knows about. Even if you explicitly specify them in LDFLAGS. Then there's this conftest program that I found while compiling gawk:

/* Define memcpy to an innocuous variant, in case <limits.h> declares memcpy.
   For example, HP-UX 11i <limits.h> declares gettimeofday.  */
#define memcpy innocuous_memcpy

/* System header to define __stub macros and hopefully few prototypes,
    which can conflict with char memcpy (); below.
    Prefer <limits.h> to <assert.h> if __STDC__ is defined, since
    <limits.h> exists even on freestanding compilers.  */

#ifdef __STDC__
# include <limits.h>
#else
# include <assert.h>
#endif
#undef memcpy

/* Override any GCC internal prototype to avoid an error.
   Use char because int might match the return type of a GCC
   builtin and then its argument prototype would still apply.  */
#ifdef __cplusplus
extern "C"
#endif
char memcpy ();
/* The GNU C library defines this for functions which it implements
    to always fail with ENOSYS.  Some functions are actually named
    something starting with __ and the normal name is an alias.  */
#if defined __stub_memcpy || defined __stub___memcpy
choke me
#endif

int
main ()
{
return memcpy ();
  ;
  return 0;
}

…I think this code speaks for itself in how broken it is as a test. One of the parts of the compiler pass involved asserted due to memcpy not being used in the right way, crashing the compiler. Naturally, this being autoconf, it proceeded to assume that I didn't have a memcpy and thus decided to provide me one, which causes later code to break in spectacular bad function when you realize that memcpy is effectively a #define in modern glibc. And heaven forbid if I should try to compile with -Werror in configure scripts (nearly every test program causes a compiler warning along the lines of "builtin function is horribly misused").

The saddest part of all is that, as bad as autoconf is, it appears to be the least broken configuration system out there…

November 08, 2012 01:04 AM

A brief rant since I don't feel much like talking about the specifics of the progress I haven't been making lately. People I work with, don't take this personally; it's more about a general trend :-)

I'm not very good at reading code, or perhaps it's that I don't like reading code and so I don't do it, and don't get better at it. I'm not sure. In any case, I certainly prefer writing code to reading code. This probably makes me like most programmers. Still, you're supposed to get something out of reading code, and I often don't.

Part of the problem is that code often doesn't have very many comments in it. When I'm writing code, I'm not always the best at commenting it either, but often I find I have to write comments just to keep track of what I'm doing, because otherwise I lose my train of thought. When I read un-commented code, I think "this person must be a better programmer than me, because they can hold so much stuff in their head without writing it down". I get intimidated and my mind shuts down and I think maybe I should be a bus driver.

Maybe people who can keep all that state in their heads are better programmers, I don't know. But when it's just in someone's head, it's not available to someone else who might want to work on the project. A friend wrote recently about the ways in which lack of documentation excludes new contributors from open-source projects, particularly new contributors who have plenty of ability but aren't comfortable joining an often-frat-boyish IRC channel to get documentation. Some people need documentation more than others, depending on their social placement.

I'm already in a project, so I should be able to get information the informal way, but sometimes I feel like I ought to know already. Because I don't like reading code that isn't well-commented, I'll do almost anything else to figure out what's going on -- running it repeatedly with lots of print statements added, for example, which incurs a high cost when builds are slow. Or tweaking test cases fairly randomly and observing the change in behavior, or reading the code the compiler generates, which for whatever reason is cumbersome (just making it visible, not understanding it). That takes a lot of time when in some situations, it would probably just be faster to read the code and develop a mental model in it in my head than to -- as I more often do -- develop a model of what it does using the scientific method, treating it as a black box.

But reading the code often brings up an intolerable amount of performance anxiety for me. I can't understand it because my brain is going "you ought to be able to understand this better". Comments would help because it would remind me that the person who wrote this isn't omnipotent, and had to think hard about it in order to get it right. That they went through a process in order to create it, and the product didn't suddenly spring fully formed from their brain.

comments

November 03, 2012 06:20 AM

I've been immersed in some painful bug-fixing, and it hasn't seemed very appealing to write about it. First, while bug triaging, I found this issue from almost a year ago. As a solution, Graydon suggested changing the AST to preserve any parentheses that existed in the source code. This isn't as weird an idea as it seems as first: we also retain things like comments, so that we can test that our pretty-printer prints out the same code it was given. The pretty-printer had heuristics for deciding what expressions to parenthesize, but it was wrong in some cases and there were some tests where the pretty-printer doesn't roundtrip correctly (normally, we check that the pretty-printer roundtrips on every test cases). I implemented Graydon's suggestion, and as a result, all those heuristics are gone, which is satisfying. I thought this was going to be a quick fix, but it was actually difficult to get all the various bits and pieces of data stored in various hash tables that get passed around the compiler right. That is, if e has some entry in some table, should it also be attached to (e)?

Now I'm working on refactoring the code in trans that handles pattern matches so we don't have two different code paths doing the same thing: one for irrefutable patterns in let statements, and one for all patterns (including irrefutable ones) in match expressions. This has been mind-bendingly hard and is making me feel totally ineffectual. Part of the problem is that due to whatever weird psychological hangup of my own, I don't like to make small test cases to isolate bugs, so for quite a while I was trying to narrow down bugs as they manifested themselves in building the compiler itself. I finally gave in and made a stand-alone test case, which makes things easier... up to a point, anyway. My least favorite bugs are those that manifest themselves as bugs in generated code. I think we're doing well if most compiler bugs manifest themselves as assertion failures in the compiler. That happens sometimes now, but not always. As I told a co-worker today, though, the past couple days at work have just made me feel like a monkey with a keyboard (well, more so than usual).

What I should be working on is making labeled breaks and continues work inside for loop bodies, but this seems to require more understanding of trans than I have, and so I've been putting it off.

comments

November 01, 2012 05:04 AM

Today, I finished fixing issue 2284 -- I'm just doing the final test run, then I'll submit a pull request (not push directly this time!)

This is sort of a weird bug. Rust had a notion of "kinds" that was distinct from our notion of traits, because kinds existed before traits did. If you know what kinds mean in other languages, it's more or less the same in Rust. If not: a kind is just the type of a type. Just like values have types so that the compiler can check that you're using them in ways that make sense, so do types have kinds, so that you can be prevented from using types in a nonsensical way.

Rust's kinds, though, are unusual: we had four constant kinds, const, owned, send, and copy, which can also be combined. So a given type could have none, some, or all of these kinds. The copy kind, for example, describes types whose values are allowed to be copied.

Once we introduced traits, it made sense to look at kinds as special traits that were wired into the compiler. So that's what I did to fix #2284. This had the nice effect of making the AST representation of type param bounds (a bound is the thing after the colon when you write <T: Copy> as a parameter list) much simpler: there were no longer any special cases for the built-in kinds. Now, resolve resolves references to things like Copy to placeholder traits that get defined in the core library, and the typechecker special-cases to turn these into specially represented bounds in the ty::param_bound data structure when it sees references to the placeholders.

For this bug, the net effect is that now, you can override Copy and other built-in traits/kinds with your own traits if you really want to. The one weird thing that happens now is that if you use the #[no_core] attribute to say you don't want to import the core library, you won't be able to use the Copy trait in your type parameter declarations -- but that's probably not so bad.

comments

October 17, 2012 07:10 AM

Today I mostly just checked in code that had been sitting in my repo waiting for the tree to be un-frozen...

...and it is un-frozen, because we released! If you really want to for some reason, you could read the Hacker News thread or Reddit thread.

I fixed the three related bugs having to do with constants and discriminator exprs; fixed another bug where the fix was to change resolve so that structs can't capture type parameters; and fixed this bug where the code in trans that normalizes types was interacting with the type substitution code badly. And that's all I feel like saying right now. Oh, and I did a code review (and totally forgot to ask for reviews before checking in any of my own code. Hopefully publicly shaming myself will encourage me to do it next time.

comments

October 16, 2012 04:49 AM