Rob HawkesJoin 250 others in the Open Data Community on Slack

This is a cross-post with the ViziCities blog.

This is a short and personal post written with the hope that it encourages you to join the new Open Data Community on Slack – a place for realtime communication and collaboration on the topic of open data.



It's important to foster open data, as it is to provide a place for the discussion and sharing of ideas around the production and use of open data. It's for this reason that we've created the Open Data Community in the hope of not only giving something back for the things that we have taken, but to provide a place for people to come together to help further this common goal.

The Open Data Community is not ViziCities; it's a group of like-minded invidividuals, non-profits and corporations alike. It's for anyone interested in open data, as well as for those who produce, use or are otherwise involved in its lifecycle.

In just 2 days the community has grown to 250 strong – I look forward to seeing you there and talking open data!

Sign up and get involved.

Robin - ViziCities Founder

Cameron Kaiser31.6.0 available

31.6.0 is available (downloads, release notes, hashes). This includes all the security issues to date, but no specific TenFourFox changes. It becomes final Monday evening Pacific time as usual assuming no critical issues are identified by you, our lovely and wonderful testing audience.

Geoff Lankow1 Million Add-Ons Downloaded

This is a celebratory post. Today I learned that the add-ons I've created for Firefox, Thunderbird, and SeaMonkey have been downloaded over 1,000,000 times in total. For some authors I'm sure that's not a major milestone – some add-ons have more than a million users – but for me it's something I think I can be proud of. (By comparison my add-ons have a collective 80,000 users.)

Here are some of them:

I started six years ago with Shrunked Image Resizer, which makes photos smaller for uploading them to websites. Later I modified it to also make photos smaller in Thunderbird email, and that's far more popular that the original purpose.

Around the same time I got frustrated when developing websites, having to open the page I was looking at in different browsers to test. The process involved far more keystrokes and mouse clicks for my liking, so I created Open With, which turned that into a one-click job.

Later on I created Tab Badge, to provide a visual alert of stuff happening with a tab. This can be quite handy when watching trees, as well as with Twitter and Facebook.

Then there's New Tab Tools – currently my most popular add-on. It's the standard Firefox new tab page, plus a few things, and minus a few things. Kudos to those of you who wrote the built in page, but I like mine better. :-)

Lastly I want to point out my newest add-on, which I think will do quite well once it gets some publicity. I call it Noise Control and it provides a visual indicator of tabs playing audio. (Yes, just like Chrome does.) I've seen lots of people asking for this sort of thing over the years, and the answer was always "it can't be done". Yes it can.

Big thanks to all of you reading this who've downloaded my add-ons, use them, helped me fix bugs, translated, sent me money, answered my inane questions or otherwise done something useful. Thank you. Really.

Robert O'CallahanEclipse + Gecko = Win

With Eclipse 4.4.1 CDT and the in-tree Eclipse project builder (./mach build-backend -b CppEclipse), the Eclipse C++ tools work really well on Gecko. Features I really enjoy:

  • Ctrl-click to navigate to definitions/declarations
  • Ctrl-T to popup the superclasses/subclasses of a class, or the overridden/overriding implementations of a method
  • Shift-ctrl-G to find all uses of a declaration (not 100% reliable, but almost always good enough)
  • Instant coloring of syntax errors as you type (useless messages, but still worth having)
  • Instant coloring of unknown identifier and type errors as you type; not 100% reliable, but good enough that most of my compiler errors are caught before doing a build.
  • Really good autocomplete. E.g. given
    nsTArray<nsRefPtr<Foo>> array;
    for (auto& v : array) {
    v->P
    Eclipse will autocomplete methods of Foo starting with P ... i.e., it handles "auto", C++ for-range loops, nsTArray and nsRefPtr operator overloading.
  • Shift-ctrl-R: automated renaming of identifiers. Again, not 100% reliable but a massive time saver nonetheless.

Thanks to Jonathan Watt and Benoit Girard for the mach support and other Eclipse work over the years!

I assume other IDEs can do these things too, but if you're not using a tool at least this powerful, you might be leaving some productivity on the table.

With Eclipse, rr, and a unified hg repo, hacking Gecko has never felt so good :-).

L. David BaronThe need for government

I've become concerned about the attitudes towards government in the technology industry. It seems to me (perhaps exaggerating a little) that much of the effort in computer security these days considers the major adversary to be the government (whether acting legally or illegally), rather than those attempting to gain illegal access to systems or information (whether private or government actors).

Democracy requires that the government have power. Not absolute power, but some, limited, power. Widespread use of technology that makes it impossible for the government to exercise certain powers could be a threat to democracy.

Let's look at one recent example: a recent article in the Economist about ransomware: malicious software that encrypts files on a computer, whose authors then demand payment to decrypt the files. The payment demanded these days is typically in Bitcoin, a system designed to avoid the government's power. This means that Bitcoin avoids the mechanisms that the international system has to find and catch criminals by following the money they make, and thus makes a perfect system for authors of ransomware and other criminals. The losers are those who don't have the mix of computer expertise and luck needed to avoid the ransomware.

One of the things that democracies often try to do is to protect the less powerful. For example, laws to protect property (in well-functioning governments) protect everybody's property, not just the property of those who can defend their property by force. Having laws like these not only (often) provides a fairer chance for the weak, but it also lets people use their labor on things that can improve people's lives rather than on zero-sum fighting over existing resources. Technology that keeps government out risks making it impossible for government to do this.

I worry that things like ransomware payment in Bitcoin could be just the tip of the iceberg. Technology is changing society quickly, and I don't think this will be the only harmful result of technology designed to keep government out. I don't want the Internet to turn into a “wild west,” where only the deepest experts in technology can survive. Such a change to the Internet risks either giving up many of the potential benefits of the Internet for society by keeping important things off of it, or alternatively risks moving society towards anarchy, where there is no government power that can do what we have relied on governments to do for centuries.

Now I'm not saying today's government is perfect; far from it. Government has responsibility too, including to deserve the trust that we need to place in it. I hope to write about that more in the future.

Andrew HalberstadtMaking mercurial bookmarks more git-like

I mentioned in my previous post a mercurial extension I wrote for making bookmarks easier to manipulate. Since then it has undergone a large overhaul, and I believe it is now stable and intuitive enough to advertise a bit more widely.

Introducing bookbinder

When working with bookmarks (or anonymous heads) I often wanted to operate on the entire series of commits within the feature I was working on. I often found myself digging out revision numbers to find the first commit in a bookmark to do things like rebasing, grafting or diffing. This was annoying. I wanted bookmarks to work more like a git-style branch, that has a definite start as well as an end. And I wanted to be able to easily refer to the set of commits contained within. Enter bookbinder.

First, you can install bookbinder by cloning:

bash $ hg clone https://bitbucket.org/halbersa/bookbinder

Then add the following to your hgrc:

ini [extensions] bookbinder = path/to/bookbinder

Usage is simple. Any command that accepts a revset with --rev, will be wrapped so that bookmark labels are replaced with the series of commits contained within the bookmark.

For example, let's say we create a bookmark to work on a feature called foo and make two commits:

```bash $ hg log -f changeset: 2:fcd3bdafbc88 bookmark: foo summary: Modify foo

changeset: 1:8dec92fc1b1c summary: Implement foo

changeset: 0:165467d1f143 summary: Initial commit ```

Without bookbinder, bookmarks are only labels to a commit:

bash $ hg log -r foo changeset: 2:fcd3bdafbc88 bookmark: foo summary: Modify foo

But with bookbinder, bookmarks become a logical series of related commits. They are more similar to git-style branches:

```bash $ hg log -r foo changeset: 2:fcd3bdafbc88 bookmark: foo summary: Modify foo

changeset: 1:8dec92fc1b1c summary: Implement foo ```

Remember hg log is just one example. Bookbinder automatically detects and wraps all commands that have a --rev option and that can receive a series of commits. It even finds commands from arbitrary extensions that may be installed! Here are few examples that I've found handy in addition to hg log:

```bash $ hg rebase -r <bookbark> -d <dest> $ hg diff -r <bookmark> $ hg graft -r <bookmark> $ hg grep -r <bookmark> $ hg fold -r <bookmark> $ hg prune -r <bookmark>

etc.

```

They all replace the single commit pointed to by the bookmark with the series of commits within the bookmark. But what if you actually only want the single commit pointed to by the bookmark label? Bookbinder uses '.' as an escape character, so using the example above:

bash $ hg log -r .foo changeset: 2:fcd3bdafbc88 bookmark: foo summary: Modify foo

Bookbinder will also detect if bookmarks are based on top of one another:

bash $ hg rebase -r my_bookmark_2 -d my_bookmark_1

Running hg log -r my_bookmark_2 will not print any of the commits contained by my_bookmark_1.

The gory details

But how does bookbinder know where one feature ends, and another begins? Bookbinder implements a new revset called "feature". The feature revset is roughly equivalent to the following alias (kudos to smacleod for coming up with it):

ini [revsetalias] feature($1) = ($1 or (ancestors($1) and not (excludemarks($1) or hg ancestors(excludemarks($1))))) and not public() and not merge() excludemarks($1) = ancestors(parents($1)) and bookmark()

Here is a formal definition. A commit C is "within" a feature branch ending at revision R if all of the following statements are true:

  1. C is R or C is an ancestor of R
  2. C is not public
  3. C is not a merge commit
  4. no bookmarks exist in [C, R) for C != R
  5. all commits in (C, R) are also within R for C != R

In easier to understand terms, this means all ancestors of a revision that aren't public, a merge commit or part of a different bookmark, are within that revision's 'feature'. One thing to be aware of, is that this definition allows empty bookmarks. For example, if you create a new bookmark on a public commit and haven't made any changes yet, that bookmark is "empty". Running hg log -r with an empty bookmark won't have any output.

The feature revset that bookbinder exposes, works just as well on revisions that don't have any associated bookmark. For example, if you are working with an anonymous head, you could do:

bash $ hg log -r 'feature(<rev>)'

In fact, when you pass in a bookmark label to a supported command, bookbinder is literally just substituting -r <bookmark> with -r feature(<bookmark>). All the hard work is happening in the feature revset.

In closing, bookbinder has helped me make a lot more sense out of my bookmark based workflow. It's solving a problem I think should be handled in mercurial core, maybe one day I'll attempt to submit a patch upstream. But until then, I hope it can be useful to others as well.

Christian HeilmannNo more excuses – a “HTML5 now” talk at #codemotion Rome

Yesterday I closed up the “inspiration” track of Codemotion Rome with a talk about the state of browsers and how we as developers make it much too hard for ourselves. You can see the slides on Slideshare and watch a screencast on YouTube.

Mozilla Privacy BlogMozilla Privacy Teaching Task Force

For about a year, the Mozilla community in Pune, India has had an informal task force where four active members go out to local universities and conferences to speak about privacy, security, surveillance, and other topics. Mozilla Reps Ankit Gadgil, … Continue reading

Cameron KaiserIonPower: phase 5!

Progress! I got IonPower past the point PPCBC ran aground at -- it can now jump in and out of Baseline and Ion code on PowerPC without crashing or asserting. That's already worth celebrating, but as the judge who gave me the restraining order on behalf of Scarlett Johansson remarked, I always have to push it. So I tried our iterative π calculator again and really gave it a workout by forcing 3 million iterations. Just to be totally unfair, I've compared the utterly unoptimized IonPower (in full Ion mode) versus the fully optimized PPCBC (Baseline) in the forthcoming TenFourFox 31.6. Here we go (Quad G5, Highest Performance mode):

% /usr/bin/time /Applications/TenFourFoxG5.app/Contents/MacOS/js --no-ion -e 'var pi=4,top=4,bot=3,minus = true;next(pi,top,bot,minus,3000000);function next(pi,top,bot,minus,num){for(var i=0;i<num;i++){pi += (minus == true)?-(top/bot):(top/bot);minus = \!minus;bot+=2;}print(pi);}'
3.1415929869229293
0.48 real 0.44 user 0.03 sys

% /usr/bin/time ../../../obj-ff-dbg/dist/bin/js --ion-offthread-compile=off -e 'var pi=4,top=4,bot=3,minus = true;next(pi,top,bot,minus,3000000);function next(pi,top,bot,minus,num){for(var i=0;i<num;i++){pi += (minus == true)?-(top/bot):(top/bot);minus = \!minus;bot+=2;}print(pi);}'
3.1415929869229293
0.37 real 0.21 user 0.16 sys

No, that's not a typo. The unoptimized IonPower, even in its primitive state, is 23 percent faster than PPCBC on this test largely due to its superior use of floating point. It gets even wider when we do 30 million iterations:

% /usr/bin/time /Applications/TenFourFoxG5.app/Contents/MacOS/js --no-ion -e 'var pi=4,top=4,bot=3,minus = true;next(pi,top,bot,minus,30000000);function next(pi,top,bot,minus,num){for(var i=0;i<num;i++){pi += (minus == true)?-(top/bot):(top/bot);minus = \!minus;bot+=2;}print(pi);}'
3.1415926869232984
4.20 real 4.15 user 0.03 sys

% /usr/bin/time ../../../obj-ff-dbg/dist/bin/js --ion-offthread-compile=off -e 'var pi=4,top=4,bot=3,minus = true;next(pi,top,bot,minus,30000000);function next(pi,top,bot,minus,num){for(var i=0;i<num;i++){pi += (minus == true)?-(top/bot):(top/bot);minus = \!minus;bot+=2;}print(pi);}'
3.1415926869232984
1.55 real 1.38 user 0.16 sys

That's 63 percent faster. And I'm not even to fun things like leveraging the G5's square root instruction (the G3 and G4 versions will use David Kilbridge's software square root from JaegerMonkey), parallel compilation on the additional cores or even working on some of the low-hanging fruit with branch optimization, and on top of all that IonPower is still running all its debugging code and sanity checks. I think this qualifies as IonPower phase 5 (basic operations), so now the final summit will be getting the test suite to pass in both sequential and parallel modes. When it does, it's time for TenFourFox 38!

By the way, for Ben's amusement, how does it compare to our old, beloved and heavily souped up JaegerMonkey implementation? (17.0.11 was our fastest version here; 19-22 had various gradual degradations in performance due to Mozilla's Ion development screwing around with methodjit.)

% /usr/bin/time /Applications/TenFourFoxG5-17.0.11.app/Contents/MacOS/js -m -n -e 'var pi=4,top=4,bot=3,minus = true;next(pi,top,bot,minus,30000000);function next(pi,top,bot,minus,num){for(var i=0;i<num;i++){pi += (minus == true)?-(top/bot):(top/bot);minus = \!minus;bot+=2;}print(pi);}'
3.1415926869232984
4.15 real 4.11 user 0.02 sys

Yup. I'm that awesome. Now I'm gonna sit back and go play some well-deserved Bioshock Infinite on the Xbox 360 (tri-core PowerPC, thank you very much, and I look forward to cracking the firmware one of these days) while the G5 is finishing the 31.6 release candidates overnight. They should be ready for testing tomorrow, so watch this space.

Jordan LundMozharness is moving into the forest

Since its beginnings, Mozharness has been living in its own world (repo). That's about to change. Next quarter we are going to be moving it in-tree.

what's Mozharness?

it's a configuration driven script harness

why in tree?
  1. First and foremost: transparency.
    • There is an overarching goal to provide developers the keys to manage and stand up their own builds & tests (AKA self-serve). Having the automation step logic side by side to the compile and test step logic provides developers transparency and a sense of determinism. Which leads to reason number 2.
  2. deterministic builds & tests
    • This is somewhat already in place thanks to Armen's work on pinning specific Mozharness revisions to in-tree revisions. However the pins can end up behind the latest Mozharness revisions so we end up often landing multiple changes to Mozharness at once to one in-tree revsion.
  3. Mozharness automated build & test jobs are not just managed by Buildbot anymore. Taskcluster is starting to take the weight off Buildbot's hands and, because of its own behaviour, Mozharness is better suited in-`tree.
  4. ateam is going to put effort this quarter into unifying how we run tests locally vs automation. Having mozharness in-tree should make this easier
this sounds great. why wouldn't we want to do this?

There are downsides. It arguably puts extra strain on Release Engineering for managing infra health. Though issues will be more isolated, it does become trickier to have a higher view of when and where Mozharness changes land.

In addition, there is going to be more friction for deployments. This is because a number of our Mozharness scripts are not directly related to continuous integration jobs: e.g. releases, vcs-sync, b2g bumper, and merge tasks.

why wasn't this done yester-year?

Mozharness now handles > 90% of our build and test jobs. Its internal components: config, script, and log logic, are starting to mature. However, this wasn't always the case.

When it was being developed and its uses were unknown, it made sense to develop on the side and tie itself close to buildbot deployments.

okay. I'm sold. can we just simply hg add mozharness?

Integrating Mozharness in-tree comes with a fe6 challenges

  1. chicken and egg issue

    • currently, for build jobs, Mozharness is in charge of managing version control of the tree itself. How can Mozharness checkout a repo if it itself lives within that repo?
  2. test jobs don't require the src tree

    • test jobs only need a binary and a tests.zip. It doesn't make sense to keep a copy of our branches on each machine that runs tests. In line with that, putting mozharness inside tests.zip also leads us back to a similar 'chicken and egg' issue.
  3. which branch and revisions do our release engineering scripts use?

  4. how do we handle releases?

  5. how do we not cause extra load on hg.m.o?

  6. what about integrating into Buildbot without interruption?

it's easy!

This shouldn't be too hard to solve. Here is a basic outline my plan of action and roadmap for this goal:

  • land copy of mozharness on a project branch
  • add an end point on relengapi with the following logic
    1. endpoint will contain 'mozharness' and a '$REVISION'
    2. look in s3 for equivalent mozharness archive
    3. if not present: download a sub repo dir archive from hg.m.o, run tests, and push that archive to s3
    4. finally, return the url to the s3 archive
  • integrate the endpoint into buildbot
    • call endpoint before scheduling jobs
    • add builder step: download and unpack the archive on the slave
  • for machines that run mozharness based releng scripts
    • add manifest that points to 'known good s3 archive'
    • fix deploy model to listen to manifest changes and downloads/unpacks mozharness in a similar manner to builds+tests

This is a loose outline of the integration strategy. What I like about this

  1. no code change required within Mozharness' code
  2. there is very little code change within Buildbot
  3. allows Taskcluster to use Mozharness in whatever way it likes
  4. no chicken and egg problem as (in Buildbot world), Mozharness will exist before the tree exists on the slave
  5. no need to manage multiple repos and keep them in sync

I'm sure I am not taking into account many edge cases and I look forward to hitting those edges head on as I start this in Q2. Stay tuned for further developments.

One day, I'd like to see Mozharness (at least its internal parts) be made into isolated python packages installable by pip. However, that's another problem for another day.

Questions? Concerns? Ideas? Please comment here or in the tracking bug

Doug BelshawWeeknote 13/2015

This week I’ve been:

Mozilla

  • Finishing off my part of the Hive Toronto Privacy badges project. GitHub repo here.
  • Submitting my final expenses and health & wellness invoices.
  • Writing about Web Literacy Map v1.5 (my last post on the Webmaker blog!)
  • Editing the Learning Pathways whitepaper. I’ll do as much as I can, but it’s up to Karen Smith to shepherd from this point forward!
  • Backing up everything.
  • Catching-up one to one with a few people.
  • Leaving Mozilla. I wrote about that here. Some colleagues gave me a Gif tribute send-off and dressed up an inflatable dinosaur in a party hat. Thanks guys!

Dynamic Skillset

  • Helping out DigitalMe with an event in Leeds around Open Badges. I wrote that up here.
  • Preparing my presentation for a keynote next week.
  • Collaborating on a proposal to scope out Open Badges for UK Scouting.
  • Replying to lots of people/organisations who’d like to work with me! :)
  • Finalising things for next week when I start working with City & Guilds for most (OK, nearly all) of my working week.
  • Getting to grips with Xero (which is what I’m using for accounting/invoicing)

Other

Next week I’m spending most of Monday with my family before heading off to London. I’ll be keynoting and running a workshop at the London College of Fashion conference on Tuesday. On Wednesday and Thursday I’ll be working from the City & Guilds offices, getting to know people and putting things into motion!

Image CC BY Kenny Louie