Niko Matsakis — Come contribute to Salsa 2022!

Have you heard of the Salsa project? Salsa is a library for incremental computation – it’s used by rust-analyzer, for example, to stay responsive as you type into your IDE (we have also discussed using it in rustc, though more work is needed there). We are in the midst of a big push right now to develop and release Salsa 2022, a major new revision to the API that will make Salsa far more natural to use. I’m writing this blog post both to advertise that ongoing work and to put out a call for contribution. Salsa doesn’t yet have a large group of maintainers, and I would like to fix that. If you’ve been looking for an open source project to try and get involved in, maybe take a look at our Salsa 2022 tracking issue and see if there is an issue you’d like to tackle?

So wait, what does Salsa do?

Salsa is designed to help you build programs that respond to rapidly changing inputs. The prototypical example is a compiler, especially an IDE. You’d like to be able to do things like “jump to definition” and keep those results up-to-date even as the user is actively typing. Salsa can help you build programs that manage that.

The key way that Salsa achieves reuse is through memoization. The idea is that you define a function that does some specific computation, let’s say it has the job of parsing the input and creating the Abstract Syntax Tree (AST):

fn parse_program(input: &str) -> AST { }

Then later I have other functions that might take parts of that AST and operate on them, such as type-checking:

fn type_check(function: &AstFunction) { }

In a setup like this, I would like to have it so that when my base input changes, I do have to re-parse but I don’t necessarily have to run the type checker. For example, if the only change to my progam was to add a comment, then maybe my AST is not affected, and so I don’t need to run the type checker again. Or perhaps the AST contains many functions, and only one of them changed, so while I have to type check that function, I don’t want to type check the others. Salsa can help you manage this sort of thing automatically.

What is Salsa 2022 and how is it different?

The original salsa system was modeled very closely on the [rustc query system]. As such, it required you to structure your program entirely in terms of functions and queries that called one another. All data was passed through return values. This is a very powerful and flexible system, but it can also be kind of mind-bending sometimes to figure out how to “close the loop”, particularly if you wanted to get effective re-use, or do lazy computation.

Just looking at the parse_program function we saw before, it was defined to return a complete AST:

fn parse_program(input: &str) -> AST { }

But that AST has, internally, a lot of structure. For example, perhaps an AST looks like a set of functions:

struct Ast {
    functions: Vec<AstFunction>
}

struct AstFunction {
    name: Name,
    body: AstFunctionBody,
}

struct AstFunctionBody {
    ...
}

Under the old Salsa, changes were tracked at a pretty coarse-grained level. So if your input changed, and the content of any function body changed, then your entire AST was considered to have changed. If you were naive about it, this would mean that everything would have to be type-checked again. In order to get good reuse, you had to change the structure of your program pretty dramatically from the “natural structure” that you started with.

Enter: tracked structs

The newer Salsa introduces tracked structs, which makes this a lot easier. The idea is that you can label a struct as tracked, and now its fields become managed by the database:

#[salsa::tracked]
struct AstFunction {
    name: Name,
    body: AstFunctionBody,
}

When a struct is declared as tracked, then we also track accesses to its fields. This means that if the parser produces the same set of functions, then its output is considered not to have changed, even if the function bodies are different. When the type checker reads the function body, we’ll track that read independently. So if just one function has changed, only that function will be type checked again.

Goal: relatively natural

The goal of Salsa 2022 is that you should be able to convert a program to use Salsa without dramatically restructuring it. It should still feel quite similar to the ‘natural structure’ that you would have used if you didn’t care about incremental reuse.

Using techniques like tracked structs, you can keep the pattern of a compiler as a kind of “big function” that passes the input through many phases, while still getting pretty good re-use:

fn typical_compiler(input: &str) -> Result {
    let ast = parse_ast(input);
    for function in &ast.functions {
        type_check(function);
    }
    ...
}

Salsa 2022 also has other nice features, such as accumulators for managing diagnostics and built-in interning.

If you’d like to learn more about how Salsa works, check out the overview page or read through the (WIP) tutorial, which covers the design of a complete compiler and interpreter.

How to get involved

As I mentioned, the purpose of this blog post is to serve as a call for contribution. Salsa is a cool project but it doesn’t have a lot of active maintainers, and we are actively looking to recruit new people.

The Salsa 2022 tracking issue contains a list of possible items to work on. Many of those items have mentoring instructions, just search for things tagged with good first issue. There is also documentation of salsa’s internal structure on the main web page that can help you navigate the code base. Finally, we have a Zulip instance where we hang out and chat (the #good-first-issue stream is a good place to ask for help!)

Mozilla Thunderbird — Thunderbird Time Machine: Windows XP + Thunderbird 1.0

Let’s step back into the Thunderbird Time Machine, and transport ourselves back to November 2004. If you were a tech-obsessed geek like me, maybe you were upgrading Windows 98 to Windows XP. Or playing Valve’s legendary shooter Half-Life 2. Maybe you were eagerly installing a pair of newly released open-source software applications called Firefox 1.0 and Thunderbird 1.0…

As we work toward a new era of Thunderbird, we’re also revisiting its roots. Because the entirety of Thunderbird’s releases and corresponding release notes have been preserved, I’ve started a self-guided tour of Thunderbird’s history. Read the first post in this series here:

Thunderbird Time Machine, 2003: A Look Back At Thunderbird 0.1

“Thunderbirds Are GO!”

Before we get into the features of Thunderbird 1.0, I have to call out the endearing credits reel that could be viewed from the “About Thunderbird” menu. You could really feel that spark of creativity and fun from the developers:

<figcaption>Yes, we have a YouTube channel! Subscribe for more. </figcaption>

Windows XP + 2 New Open-Source Alternatives

Thunderbird 1.0 launched in the prime of Windows XP, and it had a companion for the journey: Firefox 1.0! Though both of these applications had previous versions (with different logos and different names), their official 1.0 releases were milestones. Especially because they were open-source and represented quality alternatives to existing “walled-garden” options.

Thunderbird 1.0 and Firefox 1.0 installers on Windows XP<figcaption>Thunderbird 1.0 and Firefox 1.0 installers on Windows XP</figcaption>

(Thunderbird was, and always has been, completely free to download and use. But the internet was far less ubiquitous than it is now, so we offered to mail users within the United States a CD-ROM for $5.95.)

Without a doubt, Mozilla Thunderbird is a very good e-mail client. It sends and receives mail, it checks it for spam, handles multiple accounts, imports data from your old e-mail application, scrapes RSS news feeds, and is even cross-platform.

Thunderbird 1.0 Review | Ars Technica

Visually, it prided itself on having a pretty consistent look across Windows, Mac OS X, and Linux distributions like CentOS 3.3 or Red Hat. And the iconography was updated to be more colorful and playful, in a time when skeuomorphic design reigned supreme.

Groundbreaking Features In Thunderbird 1.0

Thunderbird 1.0 launched with a really diverse set of features. It offered add-ons (just like its brother Firefox) to extend functionality. But it also delivered some cutting-edge stuff like:

  • Adaptive junk mail controls
  • RSS integration (this was only 2 months after podcasts first debuted)
  • Effortless migration from Outlook Express and Eudora
  • A Global Inbox that could combine multiple POP3 email accounts
  • Message Grouping (by date, sender, priority, custom labels, and more)
  • Automatic blocking of remote image requests from unknown senders
Thunderbird 1.0 About Page<figcaption>Thunderbird 1.0 About Page</figcaption>

Feeling Adventurous? Try It For Yourself!

If you have a PowerPC Mac, a 32-bit Linux distribution, or any real or virtualized version of Windows after Windows 98, you can take your own trip down memory lane. All of Thunderbird’s releases are archived here.

Thunderbird is the leading open-source, cross-platform email and calendaring client, free for business and personal use. We want it to stay secure and become even better. Donations allow us to hire developers, pay for infrastructure, expand our userbase, and continue to improve.

Click here to make a donation. 

The post Thunderbird Time Machine: Windows XP + Thunderbird 1.0 appeared first on The Thunderbird Blog.

Wladimir Palant — Impact of extension privileges

As we’ve seen in the previous article, a browser extension isn’t very different from a website. It’s all the same HTML pages and JavaScript code. The code executes in the browser’s regular sandbox. So what can websites possibly gain by exploiting vulnerabilities in a browser extension?

Well, access to extension privileges of course. Browser extensions usually have lots of those, typically explicitly defined in the permissions entry of the extension manifest, but some are granted implicitly. Reason enough to take a closer look at some of these permissions and their potential for abuse.

Extension manifest of some Avast Secure Browser built-in extensions, declaring a huge list of permissions

Note: This article is part of a series on the basics of browser extension security. It’s meant to provide you with some understanding of the field and serve as a reference for my more specific articles. You can browse the extension-security-basics category to see other published articles in this series.

The crown jewels: host-based permissions

Have a look at the permissions entry of your favorite extension’s manifest. Chances are, you will find entries like the following there:

"permissions": [
  "*://*/*"
]

Or:

"permissions": [
  "http://*/*",
  "https://*/*"
]

Or:

"permissions": [
  "<all_urls>"
]

While these three variants aren’t strictly identical, from the security security point of view the differences don’t matter: this extension requests access to each and every website on the web.

Making requests

When regular websites use XMLHttpRequest or fetch API, they are restricted to requesting data from the own website only. Other websites are out of reach by default, unless these websites opt in explicitly by means of CORS.

For browser extensions, host-based permissions remove that obstacle. A browser extension can call fetch("https://gmail.com/") and get a response back. And this means that, as long as you are currently logged into GMail, the extension can download all your emails. It can also send a request instructing GMail to send an email in your name.

It’s similar with your social media accounts and anything else that can be accessed without entering credentials explicitly. You think that your Twitter data is public anyway? But your direct messages are not. And a compromised browser extension can potentially send tweets or direct messages in your name.

The requests can be initiated by any extension page (e.g. the persistent background page). On Firefox host-based permissions allow content scripts to make arbitrary requests as well. There are no visual clues of an extension performing unexpected requests, if an extension turns malicious users won’t usually notice.

Watching tab updates

Host-based permissions also unlock “advanced” tabs API functionality. They allow the extension to call tabs.query() and not only get a list of user’s browser tabs back but also learn which web page (meaning address and title) is loaded.

Not only that, listeners like tabs.onUpdated become way more useful as well. These will be notified whenever a new page loads into a tab.

So a compromised or malicious browser extension has everything necessary to spy on the user. It knows which web pages the user visits, how long they stay there, where they go then and when they switch tabs. This can be misused for creating browsing profiles (word is, these sell well) – or by an abusive ex/employer/government.

Running content scripts

We’ve already seen a content script and some of its potential to manipulate web pages. However, content scripts aren’t necessarily written statically into the extension manifest. Given sufficient host-based permissions, extensions can also load them dynamically by calling tabs.executeScript() or scripting.executeScript().

Both APIs allow executing not merely files contained in the extensions as content scripts but also arbitrary code. The former allows passing in JavaScript code as a string while the latter expects a JavaScript function which is less prone to injection vulnerabilities. Still, both APIs will wreak havoc if misused.

In addition to the capabilities above, content scripts could for example intercept credentials as these are entered into web pages. Another classic way to abuse them is injecting advertising on each an every website. Adding scam messages to abuse credibility of news websites is also possible. Finally, they could manipulate banking websites to reroute money transfers.

Implicit privileges

Some extension privileges don’t have to be explicitly declared. One example is the tabs API: its basic functionality is accessible without any privileges whatsoever. Any extension can be notified when you open and close tabs, it merely won’t know which website these tabs correspond with.

Sounds too harmless? The tabs.create() API is somewhat less so. It can be used to create a new tab, essentially the same as window.open() which can be called by any website. Yet while window.open() is subject to the pop-up blocker, tabs.create() isn’t. An extension can create any number of tabs whenever it wants.

If you look through possible tabs.create() parameters, you’ll also notice that its capabilities go way beyond what window.open() is allowed to control. And while Firefox doesn’t allow data: URIs to be used with this API, Chrome has no such protection. Use of such URIs on the top level has been banned due to being abused for phishing.

tabs.update() is very similar to tabs.create() but will modify an existing tab. So a malicious extension can for example arbitrarily load an advertising page into one of your tabs, and it can activate the corresponding tab as well.

Webcam, geolocation and friends

You probably know that websites can request special permissions, e.g. in order to access your webcam (video conferencing tools) or geographical location (maps). It’s features with considerable potential for abuse, so users each time have to confirm that they still want this.

Not so with browser extensions. If a browser extension wants access to your webcam or microphone, it only needs to ask for permission once. Typically, an extension will do so immediately after being installed. Once this prompt is accepted, webcam access is possible at any time, even if the user isn’t interacting with the extension at this point. Yes, a user will only accept this prompt if the extension really needs webcam access. But after that they have to trust the extension not to record anything secretly.

With access to your exact geographical location or contents of your clipboard, granting permission explicitly is unnecessary altogether. An extension simply adds geolocation or clipboard to the permissions entry of its manifest. These access privileges are then granted implicitly when the extension is installed. So a malicious or compromised extension with these privileges can create your movement profile or monitor your clipboard for copied passwords without you noticing anything.

Other means of exfiltrating browsing data

Somebody who wants to learn about the user’s browsing behavior, be it an advertiser or an abusive ex, doesn’t necessarily need host-based permissions for that. Adding the history keyword to the permissions entry of the extension manifest grants access to the history API. It allows retrieving the user’s entire browsing history all at once, without waiting for the user to visit these websites again.

The bookmarks permission has similar abuse potential, this one allows reading out all bookmarks via the bookmarks API. For people using bookmarks, their bookmarks collection and bookmark creation timestamps tell a lot about this user’s preferences.

The storage permission

We’ve already seen our example extension use the storage permission to store a message text. This permission looks harmless enough. The extension storage is merely a key-value collection, very similar to localStorage that any website could use. Sure, letting arbitrary websites access this storage is problematic if some valuable data is stored inside. But what if the extension is only storing some basic settings?

You have to remember that one basic issue of online advertising is reliably recognizing visitors. If you visit site A, advertisers will want to know whether you visited site B before and what you’ve bought there. Historically, this goal has been achieved via the cookies mechanism.

Now cookies aren’t very reliable. Browsers are giving users much control over cookies, and they are increasingly restricting cookie usage altogether. So there is a demand for cookie replacements, which led to several “supercookie” approaches to be designed: various pieces of data related to the user’s system leaked by the browser are thrown together to build a user identifier. We’ve seen this escalate into a cat and mouse game between advertisers and browser vendors, the former constantly looking for new identifiers while the latter attempt to restrict user-specific data as much as possible.

Any advertiser using supercookies will be more than happy to throw extension storage into the mix if some browser extension exposes it. It allows storing a persistent user identifier much like cookies do. But unlike with cookies, none of the restrictions imposed by browser vendors will apply here. And the user won’t be able to remove this identifier by any means other than uninstalling the problematic extension.

More privileges

The permissions entry of the extension manifest can grant more privileges. It’s too many to cover all of them here, the nativeMessaging permission in particular will be covered in a separate article. MDN provides an overview of what permissions are currently supported.

Why not restrict extension privileges?

Google’s developer policies explicitly prohibit requesting more privileges that necessary for the extension to function. In my experience this rule in fact works. I can only think of one case where a browser extension requested too many privileges, and this particular extension was being distributed with the browser rather than via some add-on store.

The reason why the majority of popular extensions request a very far-reaching set of privileges is neither malice nor incompetence. It’s rather a simple fact: in order to do something useful you need the privileges to do something useful. Extensions restricted to a handful of websites are rarely interesting enough, they need to make an impact on all of the internet to become popular. Extensions that ask you to upload or type things in manually are inconvenient, so popular extensions request webcam/geolocation/clipboard access to automate the process.

In some cases browsers could do better to limit the abuse potential of extension privileges. For example, Chrome allows screen recording via tabCapture or desktopCapture APIs. The abuse potential is low because the former can only be started as a response to a user action (typically clicking the extension icon) whereas the latter brings up a prompt to select the application window to be recorded. Both are sufficient to prevent extensions from silently starting to record in the background. Any of these approaches would have worked to limit the abuse potential of webcam access.

Such security improvements have the tendency to make extensions less flexible and less user-friendly however. A good example here is the activeTab permission. Its purpose is to make requesting host privileges for the entire internet unnecessary. Instead, the extension can access the current tab when the extension is explicitly activated, typically by clicking its icon.

That approach works well for some extensions, particularly those where the user needs to explicitly trigger an action. It doesn’t work in scenarios where extensions have to perform their work automatically however (meaning being more convenient for the user) or where the extension action cannot be executed immediately and requires preparation. So in my extension survey, I see an almost equal split: 19.5% of extensions using activeTab permission and 19.1% using host permissions for all of the internet.

But that does not account for the extension’s popularity. If I only consider the more popular extensions, the ones with 10,000 users and more, things change quite considerably. With 22% the proportion of the extensions using activeTab permission increases only slightly. Yet a whooping 33.7% of the popular extensions ask for host permissions for each and every website.

This Week In Rust — This Week in Rust 456

Hello and welcome to another issue of This Week in Rust! Rust is a programming language empowering everyone to build reliable and efficient software. This is a weekly summary of its progress and community. Want something mentioned? Tweet us at @ThisWeekInRust or send us a pull request. Want to get involved? We love contributions.

This Week in Rust is openly developed on GitHub. If you find any errors in this week's issue, please submit a PR.

Updates from Rust Community

Official
Newsletters
Project/Tooling Updates
Observations/Thoughts
Rust Walkthroughs
Research
Miscellaneous

Crate of the Week

This week's crate is cargo-pgo, a cargo subcommand to compile your code with profile-guided optimization and BOLT for good measure.

Thanks to Jakub Beránek for the self-suggestion!

Please submit your suggestions and votes for next week!

Call for Participation

Always wanted to contribute to open-source projects but didn't know where to start? Every week we highlight some tasks from the Rust community for you to pick and get started!

Some of these tasks may also have mentors available, visit the task page for more information.

If you are a Rust project owner and are looking for contributors, please submit tasks here.

Updates from the Rust Project

410 pull requests were merged in the last week

Rust Compiler Performance Triage

A fairly quiet week for performance, with the exception of the LLVM 15 upgrade which resulted in many changes, mostly to the positive.

Triage done by @simulacrum. Revision range: cc4dd6fc9f..14a459b3

Summary:

(instructions:u) mean max count
Regressions ❌
(primary)		 0.7% 7.7% 62
Regressions ❌
(secondary)		 1.3% 5.0% 51
Improvements ✅
(primary)		 -1.8% -6.9% 93
Improvements ✅
(secondary)		 -2.4% -22.0% 128
All ❌✅ (primary) -0.8% 7.7% 155

2 Regressions, 4 Improvements, 2 Mixed; 1 of them in rollups 38 artifact comparisons made in total

Full report

Call for Testing

An important step for RFC implementation is for people to experiment with the implementation and give feedback, especially before stabilization. The following RFCs would benefit from user testing before moving forward:

  • No RFCs issued a call for testing this week.

If you are a feature implementer and would like your RFC to appear on the above list, add the new call-for-testing label to your RFC along with a comment providing testing instructions and/or guidance on which aspect(s) of the feature need testing.

Approved RFCs

Changes to Rust follow the Rust RFC (request for comments) process. These are the RFCs that were approved for implementation this week:

  • No RFCs were approved this week.
Final Comment Period

Every week, the team announces the 'final comment period' for RFCs and key PRs which are reaching a decision. Express your opinions now.

RFCs
  • No RFCs entered Final Comment Period this week.
Tracking Issues & PRs
New and Updated RFCs

Upcoming Events

Rusty Events between 2022-08-17 - 2022-09-14 🦀

Virtual
Europe
North America
Oceania

If you are running a Rust event please add it to the calendar to get it mentioned here. Please remember to add a link to the event too. Email the Rust Community Team for access.

Jobs

Please see the latest Who's Hiring thread on r/rust

Quote of the Week

TL;DR: my claim is that Rust is attempting to raise the abstraction in the programming language and ultimately to join computer science and software engineering into one single discipline, an ambition that has been around since these disciplines were created.

– Linus Walleij on his blog

Thanks to Julian Wollersberger for the suggestion!

Please submit quotes and vote for next week!

This Week in Rust is edited by: nellshamrell, llogiq, cdmistman, ericseppanen, extrawurst, andrewpollack, U007D, kolharsam, joelmarcey, mariannegoldin.

Email list hosting is sponsored by The Rust Foundation

Discuss on r/rust

The Mozilla Blog — Announcing Steve Teixeira, Mozilla’s new Chief Product Officer

I am pleased to share that Steve Teixeira has joined Mozilla as our Chief Product Officer. During our search for a Chief Product Officer, Steve stood out to us because of his extensive experience at tech and internet companies where he played instrumental roles in shaping products from research, design, security, development, and getting them out to market.

<figcaption>Steve Teixeira joins Mozilla executive team. Steve was photographed in Redmond, Wash., August 5, 2022.
(Photo by Dan DeLong for Mozilla)</figcaption>

As Chief Product Officer, Steve will be responsible for leading our product teams. This will include setting a product vision and strategy that accelerates the growth and impact of our existing products and setting the foundation for new product development.  His product management and technical expertise as well as his leadership experience are the right fit to lead our product teams into Mozilla’s next chapter. 

“There are few opportunities today to build software that is unambiguously good for the world while also being loveable for customers and great for business,” said Teixeira. “I see that potential in Firefox, Pocket, and the rest of the Mozilla product family. I’m also excited about being a part of the evolution of the product family that comes from projecting Mozilla’s evergreen principles through a modern lens to solve some of today’s most vexing challenges for people on the internet.”

Steve comes to us most recently from Twitter, where he spent eight months as a Vice President of Product for their Machine Learning and Data platforms. Prior to that, Steve led Product Management, Design and Research in Facebook’s Infrastructure organization. He also spent almost 14 years at Microsoft where he was responsible for the Windows third-party software ecosystems and held leadership roles in Windows IoT, Visual Studio and the Technical Computing Group. Steve also held a variety of engineering roles at small and medium-sized companies in the Valley in spaces like developer tools, endpoint security, mobile computing, and professional services. 

Steve will report to me and sit on the steering committee.

The post Announcing Steve Teixeira, Mozilla’s new Chief Product Officer appeared first on The Mozilla Blog.

IRL (podcast) — AI from Above

An aerial picture can tell a thousand stories. But who gets to tell them? From above the clouds, our world is surveilled and datafied. Those who control the data, control the narratives. We explore the legacy of spatial apartheid in South Africa’s townships, and hear from people around the world who are reclaiming power over their own maps.

Raesetje Sefala is mapping the legacy of spatial apartheid in South Africa as a computer vision researcher with Timnit Gebru’s Distributed AI Research Institute (DAIR).

Astha Kapoor researches how communities and organizations can be ‘stewards’ of data about people and places as co-founder of the Aapti Institute in India.

Michael Running Wolf is the founder of Indigenous in AI. He is working on speech recognition and immersive spatial experiences with augmented and virtual reality in Canada.

Denise McKenzie is a location data expert who works with the global mapping organization PLACE to empower governments and communities to use advanced spatial data.

IRL is an original podcast from Mozilla, the non-profit behind Firefox. In Season 6, host Bridget Todd shares stories of people who make AI more trustworthy in real life. This season doubles as Mozilla’s 2022 Internet Health Report.  Go to the report for show notes, transcripts, and more.

The Rust Programming Language Blog — Announcing Rust 1.63.0

The Rust team is happy to announce a new version of Rust, 1.63.0. Rust is a programming language empowering everyone to build reliable and efficient software.

If you have a previous version of Rust installed via rustup, you can get 1.63.0 with:

rustup update stable

If you don't have it already, you can get rustup from the appropriate page on our website, and check out the detailed release notes for 1.63.0 on GitHub.

If you'd like to help us out by testing future releases, you might consider updating locally to use the beta channel (rustup default beta) or the nightly channel (rustup default nightly). Please report any bugs you might come across!

What's in 1.63.0 stable

Scoped threads

Rust code could launch new threads with std::thread::spawn since 1.0, but this function bounds its closure with 'static. Roughly, this means that threads currently must have ownership of any arguments passed into their closure; you can't pass borrowed data into a thread. In cases where the threads are expected to exit by the end of the function (by being join()'d), this isn't strictly necessary and can require workarounds like placing the data in an Arc.

Now, with 1.63.0, the standard library is adding scoped threads, which allow spawning a thread borrowing from the local stack frame. The std::thread::scope API provides the necessary guarantee that any spawned threads will have exited prior to itself returning, which allows for safely borrowing data. Here's an example:

let mut a = vec![1, 2, 3];
let mut x = 0;

std::thread::scope(|s| {
    s.spawn(|| {
        println!("hello from the first scoped thread");
        // We can borrow `a` here.
        dbg!(&a);
    });
    s.spawn(|| {
        println!("hello from the second scoped thread");
        // We can even mutably borrow `x` here,
        // because no other threads are using it.
        x += a[0] + a[2];
    });
    println!("hello from the main thread");
});

// After the scope, we can modify and access our variables again:
a.push(4);
assert_eq!(x, a.len());

Rust ownership for raw file descriptors/handles (I/O Safety)

Previously, Rust code working with platform APIs taking raw file descriptors (on unix-style platforms) or handles (on Windows) would typically work directly with a platform-specific representation of the descriptor (for example, a c_int, or the alias RawFd). For Rust bindings to such native APIs, the type system then failed to encode whether the API would take ownership of the file descriptor (e.g., close) or merely borrow it (e.g., dup).

Now, Rust provides wrapper types such as BorrowedFd and OwnedFd, which are marked as #[repr(transparent)], meaning that extern "C" bindings can directly take these types to encode the ownership semantics. See the stabilized APIs section for the full list of wrapper types stabilized in 1.63, currently, they are available on cfg(unix) platforms, Windows, and WASI.

We recommend that new APIs use these types instead of the previous type aliases (like RawFd).

const Mutex, RwLock, Condvar initialization

The Condvar::new, Mutex::new, and RwLock::new functions are now callable in const contexts, which allows avoiding the use of crates like lazy_static for creating global statics with Mutex, RwLock, or Condvar values. This builds on the work in 1.62 to enable thinner and faster mutexes on Linux.

Turbofish for generics in functions with impl Trait

For a function signature like fn foo<T>(value: T, f: impl Copy), it was an error to specify the concrete type of T via turbofish: foo::<u32>(3, 3) would fail with:

error[E0632]: cannot provide explicit generic arguments when `impl Trait` is used in argument position
 --> src/lib.rs:4:11
  |
4 |     foo::<u32>(3, 3);
  |           ^^^ explicit generic argument not allowed
  |
  = note: see issue #83701 <https://github.com/rust-lang/rust/issues/83701> for more information

In 1.63, this restriction is relaxed, and the explicit type of the generic can be specified. However, the impl Trait parameter, despite desugaring to a generic, remains opaque and cannot be specified via turbofish.

Non-lexical lifetimes migration complete

As detailed in this blog post, we've fully removed the previous lexical borrow checker from rustc across all editions, fully enabling the non-lexical, new, version of the borrow checker. Since the borrow checker doesn't affect the output of rustc, this won't change the behavior of any programs, but it completes a long-running migration (started in the initial stabilization of NLL for the 2018 edition) to deliver the full benefits of the new borrow checker across all editions of Rust. For most users, this change will bring slightly better diagnostics for some borrow checking errors, but will not otherwise impact which code they can write.

You can read more about non-lexical lifetimes in this section of the 2018 edition announcement.

Stabilized APIs

The following methods and trait implementations are now stabilized:

These APIs are now usable in const contexts:

Other changes

There are other changes in the Rust 1.63.0 release. Check out what changed in Rust, Cargo, and Clippy.

Contributors to 1.63.0

Many people came together to create Rust 1.63.0. We couldn't have done it without all of you. Thanks!

The Mozilla Blog — Why I joined Mozilla’s Board of Directors

I first started working with digitalization and the internet when I became CEO of Scandinavia Online in 1998. It was the leading online service in the Nordics and we were pioneers and idealists. I learnt a lot from that experience: the endless opportunities, the tricky business models and the extreme ups and downs in hypes and busts of evaluation. I also remember Mozilla during that time as a beacon of competence and idealism, as well as a champion for the open internet as a force for good.

kristin skogen lund mozilla board member<figcaption>Kristin Skogen Lund</figcaption>

Since those early days I have worked in the media industry, telecoms and interest organizations. Today I serve as CEO of Schibsted, the leading Nordic-based media company (which initially started Scandinavia Online back in the days). We own and operate around 70 digital consumer brands across media, online marketplaces, financial services, price comparison services and technology ventures. Within the global industry, we were known as one of the few traditional media companies that adapted to the digital world early on by disrupting our business model and gaining a position in the digital landscape early.

I am deeply engaged in public policy and I serve as president of the European Tech Alliance (EUTA), comprising the leading tech companies of Europe. We work to influence and improve the EU’s digital regulation and to ensure an optimal breeding ground for European digital entrepreneurship. This work is essential as our societies depend upon technology being a force for good, something that cannot be taken for granted, nor is it always the case.

I take great honor in serving on the board of Mozilla to help promote its vision and work to diversify and expand to new audiences and services. It is exciting to serve on the board of a US-based company with such strong roots and that has been an inspiration for me these past 25 years.

The process of meeting board members and management has strengthened my impression of a very capable and engaged team. To build on past successes is never easy, but in Mozilla’s case it is all the more important — not just for Mozilla, but for the health of the internet and thus our global community. I look very much forward to being part of, and contributing to, that tremendous endeavor.

The post Why I joined Mozilla’s Board of Directors appeared first on The Mozilla Blog.

Wladimir Palant — Anatomy of a basic extension

I am starting an article series explaining the basics of browser extension security. It’s meant to provide you with some understanding of the field and serve as a reference for my more specific articles. You can browse the extension-security-basics category to see other published articles in this series.

Before we go for a deeper dive, let’s get a better understanding of what a browser extension actually is. We’ll take a look at a simple example extension and the different contexts in which its code runs.

Browser extensions? What kind of browser extensions?

Browser extensions were introduced to the general public by Mozilla more than two decades ago. Seeing their success, other browser vendors developed their own extension models. However, Google Chrome becoming the prevalent browser eventually caused all the other extension models to go extinct. At this point in time, only Chrome-compatible extensions are still relevant.

These extensions are supported by Google Chrome itself and other Chromium-based browsers such as Microsoft Edge, Opera or Vivaldi. The Mozilla Firefox browser uses an independent implementation of the extension APIs, the only one as far as I am aware. While mostly compatible, Mozilla’s extension APIs have been improved in some areas. Some of these improvements have security impact but, the extension development being centered on Google Chrome these days, I doubt that many extension developers are aware.

Another interesting aspect is that Mozilla for Android also supports extensions, unlike the mobile versions of Google Chrome. This is merely of theoretical importance however as only add-ons from a very short list can be installed. Two years ago I’ve voiced my concern about this restrictive approach, yet as of today this list still contains only ten browser extensions.

The example extension

So our example extension is going to be a Chrome-compatible one. I’ll discuss the files one by one but you can download the entire source code to play around with here. Unpack this ZIP file to some directory.

All browsers support trying out extensions by loading them from a directory. In Chromium-based browsers you go to chrome://extensions/, enable developer mode and use “Load unpacked” button. In Firefox you go to about:debugging#/runtime/this-firefox and click “Load Temporary Add-on” button.

This extension uses questionable approaches on purpose. It has several potential security issues, none of these are currently exploitable however. Small changes to the extension functionality will change that however, I’ll introduce these in future articles.

The extension manifest

The central piece of an extension is its manifest, a file named manifest.json. Ours looks like this:

{
  "manifest_version": 2,
  "name": "My extension",
  "version": "1.0",
  "permissions": [
    "storage"
  ],
  "content_scripts": [
    {
      "js": [
        "script.js"
      ],
      "matches": [
        "https://example.com/*",
        "https://www.example.com/*"
      ]
    }
  ],
  "background": {
    "scripts": [
      "background.js"
    ]
  },
  "options_ui": {
    "page": "options.html"
  }
}

We use manifest version 2 here. Eventually manifest version 3 is supposed to replace it completely. Yet in my current survey of extension manifests only 16% of all extensions used the newer version.

This manifest declares that the extension requires the storage permission. This means that it can use the storage API to store its data persistently. Unlike cookies or localStorage APIs which give users some level of control, extension storage can normally only be cleared by uninstalling the extension.

It also declares that this extension contains the content script script.js, an options page options.html and a background script background.js. We’ll take a look at all of these next.

The content script

Content scripts are loaded whenever the user navigates to a matching page, in our case any page matching the https://example.com/* expression. They execute like the page’s own scripts and have arbitrary access to the page’s Document Object Model (DOM). Our content script uses that access to display a notification message:

chrome.storage.local.get("message", result =>
{
  let div = document.createElement("div");
  div.innerHTML = result.message + " <button>Explain</button>";
  div.querySelector("button").addEventListener("click", () =>
  {
    chrome.runtime.sendMessage("explain");
  });
  document.body.appendChild(div);
});

This uses the storage API to retrieve the message value from extension’s storage. That message is then added to the page along with a button labeled “Explain”. This is what it looks like on example.com:

Usual website text starting with “Example Domain.” Below it a block saying “Hi there!” followed by a button titled “Explain.”

What happens when this button is clicked? The content script uses runtime.sendMessage() API to send a message to the extension pages. That’s because a content script only has direct access to a handful of APIs such as storage. Everything else has to be done by extension pages that content scripts can send messages to.

The content script capabilities differ slightly depending on browser. For Chromium-based browsers you can find the list in the Chrome Developers documentation, for Firefox MDN is the ultimative source.

The background page

Usually, when content scripts send a message its destination is the background page. The background page is a special page that is always present unless specified otherwise in the extension manifest. It is invisible to the user, despite being a regular page with its own DOM and everything. Its function is typically coordinating all other parts of the extension.

Wait, our extension manifest doesn’t even define a background page! There is only a background script. How does that work?

There is still a background page. It is called _generated_background_page.html and contains the following code:

<!DOCTYPE html>
<body>
<script src="background.js"></script>

If a background page isn’t declared explicitly, the browser will helpfully generate one automatically and make sure all the declared background scripts are loaded into it.

And here is our background script:

chrome.runtime.onMessage.addListener((request, sender, sendResponse) =>
{
  if (request == "explain")
  {
    chrome.tabs.create({ url: "https://example.net/explanation" });
  }
})

It uses runtime.onMessage API to listen to messages. When an "explain" message is received, it uses tabs API to open a page in a new tab.

The options page

But how did that message text get into extension storage? Probably via an option page allowing to configure the extension.

Browser extensions can contain various kinds of pages. There are for example action pages that are displayed in a drop-down when the extension icon is clicked. Or pages that the extension will load in a new tab. Unlike the background page, these pages aren’t persistent but rather load when needed. Yet all of them can receive messages from content scripts. And all of them have full access to extension-specific APIs, as far as the extension’s permissions allow.

Our extension manifest declares an options page. This page displays on top of the extension details if some user manages to find the button to open it which browser vendors hide rather well:

Extension details displayed by the browser: permissions, source, and also “Extension options” button. On top of that page a modal dialog is displayed with the title “My extension”. Inside it the text “Please enter a message” followed by a text box. The value “Hi there!” is filled into the text box.

It’s a regular HTML page, nothing special about it:

<html>

<head>
  <script src="options.js"></script>
</head>

<body>
  Please enter a message:
  <input id="message" style="width: 100%;">
</body>

</html>

The script loaded by this page makes sure that any changes to the message field are immediately saved to the extension storage where our content script will retrieve them:

function init()
{
  let element = document.getElementById("message");
  chrome.storage.local.get("message", result => element.value = result.message);
  element.addEventListener("input", () =>
  {
    chrome.storage.local.set({ message: element.value });
  });
}

window.addEventListener("load", init, { once: true });

The relevant contexts

Altogether the relevant contexts for browser extensions look like this:

Browser extension consists of two sections: extension pages (background page, action page, options page) and content scripts (example.com content script, example.net content script, example.info content script). Content scripts interact with extension pages and with websites: example.com content script with example.com website, example.net content script with example.net website, example.info content script with example.info website. Extension pages interact with browser’s extension APIs, connected websites and desktop applications.

In this article we’ve already seen content scripts that can access websites directly but are barred from accessing most of the extension APIs. Instead, content scripts will communicate with extension pages.

The extension pages are way more powerful than content scripts and have full access to extension APIs. They usually won’t communicate with websites directly however, instead relying on content scripts for website access.

Desktop applications and connected websites are out of scope for this article. We’ll take a thorough look at them later.

Mozilla Thunderbird — How You Can Contribute To Thunderbird Without Knowing How To Code

Thunderbird and K-9 Mail are both open-source software projects. That means anyone can contribute to them, improve them, and make them better products. But how does one contribute? You must need some programming skills, right? No! Do you want to learn how to help make a big difference in the global Thunderbird community, without knowing a single line of code? We have a few ideas to share.

Our friend Dustin Krysak, a contributor to Ubuntu Budgie and a customer solutions engineer at Sysdig, brilliantly compares a software project to a construction job:

“Programming is the carpenter, but you still need the architects, designers, project managers, and permit people to make a construction job come together.”

Dustin Krysak

Similarly, making an open-source project like Thunderbird requires an entire community! We need the talents of programmers, graphic designers, translators, writers, financial supporters, enthusiastic fans, quality assurance helpers, bug hunters, & beta testers.

Even if you just have an idea to share, you can make a difference!

No matter what your skill set is, you can absolutely help make Thunderbird better than ever. Here are a few ideas to get you started.

Join The Support Crew

Are you an experienced Thunderbird user who knows the software inside and out? Maybe you want to pay it forward and volunteer some time to help new users! We even have a private discussion group for our support crew to help each other, so they can better support Thunderbird users.

To get involved: https://wiki.mozilla.org/Thunderbird/tb-support-crew

Testing

Want to help improve Thunderbird by simply using it? Testing is a great way to #contribute and requires no prior experience! Help us catch those bugs before they get loose!

Here’s all the info you need to get started with testing: https://wiki.mozilla.org/Thunderbird:Testing

Let’s Go Bug Hunting

Speaking of bugs, capturing and reporting Thunderbird bugs is really important. It’s invaluable! And you’ll enjoy the satisfaction of helping MILLIONS of other users avoid that bug in the future!

We use Mozilla’s Bugzilla, a very powerful tool: https://bugzilla.mozilla.org/

Translate This

We want the entire world to use Thunderbird, which is why it’s currently available in more than 60 languages. If you’re a wordsmith who understands multiple languages, you can aid in our ongoing translation efforts for translating various Thunderbird web pages.

Join 100 other contributors who help translate Thunderbird websites! https://pontoon.mozilla.org/projects/thunderbirdnet/

Document All The Things

Know what else is important? Documentation! From beginner tutorials to technical guides, there’s always a need for helpful information to be written down and easily found.

There are many ways you can contribute to Thunderbird documentation. Start here: https://www.thunderbird.net/en-US/get-involved/#documentation

Financial Support

Financial contributions are another way to help. Thunderbird is both free and freedom respecting, but we’re also completely funded by donations!

Thunderbird By The Numbers: Our 2021 Financial Report

Year after year, the generosity of our donors makes it possible for Thunderbird to thrive. You can contribute a one-time or recurring monthly donation at give.thunderbird.net.

Sharing Is Caring

Do you share our tweets, Fediverse posts, or Facebook messages? Do you tell your friends and colleagues about Thunderbird? Then yes, you are contributing!

Word of mouth and community promotions are yet another key ingredient to the success of Thunderbird, and any open source project.

Enthusiasm is contagious. Keep sharing ❤

Coding

If you DO have coding skills there are so many ways to help! One of those ways is adding new functionality and designs to Thunderbird with Extensions and Themes!

Here’s what you need to know about making add-ons for Thunderbird: https://developer.thunderbird.net/add-ons/about-add-ons

Last but certainly not least, we’re always looking for contributions to Thunderbird itself from the talented FOSS developer community!

Our Developer Hub has everything you need to start hacking away on Thunderbird: https://developer.thunderbird.net

K-9 Mail and Thunderbird Mobile

Last but certainly not least, there’s the newest member of the Thunderbird family, K-9 Mail. As we work towards bringing Thunderbird to Android, contributions are encouraged! Find out how you can help the K-9 Mail team here: https://k9mail.app/contribute

We can’t wait to see the many important ways you’ll contribute to Thunderbird. If you have questions about it, leave a comment on this post or ask us on social media.

The post How You Can Contribute To Thunderbird Without Knowing How To Code appeared first on The Thunderbird Blog.

This Week In Rust — This Week in Rust 455

Hello and welcome to another issue of This Week in Rust! Rust is a programming language empowering everyone to build reliable and efficient software. This is a weekly summary of its progress and community. Want something mentioned? Tweet us at @ThisWeekInRust or send us a pull request. Want to get involved? We love contributions.

This Week in Rust is openly developed on GitHub. If you find any errors in this week's issue, please submit a PR.

Updates from Rust Community

Official
Foundation
Newsletters
Project/Tooling Updates
Observations/Thoughts
Rust Walkthroughs
Miscellaneous

Crate of the Week

This week's crate is fang an async background processing crate.

Thanks to Ayrat Badykov for the self-suggestion.

Please submit your suggestions and votes for next week!

Call for Participation

Always wanted to contribute to open-source projects but didn't know where to start? Every week we highlight some tasks from the Rust community for you to pick and get started!

Some of these tasks may also have mentors available, visit the task page for more information.

If you are a Rust project owner and are looking for contributors, please submit tasks here.

Updates from the Rust Project

330 pull requests were merged in the last week

Rust Compiler Performance Triage

A pretty quiet week for performance. Unfortunately, by far the biggest change was a regression introduced by increasing the minimum libc version for linux-gnu targets. The exact reason for why this happened in this case is unclear, and it's not easy to investigate. Luckily, the average regression introduced by this change was 0.4% which is fairly small, and many of the larger regressions were limited to doc builds.

Triage done by @rylev. Revision range: 792bc5a0..cc4dd6fc

Summary:

mean max count
Regressions ❌
(primary)		 0.5% 1.4% 146
Regressions ❌
(secondary)		 0.8% 1.6% 78
Improvements ✅
(primary)		 N/A N/A 0
Improvements ✅
(secondary)		 -2.0% -4.0% 9
All ❌✅ (primary) 0.5% 1.4% 146

1 Regressions, 2 Improvements, 2 Mixed; 1 of them in rollups 42 artifact comparisons made in total

Full report here

Call for Testing

An important step for RFC implementation is for people to experiment with the implementation and give feedback, especially before stabilization. The following RFCs would benefit from user testing before moving forward:

  • No RFCs issued a call for testing this week.

If you are a feature implementer and would like your RFC to appear on the above list, add the new call-for-testing label to your RFC along with a comment providing testing instructions and/or guidance on which aspect(s) of the feature need testing.

Approved RFCs

Changes to Rust follow the Rust RFC (request for comments) process. These are the RFCs that were approved for implementation this week:

  • No RFCs were approved this week.
Final Comment Period

Every week, the team announces the 'final comment period' for RFCs and key PRs which are reaching a decision. Express your opinions now.

RFCs
  • No RFCs entered Final Comment Period this week.
Tracking Issues & PRs
New and Updated RFCs

Upcoming Events

Rusty Events between 2022-08-10 - 2022-09-07 🦀

Virtual
Asia
Europe
North America
Oceania

If you are running a Rust event please add it to the calendar to get it mentioned here. Please remember to add a link to the event too. Email the Rust Community Team for access.

Jobs

Please see the latest Who's Hiring thread on r/rust

Quote of the Week

Don't come empty-handed to a project saying "this could be rewritten in Rust". It's obnoxious and gives the rust community a bad name.
Do start the project on your own, adding Rust to the build system and converting some significant functions, and then ask the project's community for comments.

– moltonel on /r/rust

Thanks to zjp-CN for the suggestion!

Please submit quotes and vote for next week!

This Week in Rust is edited by: nellshamrell, llogiq, cdmistman, ericseppanen, extrawurst, andrewpollack, U007D, kolharsam, joelmarcey, mariannegoldin.

Email list hosting is sponsored by The Rust Foundation

Discuss on r/rust

The Rust Programming Language Blog — Non-lexical lifetimes (NLL) fully stable

As of Rust 1.63 (releasing next week), the "non-lexical lifetimes" (NLL) work will be enabled by default. NLL is the second iteration of Rust's borrow checker. The RFC actually does quite a nice job of highlighting some of the motivating examples. "But," I hear you saying, "wasn't NLL included in Rust 2018?" And yes, yes it was! But at that time, NLL was only enabled for Rust 2018 code, while Rust 2015 code ran in "migration mode". When in "migration mode," the compiler would run both the old and the new borrow checker and compare the results. This way, we could give warnings for older code that should never have compiled in the first place; we could also limit the impact of any bugs in the new code. Over time, we have limited migration mode to be closer and closer to just running the new-style borrow checker: in the next release, that process completes, and all Rust code will be checked with NLL.

How does removing the old borrow checker affect users?

At this point, we have almost completely merged "migration mode" and "regular mode", so switching to NLL will have very little impact on the user experience. A number of diagnostics changed, mostly for the better -- Jack Huey gives the full details in his blog post.

Credit where credit is due

The work to remove the old borrow checker has been going on for years. It's been a long, tedious, and largely thankless process. We'd like to take a moment to highlight the various people involved and make sure they are recognized for their hard work:

Jack's blog post includes a detailed narrative of all the work involved if you'd like more details! It's a fun read.

Looking forward: what can we expect for the "borrow checker of the future"?

The next frontier for Rust borrow checking is taking the polonius project and moving it from research experiment to production code. Polonius is a next-generation version of the borrow checker that was "spun off" from the main NLL effort in 2018, as we were getting NLL ready to ship in production. Its most important contribution is fixing a known limitation of the borrow checker, demonstrated by the following example:

fn last_or_push<'a>(vec: &'a mut Vec<String>) -> &'a String {
    if let Some(s) = vec.last() { // borrows vec
        // returning s here forces vec to be borrowed
        // for the rest of the function, even though it
        // shouldn't have to be
        return s; 
    }
    
    // Because vec is borrowed, this call to vec.push gives
    // an error!
    vec.push("".to_string()); // ERROR
    vec.last().unwrap()
}

This example doesn't compile today (try it for yourself), though there's not a good reason for that. You can often workaround the problem by editing the code to introduce a redundant if (as shown in this example), but with polonius, it will compile as is. If you'd like to learn more about how polonius (and the existing borrow checker) works1, you can watch my talk from Rust Belt Rust.

  1. Or where the name "polonius" comes from! ↩

The Mozilla Blog — A back-to-school checklist for online safety

The first day of school is right around the corner. Whether that brings some relief, gives you jitters or both, we’re here to support families with one major thing: internet safety.  

For parents, thinking about the dangers of the web can be scary. But it doesn’t have to be. While the internet isn’t perfect, it’s also a wonderful place for learning and connecting with others. Here’s what families can do to make the best of it while staying safe this school year. 

An illustration shows a digital pop-up box that reads: A back-to-school checklist for online safety: Set up new passwords. Check your devices' privacy settings. Protect your child's browsing information. Discuss parental controls with the whole family. Have the "tech talk."<figcaption>Credit: Nick Velazquez / Mozilla</figcaption>

1. Set up new passwords

Back-to-school season is a good time to update passwords, since students often log in to the same learning tools they use at home and on campus. It’s important to teach kids the basics of password hygiene, including keeping passwords in a safe place and regularly changing them. 

2. Check your devices’ privacy settings 

Whether you have a preschooler who uses the family tablet to watch videos or a kid who’s finally ready for a phone, make sure to set up these devices with data privacy in mind. Figure out – together, if possible – which information they’re sharing with the apps they use. 

Have a school-issued device? Take the time to look into the settings, and don’t be afraid to ask teachers and school administrators about how the tools and software used in classrooms are handling students’ data.

3. Protect your child’s browsing information

An investigation by The Markup, in collaboration with Mozilla Rally, exposed how federal financial aid applications automatically sent students’ personal information to Facebook – even if a student didn’t have a Facebook account. It’s just one example of how invasive big tech’s data tracking has become. One way to cut the amount of information companies are collecting about your kid is by protecting their internet browsing data. 

Firefox has Total Cookie Protection on by default to all users. That means that when your child visits a website, cookies (which store bits of information a page remembers about them), stays within that website and out of the hands of companies that want to track their online behavior and target them with ads. 

How to make Firefox the default browser on a desktop computer:

  • If you haven’t already, download Firefox and open the app. 
  • In the menu bar at the top of the screen, click on Firefox > Preferences.
  • In the general panel, click on the Make Default button.

How to make Firefox the default browser on mobile:

  • Download Firefox.
  • On an iOS device, go to settings, scroll down and click on Firefox > Default Browser App > Firefox. 
  • On an Android, open the app. Click on the menu button next to the address bar > Settings > Set as default browser > Firefox for Android > Set as default.

Find more information about setting Firefox as the default browser on iOS and Android here.

<figcaption>Make Firefox your default browser on mobile.</figcaption>

4. Discuss parental controls with the whole family

Relying on parental control settings to limit kids’ screen time and block websites may be tempting. But no tool can completely protect kids online. One thing that researchers and advocates agree on when it comes to technology: open communication. Parents should talk to their children about whether or not they need to use parental controls and why. They should also figure out a plan to ease restrictions as kids learn how to manage themselves online. 

Ready for that conversation? Here are some Firefox extensions to consider with your family: 

  • Unhook
    Specific to YouTube, Unhook strips away a lot of the distracting “rabbit role” elements of the site, including suggested videos, trending content and comments. 
  • Tomato Clock
    Based on a renowned time management method (Pomodoro technique), this extension helps a user focus on the computer by breaking up work intervals into defined “tomato” bursts. While this productivity extension could benefit anyone, parents might find it useful for helping kids stay focused during online school time.
  • Block Site
    Try this add-on if your family has agreed to implement restrictions on specific websites. With its password control feature, not only can parents continue to visit these websites, but they can also leave custom display messages if their kid tries to access a restricted site (“Busted! Shouldn’t you be doing homework?”) as well as redirect from one site to another (e.g. Roblox.com to a public library website).

If you’re new to extensions, you can learn more here. 

5. Have the “tech talk”

Of course, besides weak passwords and school work distractions, there’s plenty of age-appropriate topics that parents may want to talk to their children about. “It helps to talk about values first, then think through together – in developmentally appropriate ways, based on a child’s life stage – how to put those values into practice,” said Leah A. Plunkett, a Harvard Law School lecturer who teaches a course on youth and digital citizenship.

Another idea: Consider putting what your family has agreed upon on paper and have everyone sign it. Or, use a template like Common Sense Media’s, which also lists items that parents can agree to do, like recognizing the role media plays in their kids’ lives, even if they don’t fully understand it.

Like with any other aspect of parenting, providing kids safe and healthy experiences online is more complicated than it seems. The process won’t be perfect, but learning together – with help from trusted sources – can go a long way. 

The internet is a great place for families. It gives us new opportunities to discover the world, connect with others and just generally make our lives easier and more colorful. But it also comes with new challenges and complications for parents raising the next generations. Mozilla wants to help parents make the best online decisions for their kids, whatever that looks like, in our latest series, Parental Control. 

Firefox browser logo

Get Firefox

Get the browser that protects what’s important

The post A back-to-school checklist for online safety appeared first on The Mozilla Blog.

Firefox Add-on Reviews — Translate the web easily with a browser extension

Do you do a lot of language translating on the web? Are you constantly copying text from one browser tab and navigating to another to paste it? Maybe you like to compare translations from different services like Google Translate or Bing Translate? Need easy access to text-to-speech features? 

Online translation services provide a hugely valuable function, but for those of us who do a lot of translating on the web, the process is time-consuming and cumbersome. With the right browser extension, however, web translations become a whole lot easier and faster. Here are some fantastic translation extensions for folks with differing needs…

I just want a simple, efficient way to translate. I don’t need fancy features.

Simple Translate

It doesn’t get much simpler than this. Highlight the text you want to translate and click the extension’s toolbar icon to activate a streamlined pop-up. Your highlighted text automatically appears in the pop-up’s translation field and a drop-down menu lets you easily select your target language. Simple Translate also features a handy “Translate this page” button should you want that. 

Translate Web Pages

Maybe you just need to translate full web pages, like reading news articles in other languages, how-to guides, or job related sites. If so, Translate Web Pages could be the ideal solution for you with its sharp focus on full-page utility. 

However the extension also benefits from a few intriguing additional features, like the ability to select up to three top languages you most commonly translate into (each one easily accessible with a single click in the pop-up menu), designate specific sites to always translate for you upon arrival, and your choice of three translation engines: Google, Yandex, and DeepL. 

To Google Translate

Very popular, very simple translation extension that exclusively uses Google’s translation services, including text-to-speech. 

Simply highlight any text on a web page and right-click to pull up a To Google Translate context menu that allows three actions: 1) translate into your preferred language; 2) listen to audio of the text; 3) Translate the entire page

<figcaption>Right-click any highlighted text to activate To Google Translate.</figcaption>

Privacy is a priority. I’m uncomfortable sending my translations to a cloud.

Mozilla’s very own Firefox Translations is unique among its peers in that all translations occur locally in the browser instead of accessing translation data across the web or in a cloud. This approach is more private because the contents of your translations never leave your machine.

Firefox Translations is still relatively young in its development cycle and new languages are being added all the time.

I do a ton of translating. I need power features to save me time and trouble.

ImTranslator

Striking a balance between out-of-the-box ease and deep customization potential, ImTranslator leverages three top translation engines (Google, Bing, Translator) to cover 100+ languages; the extension itself is even available in nearly two-dozen languages. 

Other strong features include text-to-speech, dictionary and spell check in eight languages, hotkey customization, and a huge array of ways to tweak the look of ImTranslator’s interface—from light and dark themes to font size and more. 

Mate Translate

A slick, intuitive extension that performs all the basic translation functions very well, but it’s Mate Translate’s paid tier that unlocks some unique features, such as Sync (saved translations can appear across devices and browsers, including iPhones and Mac). 

There’s also a neat Phrasebook feature, which lets you build custom word and phrase lists so you can return to common translations you frequently need. It works offline, too, so it’s ideal for travellers who need quick reference to common foreign phrases. 

These are some of our favorites, but there are plenty more translation extensions to explore on addons.mozilla.org.

This Week In Rust — This Week in Rust 454

Hello and welcome to another issue of This Week in Rust! Rust is a programming language empowering everyone to build reliable and efficient software. This is a weekly summary of its progress and community. Want something mentioned? Tweet us at @ThisWeekInRust or send us a pull request. Want to get involved? We love contributions.

This Week in Rust is openly developed on GitHub. If you find any errors in this week's issue, please submit a PR.

Updates from Rust Community

Official
Project/Tooling Updates
Observations/Thoughts
Rust Walkthroughs
Research
Miscellaneous

Crate of the Week

This week's crate is lending-iterator, a type similar to std::iter::Iterator, but with some type trickery that allows it to .windows_mut(_) safely.

Thanks to Daniel H-M for the self-nomination!

Please submit your suggestions and votes for next week!

Call for Participation

Always wanted to contribute to open-source projects but didn't know where to start? Every week we highlight some tasks from the Rust community for you to pick and get started!

Some of these tasks may also have mentors available, visit the task page for more information.

If you are a Rust project owner and are looking for contributors, please submit tasks here.

Updates from the Rust Project

391 pull requests were merged in the last week

Rust Compiler Performance Triage
Call for Testing

An important step for RFC implementation is for people to experiment with the implementation and give feedback, especially before stabilization. The following RFCs would benefit from user testing before moving forward:

  • No RFCs issued a call for testing this week.

If you are a feature implementer and would like your RFC to appear on the above list, add the new call-for-testing label to your RFC along with a comment providing testing instructions and/or guidance on which aspect(s) of the feature need testing.

Approved RFCs

Changes to Rust follow the Rust RFC (request for comments) process. These are the RFCs that were approved for implementation this week:

  • No RFCs were approved this week.
Final Comment Period

Every week, the team announces the 'final comment period' for RFCs and key PRs which are reaching a decision. Express your opinions now.

RFCs
  • No RFCs entered Final Comment Period this week.
Tracking Issues & PRs
New and Updated RFCs
  • No New or Updated RFCs were created this week.

Upcoming Events

Rusty Events between 2022-08-03 - 2022-08-31 🦀

Virtual
Asia
Europe
North America
Oceania

If you are running a Rust event please add it to the calendar to get it mentioned here. Please remember to add a link to the event too. Email the Rust Community Team for access.

Jobs

Please see the latest Who's Hiring thread on r/rust

Quote of the Week

❤️🦀❤️

100,000 issues filled with love, compassion and a wholesome community. Thank you, Rust community, for being one of the most, if not straight out the most, welcoming programming communities out there. Thank you, Rust teams, for the tireless hours you spend every day on every aspect of this project. Thank you to the Rust team alumni for the many hours spent growing a plant and the humility of passing it to people you trust to continue taking care of it. Thank you everyone for RFCs, giving voice to the community, being those voices AND listening to each other.

This community has been and continue to be one of the best I have ever had the pleasure of being a part of. The language itself has many things to love and appreciate about it, from the humane error messages to giving the people the power to express high performance code without sacrificing readability for the ones to come after us. But nothing, truly nothing, takes the cake as much as the community that's building it, answering questions, helping and loving each other. Every single day.

Congratulations everyone for 100,000 issues and PRs! And thank you for being you. Because Rust is Beautiful, for having you as part of it.

To the times we spent together and the many more to come!

– mathspy on the rust-lang/rust github

Thanks to Sean Chen for the suggestion!

Please submit quotes and vote for next week!

This Week in Rust is edited by: nellshamrell, llogiq, cdmistman, ericseppanen, extrawurst, andrewpollack, U007D, kolharsam, joelmarcey, mariannegoldin.

Email list hosting is sponsored by The Rust Foundation

Discuss on r/rust

Manish Goregaokar — So Zero It's ... Negative? (Zero-Copy #3)

This is part 3 of a three-part series on interesting abstractions for zero-copy deserialization I’ve been working on over the last year. This part is about eliminating the deserialization step entirely. Part 1 is about making it more pleasant to work with and can be found here; while Part 2 is about making it work for more types and can be found here. The posts can be read in any order, though only the first post contains an explanation of what zero-copy deserialization is.

And when Alexander saw the breadth of his work, he wept. For there were no more copies left to zero.

—Hans Gruber, after designing three increasingly unhinged zero-copy crates

Part 1 of this series attempted to answer the question “how can we make zero-copy deserialization pleasant”, while part 2 answered “how do we make zero-copy deserialization more useful?”.

This part goes one step further and asks “what if we could avoid deserialization altogether?”.

Speech bubble for character Confused pion
Wait, what?

Bear with me.

As mentioned in the previous posts, internationalization libraries like ICU4X need to be able to load and manage a lot of internationalization data. ICU4X in particular wants this part of the process to be as flexible and efficient as possible. The focus on efficiency is why we use zero-copy deserialization for basically everything, whereas the focus on flexibility has led to a robust and pluggable data loading infrastructure that allows you to mix and match data sources.

Deserialization is a great way to load data since it’s in and of itself quite flexible! You can put your data in a neat little package and load it off the filesystem! Or send it over the network! It’s even better when you have efficient techniques like zero-copy deserialization because the cost is low.

But the thing is, there is still a cost. Even with zero-copy deserialization, you have to validate the data you receive. It’s often a cost folks are happy to pay, but that’s not always the case.

For example, you might be, say, a web browser interested in using ICU4X, and you really care about startup times. Browsers typically need to set up a lot of stuff when being started up (and when opening a new tab!), and every millisecond counts when it comes to giving the user a smooth experience. Browsers also typically ship with most of the internationalization data they need already. Spending precious time deserializing data that you shipped with is suboptimal.

What would be ideal would be something that works like this:

static DATA: &Data = &serde_json::deserialize!(include_bytes!("./testdata.json"));

where you can have stuff get deserialized at compile time and loaded into a static. Unfortunately, Rust const support is not at the stage where the above code is possible whilst working within serde’s generic framework, though it might be in a year or so.

You could write a very unsafe version of serde::Deserialize that operates on fully trusted data and uses some data format that is easy to zero-copy deserialize whilst avoiding any kind of validation. However, this would still have some cost: you still have to scan the data to reconstruct the full deserialized output. More importantly, it would require a parallel universe of unsafe serde-like traits that everyone has to derive or implement, where even small bugs in manual implementations would likely cause memory corruption.

Speech bubble for character Positive pion
Sounds like you need some format that needs no validation or scanning to zero-copy deserialize, and can be produced safely. But that doesn’t exist, does it?

It does.

… but you’re not going to like where I’m going with this.

Speech bubble for character Positive pion
Oh no.

There is such a format: Rust code. Specifically, Rust code in statics. When compiled, Rust statics are basically “free” to load, beyond the typical costs involved in paging in memory. The Rust compiler trusts itself to be good at codegen, so it doesn’t need validation when loading a compiled static from memory. There is the possibility of codegen bugs, however we have to trust the compiler about that for the rest of our program anyway!

This is even more “zero” than “zero-copy deserialization”! Regular “zero copy deserialization” still involves a scanning and potentially a validation step, it’s really more about “zero allocations” than actually avoiding all of the copies. On the other hand, there’s truly no copies or anything going on when you load Rust statics; it’s already ready to go as a &'static reference!

We just have to figure out a way to “serialize to const Rust code” such that the resultant Rust code could just be compiled in to the binary, and people who need to load trusted data into ICU4X can load it for free!

Speech bubble for character Confused pion
What does “const code” mean in this context?

In Rust, const code essentially is code that can be proven to be side-effect-free, and it’s the only kind of code allowed in statics, consts, and const fns.

Speech bubble for character Confused pion
I see! Does this code actually have to be “constant”?

Not quite! Rust supports mutation and even things like for loops in const code! Ultimately, it has to be the kind of code that can be computed at compile time with no difference of behavior: so no reading from files or the network, or using random numbers.

For a long time only very simple code was allowed in const, but over the last year the scope of what that environment can do has expanded greatly, and it’s actually possible to do complicated things here, which is precisely what enables us to actually do “serialize to Rust code” in a reasonable way.

databake

A lot of the design here can also be found in the design doc. While I did the bulk of the design for this crate, it was almost completely implemented by Robert, who also worked on integrating it into ICU4X, and cleaned up the design in the process.

Enter databake (née crabbake). databake is a crate that provides just this; the ability to serialize your types to const code that can then be used in statics allowing for truly zero-cost data loading, no deserialization necessary!

The core entry point to databake is the Bake trait:

pub trait Bake {
    fn bake(&self, ctx: &CrateEnv) -> TokenStream;
}

A TokenStream is the type typically used in Rust procedural macros to represent a snippet of Rust code. The Bake trait allows you to take an instance of a type, and convert it to Rust code that represents the same value.

The CrateEnv object is used to track which crates are needed, so that it is possible for tools generating this code to let the user know which direct dependencies are needed.

This trait is augmented by a #[derive(Bake)] custom derive that can be used to apply it to most types automatically:

// inside crate `bar`, module `module.rs`

use databake::Bake;

#[derive(Bake)]
#[databake(path = bar::module)]
pub struct Person<'a> {
   pub name: &'a str,
   pub age: u32,
}

As with most custom derives, this only works on structs and enums that contain other types that already implement Bake. Most types not involving mandatory allocation should be able to.

How to use it

databake itself doesn’t really prescribe any particular code generation strategy. It can be used in a proc macro or in a build.rs, or, even in a separate binary. ICU4X does the latter, since that’s just what ICU4X’s model for data generation is: clients can use the binary to customize the format and contents of the data they need.

So a typical way of using this crate might be to do something like this in build.rs:

use some_dep::Data;
use databake::Bake;
use quote::quote;

fn main() {
   // load data from file
   let json_data = include_str!("data.json");

   // deserialize from json
   let my_data: Data = serde_json::from_str(json_data);

   // get a token tree out of it
   let baked = my_data.bake();


   // Construct rust code with this in a static
   // The quote macro is used by procedural macros to do easy codegen,
   // but it's useful in build scripts as well.
   let my_data_rs = quote! {
      use some_dep::Data;
      static MY_DATA: Data = #baked;
   }

   // Write to file
   let out_dir = env::var_os("OUT_DIR").unwrap();
   let dest_path = Path::new(&out_dir).join("data.rs");
   fs::write(
      &dest_path,
      &my_data_rs.to_string()
   ).unwrap();

   // (Optional step omitted: run rustfmt on the file)

   // tell Cargo that we depend on this file
   println!("cargo:rerun-if-changed=src/data.json");
}

What it looks like

ICU4X generates all of its test data into JSON, postcard, and “baked” formats. For example, for this JSON data representing how a particular locale does numbers, the “baked” data looks like this. That’s a rather simple data type, but we do use this for more complex data like date time symbol data, which is unfortunately too big for GitHub to render normally.

ICU4X’s code for generating this is in this file. It’s complicated primarily because ICU4X’s data generation pipeline is super configurable and complicated, The core thing that it does is, for each piece of data, it calls tokenize(), which is a thin wrapper around calling .bake() on the data and some other stuff. It then takes all of the data and organizes it into files like those linked above, populated with a static for each piece of data. In our case, we include all this generated rust code into our “testdata” crate as a module, but there are many possibilities here!

For our “test” data, which is currently 2.7 MB in the postcard format (which is optimized for being lightweight), the same data ends up being 11 MB of JSON, and 18 MB of generated Rust code! That’s … a lot of Rust code, and tools like rust-analyzer struggle to load it. It’s of course much smaller once compiled into the binary, though that’s much harder to measure, because Rust is quite aggressive at optimizing unused data out in the baked version (where it has ample opportunity to). From various unscientific tests, it seems like 2MB of deduplicated postcard data corresponds to roughly 500KB of deduplicated baked data. This makes sense, since one can expect baked data to be near the theoretical limit of how small the data is without applying some heavy compression. Furthermore, while we deduplicate baked data at a per-locale level, it can take advantage of LLVM’s ability to deduplicate statics further, so if, for example, two different locales have mostly the same data for a given data key1 with some differences, LLVM may be able to use the same statics for sub-data.

Limitations

const support in Rust still has a ways to go. For example, it doesn’t yet support creating objects like Strings which are usually on the heap, though they are working on allowing this. This isn’t a huge problem for us; all of our data already supports zero-copy deserialization, which means that for every instance of our data types, there is some way to represent it as a borrow from another static.

A more pesky limitation is that you can’t interact with traits in const environments. To some extent, were that possible, the purpose of this crate could also have been fulfilled by making the serde pipeline const-friendly2, and then the code snippet from the beginning of this post would work:

static DATA: &Data = &serde_json::deserialize!(include_bytes!("./testdata.json"));

This means that for things like ZeroVec (see part 2), we can’t actually just make their safe constructors const and pass in data to be validated — the validation code is all behind traits — so we have to unsafely construct them. This is somewhat unfortunate, however ultimately if the zerovec byte representation had trouble roundtripping we would have larger problems, so it’s not an introduction of a new surface of unsafety. We’re still able to validate things when generating the baked data, we just can’t get the compiler to also re-validate before agreeing to compile the const code.

Try it out!

databake is much less mature compared to yoke and zerovec, but it does seem to work rather well so far. Try it out! Let me know what you think!

Thanks to Finch, Jane, Shane, and Robert for reviewing drafts of this post

  1. In ICU4X, a “data key” can be used to talk about a specific type of data, for example the decimal symbols data has a decimal/symbols@1 data key. ↩

  2. Mind you, this would not be an easy task, but it would likely integrate with the ecosystem really well. ↩

Manish Goregaokar — Zero-Copy All the Things! (Zero-Copy #2)

This is part 2 of a three-part series on interesting abstractions for zero-copy deserialization I’ve been working on over the last year. This part is about making zero-copy deserialization work for more types. Part 1 is about making it more pleasant to work with and can be found here; while Part 3 is about eliminating the deserialization step entirely and can be found here. The posts can be read in any order, though only the first post contains an explanation of what zero-copy deserialization is.

Background

This section is the same as in the last article and can be skipped if you’ve read it

For the past year and a half I’ve been working full time on ICU4X, a new internationalization library in Rust being built under the Unicode Consortium as a collaboration between various companies.

There’s a lot I can say about ICU4X, but to focus on one core value proposition: we want it to be modular both in data and code. We want ICU4X to be usable on embedded platforms, where memory is at a premium. We want applications constrained by download size to be able to support all languages rather than pick a couple popular ones because they cannot afford to bundle in all that data. As a part of this, we want loading data to be fast and pluggable. Users should be able to design their own data loading strategies for their individual use cases.

See, a key part of performing correct internationalization is the data. Different locales1 do things differently, and all of the information on this needs to go somewhere, preferably not code. You need data on how a particular locale formats dates2, or how plurals work in a particular language, or how to accurately segment languages like Thai which are typically not written with spaces so that you can insert linebreaks in appropriate positions.

Given the focus on data, a very attractive option for us is zero-copy deserialization. In the process of trying to do zero-copy deserialization well, we’ve built some cool new libraries, this article is about one of them.

What can you zero-copy?

Speech bubble for character Positive pion
If you’re unfamiliar with zero-copy deserialization, check out the explanation in the previous article!

In the previous article we explored how zero-copy deserialization could be made more pleasant to work with by erasing the lifetimes. In essence, we were expanding our capabilities on what you can do with zero-copy data.

This article is about expanding our capabilities on what we can make zero-copy data.

We previously saw this struct:

#[derive(Serialize, Deserialize)]
struct Person {
    // this field is nearly free to construct
    age: u8,
    // constructing this will involve a small allocation and copy
    name: String,
    // this may take a while
    rust_files_written: Vec<String>,
}

and made the name field zero-copy by replacing it with a Cow<'a, str>. However, we weren’t able to do the same with the rust_files_written field because serde does not handle zero-copy deserialization for things other than [u8] and str. Forget nested collections like Vec<String> (as &[&str]), even Vec<u32> (as &[u32]) can’t be made zero-copy easily!

This is not a fundamental restriction in zero-copy deserialization, indeed, the excellent rkyv library is able to support data like this. However, it’s not as slam-dunk easy as str and [u8] and it’s understandable that serde wishes to not pick sides on any tradeoffs here and leave it up to the users.

So what’s the actual problem here?

Blefuscudian Bewilderment

The short answer is: endianness, alignment, and for Vec<String>, indirection.

See, the way zero-copy deserialization works is by directly taking a pointer to the memory and declaring it to be the desired value. For this to work, that data must be of a kind that looks the same on all machines, and must be legal to take a reference to.

This is pretty straightforward for [u8] and str, their data is identical on every system. str does need a validation step to ensure it’s valid UTF-8, but the general thrust of zero-copy serialization is to replace expensive deserialization with cheaper validation, so we’re fine with that.

On the other hand, the borrowed version of Vec<String>, &[&str] is unlikely to look the same even across different executions of the program on the same system, because it contains pointers (indirection) that’ll change each time depending on the data source!

Pointers are hard. What about Vec<u32>/[u32]? Surely there’s nothing wrong with a pile of integers?

<figcaption class="caption-text">

Dracula, dispensing wisdom on the subject of zero-copy deserialization.

</figcaption>

This is where the endianness and alignment come in. Firstly, a u32 doesn’t look exactly the same on all systems, some systems are “big endian”, where the integer 0x00ABCDEF would be represented in memory as [0x00, 0xAB, 0xCD, 0xEF], whereas others are “little endian” and would represent it [0xEF, 0xCD, 0xAB, 0x00]. Most systems these days are little-endian, but not all, so you may need to care about this.

This would mean that a [u32] serialized on a little endian system would come out completely garbled on a big-endian system if we’re naïvely zero-copy deserializing.

Secondly, a lot of systems impose alignment restrictions on types like u32. A u32 cannot be found at any old memory address, on most modern systems it must be found at a memory address that’s a multiple of 4. Similarly, a u64 must be at a memory address that’s a multiple of 8, and so on. The subsection of data being serialized, however, may be found at any address. It’s possible to design a serialization framework where a particular field in the data is forced to have a particular alignment (rkyv has this), however it’s kinda tricky and requires you to have control over the alignment of the original loaded data, which isn’t a part of serde’s model.

So how can we address this?

ZeroVec and VarZeroVec

A lot of the design here can be found explained in the design doc

After a bunch of discussions with Shane, we designed and wrote zerovec, a crate that attempts to solve this problem, in a way that works with serde.

The core abstractions of the crate are the two types, ZeroVec and VarZeroVec, which are essentially zero-copy enabled versions of Cow<'a, [T]>, for fixed-size and variable-size T types.

ZeroVec can be used with any type implementing ULE (more on what this means later), which is by default all of the integer types and can be extended to most Copy types. It’s rather similar to &[T], however instead of returning references to its elements, it copies them out. While ZeroVec is a Cow-like borrowed-or-owned type3, there is a fully borrowed variant ZeroSlice that it derefs to.

Similarly, VarZeroVec may be used with types implementing VarULE (e.g. str). It is able to hand out references VarZeroVec<str> behaves very similarly to how &[str] would work if such a type were allowed to exist in Rust. You can even nest them, making types like VarZeroVec<VarZeroSlice<ZeroSlice<u32>>>, the zero-copy equivalent of Vec<Vec<Vec<u32>>>.

There’s also a ZeroMap type that provides a binary-search based map that works with types compatible with either ZeroVec or VarZeroVec.

So, for example, to make the following struct zero-copy:

#[derive(serde::Serialize, serde::Deserialize)]
struct DataStruct {
    nums: Vec<u32>,
    chars: Vec<char>,
    strs: Vec<String>,
}

you can do something like this:

#[derive(serde::Serialize, serde::Deserialize)]
pub struct DataStruct<'data> {
    #[serde(borrow)]
    nums: ZeroVec<'data, u32>,
    #[serde(borrow)]
    chars: ZeroVec<'data, char>,
    #[serde(borrow)]
    strs: VarZeroVec<'data, str>,
}

Once deserialized, the data can be accessed with data.nums.get(index) or data.strs[index], etc.

Custom types can also be supported within these types with some effort, if you’d like the following complex data to be zero-copy:

#[derive(Copy, Clone, PartialEq, Eq, Ord, PartialOrd, serde::Serialize, serde::Deserialize)]
struct Date {
    y: u64,
    m: u8,
    d: u8
}

#[derive(Clone, PartialEq, Eq, Ord, PartialOrd, serde::Serialize, serde::Deserialize)]
struct Person {
    birthday: Date,
    favorite_character: char,
    name: String,
}

#[derive(serde::Serialize, serde::Deserialize)]
struct Data {
    important_dates: Vec<Date>,
    important_people: Vec<Person>,
    birthdays_to_people: HashMap<Date, Person>
}

you can do something like this:

// custom fixed-size ULE type for ZeroVec
#[zerovec::make_ule(DateULE)]
#[derive(Copy, Clone, PartialEq, Eq, Ord, PartialOrd, serde::Serialize, serde::Deserialize)]
struct Date {
    y: u64,
    m: u8,
    d: u8
}

// custom variable sized VarULE type for VarZeroVec
#[zerovec::make_varule(PersonULE)]
#[zerovec::derive(Serialize, Deserialize)] // add Serde impls to PersonULE
#[derive(Clone, PartialEq, Eq, Ord, PartialOrd, serde::Serialize, serde::Deserialize)]
struct Person<'data> {
    birthday: Date,
    favorite_character: char,
    #[serde(borrow)]
    name: Cow<'data, str>,
}

#[derive(serde::Serialize, serde::Deserialize)]
struct Data<'data> {
    #[serde(borrow)]
    important_dates: ZeroVec<'data, Date>,
    // note: VarZeroVec always must reference the unsized ULE type directly
    #[serde(borrow)]
    important_people: VarZeroVec<'data, PersonULE>,
    #[serde(borrow)]
    birthdays_to_people: ZeroMap<'data, Date, PersonULE>
}

Unfortunately the inner “ULE type” workings are not completely hidden from the user, especially for VarZeroVec-compatible types, but the crate does a fair number of things to attempt to make it pleasant to work with.

In general, ZeroVec should be used for types that are fixed-size and implement Copy, whereas VarZeroVec is to be used with types that logically contain a variable amount of data, like vectors, maps, strings, and aggregates of the same. VarZeroVec will always be used with a dynamically sized type, yielding references to that type.

I’ve noted before that these types are like Cow<'a, T>; they can be dealt with in a mutable-owned fashion, but it’s not the primary focus of the crate. In particular, VarZeroVec<T> will be significantly slower to mutate than something like Vec<String>, since all operations are done on the same buffer format. The general idea of this crate is that you probably will be generating your data in a situation without too many performance constraints, but you want the operation of reading the data to be fast. So, where necessary, the crate trades off mutation performance for deserialization/read performance. Still, it’s not terribly slow, just something to look out for and benchmark if necessary.

How it works

Most of the crate is built on the ULE and VarULE traits. Both of these traits are unsafe traits (though as shown above most users need not manually implement them). “ULE” stands for “unaligned little-endian”, and marks types which have no alignment requirements and have the same representation across endiannesses, preferring to be identical to the little-endian representation where relevant4.

There’s also a safe AsULE trait that allows one to convert a type between itself and some corresponding ULE type.

pub unsafe trait ULE: Sized + Copy + 'static {
    // Validate that a byte slice is appropriate to treat as a reference to this type
    fn validate_byte_slice(bytes: &[u8]) -> Result<(), ZeroVecError>;

    // less relevant utility methods omitted
}

pub trait AsULE: Copy {
    type ULE: ULE;

    // Convert to the ULE type
    fn to_unaligned(self) -> Self::ULE;
    // Convert back from the ULE type
    fn from_unaligned(unaligned: Self::ULE) -> Self;
}

pub unsafe trait VarULE: 'static {
    // Validate that a byte slice is appropriate to treat as a reference to this type
    fn validate_byte_slice(_bytes: &[u8]) -> Result<(), ZeroVecError>;

    // Construct a reference to Self from a known-valid byte slice
    // This is necessary since VarULE types are dynamically sized and the working of the metadata
    // of the fat pointer varies between such types
    unsafe fn from_byte_slice_unchecked(bytes: &[u8]) -> &Self;

    // less relevant utility methods omitted
}

ZeroVec<T> takes in types that are AsULE and stores them internally as slices of their ULE types (&[T::ULE]). Such slices can be freely zero-copy serialized. When you attempt to index a ZeroVec, it converts the value back to T on the fly, an operation that’s usually just an unaligned load.

VarZeroVec<T> is a bit more complicated. The beginning of its memory stores the indices of every element in the vector, followed by the data for all of the elements just splatted one after the other. As long as the dynamically sized data can be represented in a flat fashion (without further internal indirection), it can implement VarULE, and thus be used in VarZeroVec<T>. str implements this, but so do ZeroSlice<T> and VarZeroSlice<T>, allowing for infinite nesting of zerovec types!

ZeroMap<T> works similarly to the litemap crate, it’s a map built out of two vectors, using binary search to find keys. This isn’t always as efficient as a hash map but it can work well in a zero-copy way since it can just be backed by ZeroVec and VarZeroVec. There’s a bunch of trait infrastructure that allows it to automatically select ZeroVec or VarZeroVec for each of the key and value vectors based on the type of the key or value.

What about rkyv?

An important question when we started down this path was: what about rkyv? It had at the time just received a fair amount of attention in the Rust community, and seemed like a pretty cool library targeting the same space.

And in general if you’re looking for zero-copy deserialization, I wholeheartedly recommend looking at it! It’s an impressive library with a lot of thought put into it. When I was refining zerovec I learned a lot from rkyv having some insightful discussions with David and comparing notes on approaches.

The main sticking point, for us, was that rkyv works kinda separately from serde: it uses its own traits and own serialization mechanism. We really liked serde’s model and wanted to keep using it, especially since we wanted to support a variety of human-readable and non-human-readable data formats, including postcard, which is explicitly designed for low-resource environments. This becomes even more important for data interchange; we’d want programs written in other languages to be able to construct and send over data without necessarily being constrained to a particular wire format.

The goal of zerovec is essentially to bring rkyv-like improvements to a serde universe without disrupting that universe too much. zerovec types, on human-readable formats like JSON, serialize to a normal human-readable representation of the structure, and on binary formats like postcard, serialize to a compact, zero-copy-friendly representation that Just Works.

How does it perform?

So off the bat I’ll mention that rkyv maintains a very good benchmark suite that I really need to get around to integrating with zerovec, but haven’t yet.

Speech bubble for character Negative pion
Why not go do that first? It would make your post better!

Well, I was delaying working on this post until I had those benchmarks integrated, but that’s not how executive function works, and at this point I’d rather publish with the benchmarks I have rather than delaying further. I might update this post with the Good Benchmarks later!

Speech bubble for character Negative pion
Hmph.

The complete benchmark run details can be found here (run via cargo bench at 1e072b32. I’m pulling out some specific data points for illustration:

ZeroVec:

BenchmarkSliceZeroVec
Deserialization (with bincode)
Deserialize a vector of 100 u32s141.55 ns12.166 ns
Deserialize a vector of 15 chars225.55 ns25.668 ns
Deserialize and then sum a vector of 20 u32s47.423 ns14.131 ns
Element fetching performance
Sum a vector of 75 u32 elements4.3091 ns5.7108 ns
Binary search a vector of 1000 u32 elements, 50 times428.48 ns565.23 ns
Binary search a vector of 1000 u32 elements, 50 times428.48 ns565.23 ns
Serialization
Serialize a vector of 20 u32s51.324 ns21.582 ns
Serialize a vector of 15 chars195.75 ns21.123 ns


In general we don’t care about serialization performance much, however serialization is fast here because ZeroVecs are always stored in memory as the same form they would be serialized at. This can make mutation slower. Fetching operations are a little bit slower on ZeroVec. The deserialization performance is where we see our real wins, sometimes being more than ten times as fast!

VarZeroVec:

The strings are randomly generated, picked with sizes between 2 and 20 code points, and the same set of strings is used for any given row.

BenchmarkVec<String>Vec<&str>VarZeroVec
Deserialize (len 100)11.274 us2.2486 us1.9446 us
Count code points (len 100)728.99 ns1265.0 ns
Binary search for 1 element (len 500)57.788 ns122.10 ns
Binary search for 10 elements (len 500)451.40 ns803.67 ns


Here, fetching operations are a bit slower since they need to read the indexing array, but there’s still a decent win for zero-copy deserialization. The deserialization wins stack up for more complex data; for Vec<String> you can get most of the wins by using Vec<&str>, but that’s not necessarily possible for something more complex. We don’t currently have mutation benchmarks for VarZeroVec, but mutation can be slow and as mentioned before it’s not intended to be used much in client code.

Some of this is still in flux; for example we are in the process of making VarZeroVec’s buffer format configurable so that users can pick their precise tradeoffs.

Try it out!

Similar to yoke, I don’t consider the zerovec crate “done” yet, but it’s been in use in ICU4X for a year now and I consider it mature enough to recommend to others. Try it out! Let me know what you think!

Thanks to Finch, Jane, and Shane for reviewing drafts of this post

  1. A locale is typically a language and location, though it may contain additional information like the writing system or even things like the calendar system in use. ↩

  2. Bear in mind, this isn’t just a matter of picking a format like MM-DD-YYYY! Dates in just US English can look like 4/10/22 or 4/10/2022 or April 10, 2022, or Sunday, April 10, 2022 C.E., or Sun, Apr 10, 2022, and that’s not without thinking about week numbers, quarters, or time! This quickly adds up to a decent amount of data for each locale. ↩

  3. As mentioned in the previous post, while zero-copy deserializing, it is typical to use borrowed-or-owned types like Cow over pure borrowed types because it’s not necessary that data in a human-readable format will be able to zero-copy deserialize. ↩

  4. Most modern systems are little endian, so this imposes one fewer potential cost on conversion. ↩

Manish Goregaokar — Not a Yoking Matter (Zero-Copy #1)

This is part 1 of a three-part series on interesting abstractions for zero-copy deserialization I’ve been working on over the last year. This part is about making zero-copy deserialization more pleasant to work with. Part 2 is about making it work for more types and can be found here; while Part 3 is about eliminating the deserialization step entirely and can be found here. The posts can be read in any order, though this post contains an explanation of what zero-copy deserialization is.

Background

For the past year and a half I’ve been working full time on ICU4X, a new internationalization library in Rust being built under the Unicode Consortium as a collaboration between various companies.

There’s a lot I can say about ICU4X, but to focus on one core value proposition: we want it to be modular both in data and code. We want ICU4X to be usable on embedded platforms, where memory is at a premium. We want applications constrained by download size to be able to support all languages rather than pick a couple popular ones because they cannot afford to bundle in all that data. As a part of this, we want loading data to be fast and pluggable. Users should be able to design their own data loading strategies for their individual use cases.

See, a key part of performing correct internationalization is the data. Different locales1 do things differently, and all of the information on this needs to go somewhere, preferably not code. You need data on how a particular locale formats dates2, or how plurals work in a particular language, or how to accurately segment languages like Thai which are typically not written with spaces so that you can insert linebreaks in appropriate positions.

Given the focus on data, a very attractive option for us is zero-copy deserialization. In the process of trying to do zero-copy deserialization well, we’ve built some cool new libraries, this article is about one of them.

<figcaption class="caption-text">

Gary Larson, “Cow Tools”, The Far Side. October 1982

</figcaption>

Zero-copy deserialization: the basics

This section can be skipped if you’re already familiar with zero-copy deserialization in Rust

Deserialization typically involves two tasks, done in concert: validating the data, and constructing an in-memory representation that can be programmatically accessed; i.e., the final deserialized value.

Depending on the format, the former is typically rather fast, but the latter can be super slow, typically around any variable-sized data which needs a new allocation and often a large copy.

#[derive(Serialize, Deserialize)]
struct Person {
    // this field is nearly free to construct
    age: u8,
    // constructing this will involve a small allocation and copy
    name: String,
    // this may take a while
    rust_files_written: Vec<String>,
}

A typical binary data format will probably store this as a byte for the age, followed by the length of name, followed by the bytes for name, followed by another length for the vector, followed by a length and string data for each String value. Deserializing the u8 age just involves reading it, but the other two fields require allocating sufficient memory and copying each byte over, in addition to any validation the types may need.

A common technique in this scenario is to skip the allocation and copy by simply validating the bytes and storing a reference to the original data. This can only be done for serialization formats where the data is represented identically in the serialized file and in the deserialized value.

When using serde in Rust, this is typically done by using a Cow<'a, T> with #[serde(borrow)]:

#[derive(Serialize, Deserialize)]
struct Person<'a> {
    age: u8,
    #[serde(borrow)]
    name: Cow<'a, str>,
}

Now, when name is being deserialized, the deserializer only needs to validate that it is in fact a valid UTF-8 str, and the final value for name will be a reference to the original data being deserialized from itself.

An &'a str can also be used instead of the Cow, however this makes the Deserialize impl much less general, since formats that do not store strings identically to their in-memory representation (e.g. JSON with strings that include escapes) will not be able to fall back to an owned value. As a result of this, owned-or-borrowed Cow<'a, T> is often a cornerstone of good design when writing Rust code partaking in zero-copy deserialization.

You may notice that rust_files_written can’t be found in this new struct. This is because serde, out of the box, can’t handle zero-copy deserialization for anything other than str and [u8], for very good reasons. Other frameworks like rkyv can, however we’ve also managed to make this possible with serde. I’ll go in more depth about said reasons and our solution in part 2.
Speech bubble for character Confused pion
Aren’t there still copies occurring here with the age field?

Yes, “zero-copy” is somewhat of a misnomer, what it really means is “zero allocations”, or, alternatively, “zero large copies”. Look at it this way: data like age does get copied, but without, say, allocating a vector of Person<'a>, you’re only going to see that copy occur a couple times when individually deserializing Person<'a>s or when deserializing some struct that contains Person<'a> a couple times. To have a large copy occur without involving allocations, your type would have to be something that is that large on the stack in the first place, which people avoid in general because it means a large copy every time you move the value around even when you’re not deserializing.

When life gives you lifetimes ….

Zero-copy deserialization in Rust has one very pesky downside: the lifetimes. Suddenly, all of your deserialized types have lifetimes on them. Of course they would; they’re no longer self-contained, instead containing references to the data they were originally deserialized from!

This isn’t a problem unique to Rust, either, zero-copy deserialization always introduces more complex dependencies between your types, and different frameworks handle this differently; from leaving management of the lifetimes to the user to using reference counting or a GC to ensure the data sticks around. Rust serialization libraries can do stuff like this if they wish, too. In this case, serde, in a very Rusty fashion, wants the library user to have control over the precise memory management here and surfaces this problem as a lifetime.

Unfortunately, lifetimes like these tend to make their way into everything. Every type holding onto your deserialized type needs a lifetime now and it’s likely going to become your users’ problem too.

Furthermore, Rust lifetimes are a purely compile-time construct. If your value is of a type with a lifetime, you need to know at compile time by when it will definitely no longer be in use, and you need to hold on to its source data until then. Rust’s design means that you don’t need to worry about getting this wrong, since the compiler will catch you, but you still need to do it.

All of this isn’t ideal for cases where you want to manage the lifetimes at runtime, e.g. if your data is being deserialized from a larger file and you wish to cache the loaded file as long as data deserialized from it is still around.

Typically in such cases you can use Rc<T>, which is effectively the “runtime instead of compile time” version of &'a Ts safe shared reference, but this only works for cases where you’re sharing homogenous types, whereas in this case we’re attempting to share different types deserialized from one blob of data, which itself is of a different type.

ICU4X would like users to be able to make use of caching and other data management strategies as needed, so this won’t do at all. For a while ICU4X had not one but two pervasive lifetimes threaded throughout most of its types: it was both confusing and not in line with our goals.

… make life take the lifetimes back

A lot of the design here can be found explained in the design doc

After a bunch of discussion on this, primarily with Shane, I designed yoke, a crate that attempts to provide lifetime erasure in Rust via self-referential types.

Speech bubble for character Confused pion
Wait, lifetime erasure?

Like type erasure! “Type erasure” (in Rust, done using dyn Trait) lets you take a compile time concept (the type of a value) and move it into something that can be decided at runtime. Analogously, the core value proposition of yoke is to take types burdened with the compile time concept of lifetimes and allow you to decide they be decided at runtime anyway.

Speech bubble for character Confused pion
Doesn’t Rc<T> already let you make lifetimes a runtime decision?

Kind of, Rc<T> on its own lets you avoid compile-time lifetimes, whereas Yoke works with situations where there is already a lifetime (e.g. due to zero copy deserialization) that you want to paper over.

Speech bubble for character Confused pion
Cool! What does that look like?

The general idea is that you can take a zero-copy deserializeable type like a Cow<'a, str> (or something more complicated) and “yoke” it to the value it was deserialized from, which we call a “cart”.

Speech bubble for character Negative pion
*groan* not another crate named with a pun, Manish.

I will never stop.

Anyway, here’s what that looks like.

// Some types explicitly mentioned for clarity

// load a file
let file: Rc<[u8]> = fs::read("data.postcard")?.into();

// create a new Rc reference to the file data by cloning it,
// then use it as a cart for a Yoke
let y: Yoke<Cow<'static, str>, Rc<[u8]>> = Yoke::attach_to_cart(file.clone(), |contents| {
    // deserialize from the file
    let cow: Cow<str> =  postcard::from_bytes(&contents);
    cow
})

// the string is still accessible with `.get()`
println!("{}", y.get())

drop(y);
// only now will the reference count on the file be decreased
Some of the APIs here may not quite work due to current compiler bugs. In this blog post I’m using the ideal version of these APIs for illustrative purposes, but it’s worth checking with the Yoke docs to see if you may need to use an alternate workaround API. Most of the bugs have been fixed as of Rust 1.61.
Speech bubble for character Positive pion
The example above uses postcard: postcard is a really neat serde-compatible binary serialization format, designed for use on resource constrained environments. It’s quite fast and has a low codesize, check it out!

The type Yoke<Cow<'static, str>, Rc<[u8]>> is “a lifetime-erased Cow<str> ‘yoked’ to a backing data store ‘cart’ that is an Rc<[u8]>”. What this means is that the Cow contains references to data from the cart, however, the Yoke will hold on to the cart type until it is done, which ensures the references from the Cow no longer dangle.

Most operations on the data within a Yoke operate via .get(), which in this case will return a Cow<'a, str>, where 'a is the lifetime of borrow of .get(). This keeps things safe: a Cow<'static, str> is not really safe to distribute in this case since Cow is not actually borrowing from static data; however it’s fine as long as we transform the lifetime to something shorter during accesses.

Turns out, the 'static found in Yoke types is actually a lie! Rust doesn’t really let you work with types with borrowed content without mentioning some lifetime, and here we want to relieve the compiler from its duty of managing lifetimes and manage them ourselves, so we need to give it something so that we can name the type, and 'static is the only preexisting named lifetime in Rust.

The actual signature of .get() is a bit weird since it needs to be generic, but if our borrowed type is Foo<'a>, then the signature of .get() is something like this:

impl Yoke<Foo<'static>> {
    fn get<'a>(&'a self) -> &'a Foo<'a> {
        ...
    }
}

For a type to be allowed within a Yoke<Y, C>, it must implement Yokeable<'a>. This trait is unsafe to manually implement, in most cases you should autoderive it with #[derive(Yokeable)]:

#[derive(Yokeable, Serialize, Deserialize)]
struct Person<'a> {
    age: u8,
    #[serde(borrow)]
    name: Cow<'a, str>,
}

let person: Yoke<Person<'static>, Rc<[u8]> = Yoke::attach_to_cart(file.clone(), |contents| {
    postcard::from_bytes(&contents)
});

Unlike most #[derive]s, Yokeable can be derived even if the fields do not already implement Yokeable, except for cases when fields with lifetimes also have other generic parameters. In such cases it typically suffices to tag the type with #[yoke(prove_covariance_manually)] and ensure any fields with lifetimes also implement Yokeable.

There’s a bunch more you can do with Yoke, for example you can “project” a yoke to get a new yoke with a subset of the data found in the initial one:

let person: Yoke<Person<'static>, Rc<[u8]>> = ....;

let person_name: Yoke<Cow<'static, str>> = person.project(|p, _| p.name);

This allows one to mix data coming from disparate Yokes.

Yokes are, perhaps surprisingly, mutable as well! They are, after all, primarily intended to be used with copy-on-write data, so there are ways to mutate them provided that no additional borrowed data sneaks in:

let mut person: Yoke<Person<'static>, Rc<[u8]>> = ....;

// make the name sound fancier
person.with_mut(|person| {
    // this will convert the `Cow` into owned one
    person.name.to_mut().push(", Esq.")
})

Overall Yoke is a pretty powerful abstraction, useful for a host of situations involving zero-copy deserialization as well as other cases involving heavy borrowing. In ICU4X the abstractions we use to load data always use Yokes, allowing various data loading strategies — including caching — to be mixed

How it works

Speech bubble for character Positive pion
Manish is about to say the word “covariant” so I’m going to get ahead of him and say: If you have trouble understanding this and the next section, don’t worry! The internal workings of his crate rely on multiple niche concepts that most Rustaceans never need to care about, even those working on otherwise advanced code.

Yoke works by relying on the concept of a covariant lifetime. The Yokeable trait looks like this:

pub unsafe trait Yokeable<'a>: 'static {
    type Output: 'a;
    // methods omitted
}

and a typical implementation would look something like this:

unsafe impl<'a> Yokeable<'a> for Cow<'static, str> {
    type Output: 'a = Cow<'a, str>;
    // ...
}

An implementation of this trait will be implemented on the 'static version of a type with a lifetime (which I will call Self<'static>3 in this post), and maps the type to a version of it with a lifetime (Self<'a>). It must only be implemented on types where the lifetime 'a is covariant, i.e., where it’s safe to treat Self<'a> as Self<'b> when 'b is a shorter lifetime. Most types with lifetimes fall in this category4, especially in the space of zero-copy deserialization.

Speech bubble for character Positive pion
You can read more about variance in the nomicon!

For any Yokeable type Foo<'static>, you can obtain the version of that type with a lifetime 'a with <Foo as Yokeable<'a>>::Output. The Yokeable trait exposes some methods that allow one to safely carry out the various transforms that are allowed on a type with a covariant lifetime.

#[derive(Yokeable)], in most cases, relies on the compiler’s ability to determine if a lifetime is covariant, and doesn’t actually generate much code! In most cases, the bodies of the various functions on Yokeable are pure safe code, looking like this:

impl<'a> Yokeable for Foo<'static> {
    type Output: 'a = Foo<'a>;
    fn transform(&self) -> &Self::Output {
        self
    }
    fn transform_owned(self) -> Self::Output {
        self
    }
    fn transform_mut<F>(&'a mut self, f: F)
    where
        F: 'static + for<'b> FnOnce(&'b mut Self::Output) {
        f(self)
    }
    // fn make() omitted since it's not as relevant
}

The compiler knows these are safe because it knows that the type is covariant, and the Yokeable trait allows us to talk about types where these operations are safe, generically.

Speech bubble for character Positive pion
In other words, there’s a certain useful property about lifetime “stretchiness” that the compiler knows about, and we can check that the property applies to a type by generating code that the compiler would refuse to compile if the property did not apply.

Using this trait, Yoke then works by storing Self<'static> and transforming it to a shorter, more local lifetime before handing it out to any consumers, using the methods on Yokeable in various ways. Knowing that the lifetime is covariant is what makes it safe to do such lifetime “squeezing”. The 'static is a lie, but it’s safe to do that kind of thing as long as the value isn’t actually accessed with the 'static lifetime, and we take great care to ensure it doesn’t leak.

Better conversions: ZeroFrom

A crate that pairs well with this is zerofrom, primarily designed and written by Shane. It comes with the ZeroFrom trait:

pub trait ZeroFrom<'zf, C: ?Sized>: 'zf {
    fn zero_from(other: &'zf C) -> Self;
}

The idea of this trait is to be able to work generically with types convertible to (often zero-copy) borrowed types.

For example, Cow<'zf, str> implements both ZeroFrom<'zf, str> and ZeroFrom<'zf, String>, as well as ZeroFrom<'zf, Cow<'a, str>>. It’s similar to the AsRef trait but it allows for more flexibility on the kinds of borrowing occuring, and implementors are supposed to minimize the amount of copying during such a conversion. For example, when ZeroFrom-constructing a Cow<'zf, str> from some other Cow<'a, str>, it will always construct a Cow::Borrowed, even if the original Cow<'a, str> were owned.

Yoke has a convenient constructor Yoke::attach_to_zero_copy_cart() that can create a Yoke<Y, C> out of a cart type C if Y<'zf> implements ZeroFrom<'zf, C> for all lifetimes 'zf. This is useful for cases where you want to do basic self-referential types but aren’t doing any fancy zero-copy deserialization.

… make life rue the day it thought it could give you lifetimes

Life with this crate hasn’t been all peachy. We’ve, uh … unfortunately discovered a toweringly large pile of gnarly compiler bugs. A lot of this has its root in the fact that Yokeable<'a> in most cases is bound via for<'a> Yokeable<'a> (“Yokeable<'a> for all possible lifetimes 'a”). The for<'a> is a niche feature known as a higher-ranked lifetime or trait bound (often referred to as “HRTB”), and while it’s always been necessary in some capacity for Rust’s typesystem to be able to reason about function pointers, it’s also always been rather buggy and is often discouraged for usages like this.

We’re using it so that we can talk about the lifetime of a type in a generic sense. Fortunately, there is a language feature under active development that will be better suited for this: Generic Associated Types.

This feature isn’t stable yet, but, fortunately for us, most compiler bugs involving for<'a> also impact GATs, so we have been benefitting from the GAT work, and a lot of our bug reports have helped shore up the GAT code. Huge shout out to Jack Huey for fixing a lot of these bugs, and eddyb for helping out in the debugging process.

As of Rust 1.61, a lot of the major bugs have been fixed, however there are still some bugs around trait bounds for which the yoke crate maintains some workaround helpers. It has been our experience that most compiler bugs here are not restrictive when it comes to what you can do with the crate, but they may end up with code that looks less than ideal. Overall, we still find it worth it, we’re able to do some really neat zero-copy stuff in a way that’s externally convenient (even if some of the internal code is messy), and we don’t have lifetimes everywhere.

Try it out!

While I don’t consider the yoke crate “done” yet, it’s been in use in ICU4X for a year now and I consider it mature enough to recommend to others. Try it out! Let me know what you think!

Thanks to Finch, Jane, and Shane for reviewing drafts of this post

  1. A locale is typically a language and location, though it may contain additional information like the writing system or even things like the calendar system in use. ↩

  2. Bear in mind, this isn’t just a matter of picking a format like MM-DD-YYYY! Dates in just US English can look like 4/10/22 or 4/10/2022 or April 10, 2022, or Sunday, April 10, 2022 C.E., or Sun, Apr 10, 2022, and that’s not without thinking about week numbers, quarters, or time! This quickly adds up to a decent amount of data for each locale. ↩

  3. This isn’t real Rust syntax; since Self is always just Self, but we need to be able to refer to Self as a higher-kinded type in this scenario. ↩

  4. Types that aren’t are ones involving mutability (&mut or interior mutability) around the lifetime, and ones involving function pointers and trait objects. ↩

IRL (podcast) — When an Algorithm is Your Boss

Gig workers around the world report directly to algorithms in precarious jobs created by secretive corporations. We take you to the streets of Quito, Ecuador where delivery workers are protesting against artificial intelligence, and we hear solutions from people in several countries on how to audit the algorithms and reclaim rights.

Eduardo Meneses is gearing up with allies to ‘audit the algorithms’ of delivery platforms in Ecuador as the Global Head of Social Change at Thoughtworks.

Dan Calacci at the MIT Media Lab is developing open source tools and systems that empower workers to take control of their data.

Aída Ponce Del Castillo is working on AI regulation to protect the rights of platform workers as a lawyer with the European Trade Union Institute in Brussels.

Yuly Ramirez is the general secretary of a coalition of digital platform workers in Ecuador and José Gonzalez is a delivery driver in Quito, Ecuador.

IRL is an original podcast from Mozilla, the non-profit behind Firefox. In Season 6, host Bridget Todd shares stories of people who make AI more trustworthy in real life. This season doubles as Mozilla’s 2022 Internet Health Report. Go to the report for show notes, transcripts, and more.

 

The Rust Programming Language Blog — Increasing the glibc and Linux kernel requirements

The minimum requirements for Rust toolchains targeting Linux will increase with the Rust 1.64.0 release (slated for September 22nd, 2022). The new minimum requirements are:

  • glibc >= 2.17 (previously glibc >= 2.11)
  • kernel >= 3.2 (previously kernel >= 2.6.32)

These requirements apply both to running the Rust compiler itself (and other Rust tooling like Cargo or Rustup), and to running binaries produced by Rust, if they use the libstd.

If you are not targeting an old long-term-support distribution, or embedded hardware running an old Linux version, this change is unlikely to affect you. Otherwise, read on!

Affected targets

In principle, the new kernel requirements affect all *-linux-* targets, while the glibc requirements affect all *-linux-gnu* targets. In practice, many targets were already requiring newer kernel or glibc versions. The requirements for such targets do not change.

Among targets for which a Rust host toolchain is distributed, the following are affected:

  • i686-unknown-linux-gnu (Tier 1)
  • x86_64-unknown-linux-gnu (Tier 1)
  • x86_64-unknown-linux-musl (Tier 2 with host tools)
  • powerpc-unknown-linux-gnu (Tier 2 with host tools)
  • powerpc64-unknown-linux-gnu (Tier 2 with host tools)
  • s390x-unknown-linux-gnu (Tier 2 with host tools)

The following are not affected, because they already had higher glibc/kernel requirements:

  • aarch64-unknown-linux-gnu (Tier 1)
  • aarch64-unknown-linux-musl (Tier 2 with host tools)
  • arm-unknown-linux-gnueabi (Tier 2 with host tools)
  • arm-unknown-linux-gnueabihf (Tier 2 with host tools)
  • armv7-unknown-linux-gnueabihf (Tier 2 with host tools)
  • mips-unknown-linux-gnueabihf (Tier 2 with host tools)
  • powerpc64le-unknown-linux-gnueabihf (Tier 2 with host tools)
  • riscv64gc-unknown-linux-gnueabihf (Tier 2 with host tools)

For other tier 2 or tier 3 targets, for which no Rust toolchain is distributed, we do not accurately track minimum requirements, and they may or may not be affected by this change. *-linux-musl* targets are only affected by the kernel requirements, not the glibc requirements. Targets which only use libcore and not libstd are unaffected.

A list of supported targets and their requirements can be found on the platform support page.

Affected systems

The glibc and kernel versions used for the new baseline requirements are already close to a decade old. As such, this change should only affect users that either target old long-term-support Linux distributions, or embedded hardware running old versions of Linux.

The following Linux distributions are still supported under the new requirements:

  • RHEL 7 (glibc 2.17, kernel 3.10)
  • SLES 12-SP5 (glibc 2.22, kernel 4.12.14)
  • Debian 8 (glibc 2.19, kernel 3.16.7)
  • Ubuntu 14.04 (glibc 2.19, kernel 3.13)

The following distributions are not supported under the new requirements:

  • RHEL 6 (glibc 2.12, kernel 2.6.32)
  • SLES 11-SP4 (glibc 2.11.3, kernel 3.0.101)
  • Debian 6 (glibc 2.11, kernel 2.6.32), Debian 7 (glibc 2.13, kernel 3.2.41)
  • Ubuntu 12.04 (glibc 2.15, kernel 3.2)

Out of the distributions in the second list, only RHEL 6 still has limited vendor support (ELS).

Why increase the requirements?

We want Rust, and binaries produced by Rust, to be as widely usable as possible. At the same time, the Rust project only has limited resources to maintain compatibility with old environments.

There are two parts to the toolchain requirements: The minimum requirements for running the Rust compiler on a host system, and the minimum requirements for cross-compiled binaries.

The minimum requirements for host toolchains affect our build system. Rust CI produces binary artifacts for dozens of different targets. Creating binaries that support old glibc versions requires either building on an operating system with old glibc (for native builds) or using a buildroot with an old glibc version (for cross-compiled builds).

At the same time, Rust relies on LLVM for optimization and code generation, which regularly increases its toolchain requirements. LLVM 16 will require GCC 7.1 or newer (and LLVM 15 supports GCC 5.1 in name only). Creating a build environment that has both a very old glibc and a recent compiler becomes increasingly hard over time. crosstool-ng (which we use for most cross-compilation needs) does not support targeting both glibc 2.11, and using a compiler that satisfies the new LLVM requirements.

The requirements for cross-compiled binaries have a different motivation: They affect which kernel versions need to be supported by libstd. Increasing the kernel requirements allows libstd to use newer syscalls, without having to maintain and test compatibility with kernels that do not support them.

The new baseline requirements were picked as the least common denominator among long-term-support distributions that still have active support. This is currently RHEL 7 with glibc 2.17 and kernel 3.10. The kernel requirement is picked as 3.2 instead, because this is the minimum requirement of glibc itself, and there is little relevant API difference between these versions.

What should I do?

If you or your organization are affected by this change, there are a number of viable options depending on your situation:

  • Upgrade your target system, or raise the minimum requirements of your software, to satisfy the new constraints.
  • If you are running the Rust compiler on an old host, consider cross-compiling from a newer host instead.
  • If you are targeting an old glibc version, consider targeting musl instead.
  • If you are targeting an old kernel version and use libstd, you may be out of luck: In this case you may have to either freeze your current Rust version, or maintain a fork of libstd that supports older kernels.

Mike Hommey — Announcing git-cinnabar 0.5.10

Git-cinnabar is a git remote helper to interact with mercurial repositories. It allows to clone, pull and push from/to mercurial remote repositories, using git.

Get it on github.

These release notes are also available on the git-cinnabar wiki.

What’s new since 0.5.9?

  • Fixed exceptions during config initialization.
  • Fixed swapped error messages.
  • Fixed correctness issues with bundle chunks with no delta node.
  • This is probably the last 0.5.x release before 0.6.0.

Mozilla Localization (L10N) — L10n Report: July 2022 Edition

Please note some of the information provided in this report may be subject to change as we are sometimes sharing information about projects that are still in early stages and are not final yet. 

Welcome!

Are you a locale leader and want us to include new members in our upcoming reports? Contact us!

New content and projects

What’s new or coming up in Firefox desktop

While the last months have been pretty quiet in terms of new content for Firefox, we’re approaching a new major release for 2022, and that will include new features and dedicated onboarding.

Part of the content has already started landing in the last days, expect more in the coming weeks. In the meantime, make sure to check out the feature name guidelines for Firefox View and Colorways.

In terms of upcoming deadlines: Firefox 104 is currently in Beta and it will be possible to update translations up to August 14.

What’s new or coming up in mobile

Mobile releases now align more closely to desktop release schedules, so you may notice that target dates for these projects are the same in Pontoon. As with desktop, things are quiet now for mobile, but we’ll be seeing more strings landing in the coming weeks for the next major release.

What’s new or coming up in web projects

Firefox Relay website & add-on

We’re expanding Firefox Relay Premium into new locales across Europe: Austria, Belgium, Cyprus, Estonia, Finland, France, Germany, Greece, Ireland, Italy, Latvia, Lithuania, Luxembourg, Malta, Netherlands, Portugal, Slovakia, Slovenia, Spain, Sweden, and Switzerland. In order to deliver a truly great experience to our users in these new locales, we would like to make sure that users can utilize our products in the language they feel most comfortable with. Having these languages localized will take already complex topics like privacy and security and help connect more with users and offer them greater protections.

If you don’t see the product offered in the language in the markets above, maybe you can help by requesting to localize the product. Thank you for helping spread the word.

What’s new or coming up in Pontoon

  • When 100% TM match is available, it now automatically appears in the editor if the string doesn’t have any translations yet.

    100% matches from Translation Memory now automatically appear in the editor

  • Before new users make their first contribution to a locale, they are now provided with guidelines. And when they submit their first suggestion, team managers get notified.

    Tooltip with guidelines for new contributors.

  • The Contributors page on the Team dashboard has been reorganized. Contributors are grouped by their role within the team, which makes it easier to identify and reach out to team managers.

    Team contributors grouped by role.

  • We have introduced a new list parameter in translate view URLs, which allows for presenting a selected list of strings in the sidebar.
  • Deadlines have been renamed to Target Dates.
  • Thanks to Eemeli for making a bunch of under-the-hood improvements, which make our codebase much easier to build on.

Events

Want to showcase an event coming up that your community is participating in? Contact us and we’ll include it.

Friends of the Lion

Know someone in your l10n community who’s been doing a great job and should appear here? Contact us and we’ll make sure they get a shout-out!

Useful Links

Questions? Want to get involved?

If you want to get involved, or have any question about l10n, reach out to:

Did you enjoy reading this report? Let us know how we can improve it.

The Mozilla Blog — Riot Games’ head of player community, known as ‘Aureylian’ to game streaming fans, on her favorite corners of the internet

Here at Mozilla, we are the first to admit the internet isn’t perfect, but we are also quick to point out that the internet is pretty darn magical. The internet opens up doors and opportunities, allows for people to connect with others, and lets everyone find where they belong — their corners of the internet. We all have an internet story worth sharing. In My Corner Of The Internet, we talk with people about the online spaces they can’t get enough of, what we should save in Pocket to read later, and what sites and forums shaped them.

This month we chat with Erin Wayne, who’s known as “Aureylian” in the game streaming community and now the global head of player community for Riot Games. She talks about what she’s reading, how she launched a YouTube career by accident and what we can all learn from gamers.

What is your favorite corner of the internet?

I love baking, party planning, and all things crafty, so I’m a daily Pinterest and Etsy user. Right now, my mom and I are renovating a cabin built in the 1950s, so finding vintage items and inspiration to update our little piece of the world while retaining the original charm has been such a fun side project. I also find a lot of enjoyment supporting local/small business owners, and these are great ways to do that!

What is an internet deep dive that you can’t wait to jump back into? 

I love history and have recently become really invested in the time period leading up to the reign of Elizabeth I of England. To me, she’s one of the most fascinating people in history, and the period leading up to her reign is something I’m really excited about learning through historical records and documentation in addition to the many intriguing movies and TV shows about her life.

What is the one tab you always regret closing? 

My calendar! Between being a mom and running four global teams, I live by my calendar. I am lost without it!

What can you not stop talking about on the internet right now? 

Scotland. I’m going to be in Scotland this summer and I know that it will be an absolutely once-in-a-lifetime experience. I’m part Scottish by heritage, but it’s also a beautiful country, with incredible people, great food, deep history, and the most breathtaking sights. By sharing what I experience there, I hope it inspires others to go and take part in the culture.

What was the first online community you engaged with? 

Though I played World of Warcraft starting in 2005, I played with local friends and at LAN parties, not truly connected to guilds or an online community. It wasn’t really until 2012 when I began reaching out to people on Twitter to join a WoW-inspired Minecraft server I helped create that I really got into online communities. That in turn accidentally launched my YouTube career and ultimately is why I’m where I am today!

What articles and videos are in your Pocket waiting to be read/watched right now? 

Right now on my list, I’ve got a few lined up. As a woman in the gaming industry, I can’t wait to read “Pioneer Rediscovered: The Woman Who Brought Female Representation to Games.” I’m also a huge history lover, so I’ve saved “A History of the Smile Through Art, Culture and Etiquette”. Since our team recently onboarded about 20 new people, I’m also really interested in what the Harvard Business Review wrote regarding retaining employees, “The Key to Retaining Young Workers? Better Onboarding.”

What do you think the future of gaming will look like?

I think gaming companies are starting to finally understand that gamers, like all people, are multifaceted. We like games, but we also like music, TV, sports and fashion. The list goes on, and I’d expect gaming companies to start diversifying how they engage with their players to continue serving them in all of these places. That’s why I joined Riot. I believe Riot is at the forefront of innovating what it means to be a community-driven, player-focused game company that doesn’t just provide meaningful gaming experiences for players, but also serves them with countless opportunities to engage with its [intellectual property] across different lifestyles.

Generally speaking, I also expect games to continue focusing on the community aspect in-product to drive connection between people. Coming out of the pandemic, I hope industries outside of gaming start to understand what gamers always have: that connecting with each other in valuable and meaningful ways can, and does, happen digitally. I met my husband and best friends through social media/gaming and even my dad has found ways to move his tabletop games online to stay in touch with his lifelong friends during the pandemic. It’s one of the most personal and meaningful connections we have, free from the overwhelming nature of social media’s likes, comments, shares and virality to let us get back to the fundamentals of purely connecting with each other.

If you could create your own corner of the internet what would it look like? 

Probably a lot like my Pinterest boards (lol). It’s a great way to share what I’m working on or projects I want to start, the things I’ve been cooking (honestly, I wish Pinterest would put more effort into the “tried it” feature), places I want to travel, and funny or inspiring images, memes, and graphic design I’ve loved. Also, lots of Leslie Knope and Ted Lasso quotes.

Erin “Aureylian” Wayne is the global head of player community for Riot Games, overseeing community engagement across all of Riot’s titles including League of Legends, one of the most-played competitive video games in the world. She also leads the development of strategies and programming for Riot’s interactions with players across the globe related to community, editorial, influencer management and social media. Prior to Riot, Erin spent seven years at Twitch, where she created and led their community and creator marketing teams, which focused on engaging, exciting, and educating Twitch users.

The post Riot Games’ head of player community, known as ‘Aureylian’ to game streaming fans, on her favorite corners of the internet appeared first on The Mozilla Blog.

Mozilla Performance Blog — Performance Tools Newsletter (H1 2002)

As the Perf-Tools team, we are responsible for the Firefox Profiler. This newsletter gives an overview of the new features and improvements we’ve done in the first half of 2022.

You can find the previous newsletter here which was about the Q4 2021. This newsletter contains updates related to work done in the first half of 2022.

Here are some highlights.

Documentation

The main Profiler documentation was refreshed, to better match what has changed in recent years.

If you are new to the Profiler, or if you would like your friends, colleagues, or customers to learn about it, please visit: https://profiler.firefox.com/docs/#/./guide-getting-started.
(This link is also accessible from the landing page at https://profiler.firefox.com.)
Screenshot of the top of https://profiler.firefox.com/docs/#/./guide-getting-started

As a reminder, other aspects of the Profiler are captured in separate locations:

For developers of the Firefox application, or if you are just interested in some technical details, see the internal documentation at https://firefox-source-docs.mozilla.org/tools/profiler/.

For web developers, one point of interest is that you can add “markers” to your own pages through the JavaScript User Timing API, which will then be visible in profiles. This can be useful to measure some specific actions in your web pages.

Finally, the GitHub repository for the front-end side of the Profiler (the website that actually displays profiles after they have been captured) contains some detailed documentation about profile file formats and inner workings of the website.

New DevTools performance panel

The new performance panel was released in Firefox 98. This panel makes it easier for web developers to record and share profiles, as it opens a new profiler.firefox.com tab after capturing the profile.
Screenshot of the devtools performance panel

To access it, open the DevTools (main menu > More tools > Web Developer Tools), and select the “Performance” panel.

Note that having the DevTools open may use some processing power, so the resulting profile of any web page may be slightly skewed, with some of the profile data being related to DevTools. If these effects are noticeable in your profiles, just close the DevTools window while profiling. As an alternative, you may choose to control the profiler through the toolbar button (enabled when first visiting https://profiler.firefox.com), or the accompanying shortcuts: Ctrl+Shift+1 to start, and Ctrl+Shift+2 to stop.

Internationalization

The profiler.firefox.com interface is now available in languages other than English, and may be changed on the fly from the language picker at the bottom right cornerScreenshot of the language pickerYou may request new languages, and contribute, at https://pontoon.mozilla.org/projects/firefox-profiler/ after signing in with a Firefox Account.

Markers

A number of improvements happened around markers: how they’re collected, some new useful markers that were added around Firefox, and how they are better presented.

  • Instant markers (those without duration) with the same name are now grouped in the top line, instead of being sprinkled semi-randomly among interval markers. And they are all displayed as diamond shapes, making it easier to notice multiple adjacent markers.
  • Interval markers are now all displayed as rectangles with a visible darker outline, which better indicates if and where they start and stop. Very short markers that fit in 1 pixel look like dark vertical lines.
    Examples of different marker displays
  • Inter-Process Communication (IPC) markers are now captured from any thread, even those not currently being profiled, in order to give a more complete picture of interactions between Firefox processes. These messages have become more numerous and important since site isolation was improved by using separate Operating System processes for each web server address – so that web servers cannot spy on each other through a common web page.
    IPC markers also have a new context menu option, to select the other thread on the other side of the IPC message channel.
    IPC marker context menu showing "Select the send thread", and from there an arrow pointing at the corresponding marker for the IPC that was sent.
  • "Awake": In most threads, this shows the actual time spent running tasks (as opposed to waiting for these tasks). It also includes how much CPU is spent, and where available which CPU core ran it, and at which priority level. These markers are especially useful to see how often and when a thread woke up.
    "Awake" marker tooltip, with duration, CPU Id, and priority details
  • "DLLLoad": Time spent loading DLLs on Windows.
  • "Set/Sample/Clear Animation" markers on the compositor thread.
  • DOMEvent and CSS Animation/Transition markers now show the target of the event/animation
  • Screenshot marker tooltips now show the actual screenshot.
  • New context menu option “Copy page URL”.

Other improvements

  • We now support profiling of private browsing windows. Previously we were disabling the profiler as soon as a private window was opened. Now we profile them but mark everything as coming from a private window, so that we can remove the data if the profile is shared (which is the default).
  • Per-process CPU utilization. The feature is still marked as experimental but should be usable. Enable it in about:profiling, and when viewing the profile, open the JavaScript console and run experimental.enableProcessCPUTracks() to show the graphs.
    In particular, it can highlight work done in the whole process, which may not happen in the currently-visible threads; If you notice some high values there, but cannot find the corresponding work in individual threads, consider selecting more threads to profile before re-running the profiler.
    For example, in the following screenshot the zone marked ① looks idle in the main thread; but the “Process CPU” reveals that there was significant activity at ② in this process, and a bit of exploring hidden threads found that the “StyleThreads” at ③ were the ones working hard at that time.
    Profile with Process CPU track, see text above for explanation
  • Ability to capture stack traces on all versions of Firefox for macos. It used to only work on some pre-release channels.
  • Profiling data from early times during a process startup used to be split in their own tracks (annotated with “pre-xul”), they are now combined with the correct threads past this early startup phase.
  • The memory track’s tooltip now shows the number of memory operations performed between two samples.
  • Animations were removed from various places for users with prefer-reduced-motion.
  • Profiles captured from Linux perf now distinguish kernel and user frames with different colors. (Thank you to contributor Dave Rigby.)
  • Experimental support for capturing actual power usage (not just CPU utilization times) on some platforms (Windows 11 and Apple Silicon as of this writing). This work is still progressing, and will bring another set of measurements that are very important to improve the performance of Firefox and websites.
  • Miscellaneous internal performance improvements of the Profiler itself, to better handle capturing and displaying more and more data.

Meet the team, and get help

If you profiled something and are puzzled with the profile you captured, we have the Joy of Profiling (#joy-of-profiling:mozilla.org) channel where people share their profiles and get help from the people who are more familiar with the Firefox Profiler. If you’re interested in the Profiler itself, our main channel is https://matrix.to/#/#profiler:mozilla.org.

We also have Joy of Profiling Open Sessions where people bring their profile and we try to analyze the profile together over Zoom. See the Performance Office Hours calendar.

And if you would just like to see some Firefox developers analyze profiles, watch Mike Conley and friends in the original Joy of Profiling, “unscripted, unplanned, uncensored, and true to life”.

 

Until next time, Happy Profiling!

Firefox Nightly — These Weeks In Firefox: Issue 121

Highlights

Friends of the Firefox team

Resolved bugs (excluding employees)

Volunteers that fixed more than one bug

  • Janvi Bajoria [:janvi01]
  • sayuree
  • Shane Hughes [:aminomancer]

New contributors (🌟 = first patch)

Project Updates

Add-ons / Web Extensions

WebExtensions Framework
    • As part of the ongoing ManifestVersion 3 (MV3) work:
        • The new unified toolbar button is meant to replace the browserAction toolbar buttons but it does not cover all the features that are currently provided by the browserActions toolbar buttons yet. The new toolbar button is currently only enabled when the “extensions.unifiedExtensions.enabled” preference is explicitly set to true in about:config.
        • A huge shout out to both Itiel and James Teh for the great support they provided in the reviews for the unified extensions button!
      • Event Pages (non persistent background pages): In Firefox >= 104, the event page will not be terminated if there is an active connection to a native app through the native messaging APIs (browser.runtime.connectNative and browser.runtime.sendNativeMessage) – Bug 1770696
      • New web_accessible_resources manifest field syntax: In Firefox >= 104, in the “matches” properties of the web_accessible_resources entries in the manifest_version 3 format supports the “<all_urls>” host permission string – Bug 1776841
WebExtension APIs
  • In Firefox >=  104, restricted schemes (eg. “resources” or “chrome” schemes) are allowed in scripting.registerContentScripts API when called from a (MV2) privileged extension – Bug 1756758
  • Fixed a bug with browser.extension.getViews and preloaded browserAction popup pages – Bug 1780008
  • Starting from Firefox 104, the history WebExtensions API will use internal PlacesUtils.history async API
    • Some extensions (e.g. DownThemAll! add-on) were calling the history API during startup and so this fix will also result in a startup performance improvement for Firefox users that have extensions using this API and blocking web requests triggered during the Firefox startup.
    • Huge shout out to Emilio for investigating and fixing this issue!

Developer Tools

Toolbox
  • Many thanks to arai for helping us with instant evaluation issues (bug & bug).
    • Those patches will prevent the following expression to be instantly evaluated as the user is typing
      • [1, 2, 3].map(alert)
      • Function.prototype.call.bind(Function.prototype.call)(alert);
  • We fixed a bug for log points in the Browser Toolbox where the logs would only appear if the “Show Content Messages” setting was checked (bug)
  • Julian fixed an issue with debugging addons + reloading (easily triggered when using webext to write your extension) (bug)
    • Uplifted to ESR
  • Bomsy worked on a couple things in the Netmonitor we should improve memory usage and performance
    • Network monitoring is now disabled in the Browser Toolbox until the user selects the  Netmonitor (bug)
    • We now properly clear data when the request list is cleared (bug)
  • Ochameau is still making progress on Debugger source tree stability and performance (bug,  bug,  bug and bug, showing decent performance improvements to the Browser Toolbox: DAMP results)
WebDriver BiDi

ESMification status

Lint, Docs and Workflow

Picture-in-Picture

Performance

  • hiro and florian closed out this bug which can cause the compositor to run at 60hz at all times even when nothing is animating on Windows!
  • emilio made chrome windows support document.visibilityState, which means the refresh driver is now throttled in fully occluded background windows.

Performance Tools (aka Firefox Profiler)

  • Removed the timeline graph type radio buttons (#4147)

    • The "before" state of the Profiler UI is shown, with a section showing the radio buttons for changing the graph type between "Categories with CPU", "Categories" and "Stack height".

      Before

    • The "after" state of the Profiler UI is shown. The section showing the graph type radio buttons is gone.

      After!

    • If you are a power user and would like to use another timeline graph type, you can call window.toggleTimelineType from the devtools console with various types. See the console message in profiler.firefox.com for more details.
  • Profiler no longer crashes when the profile data is too big, instead we discard only the profile of the child process that was too big, and we log error messages in the JSON file. It’s visible in profile.profileGatheringLog from the console. (Bug 1779685, Bug 1758643, Bug 1779367)
  • Added a power profiling setting to the profiler popup (Power usage data in Watt is available only on Windows 11 with Intel CPUs and Apple Silicon, but the preset can still be used elsewhere for a low overhead profiling of what’s causing thread wake-ups) (Bug 1778282) You can change the profiler setting either via profiler popup or about:profiling.

    • The Profiler settings UI showing a list of radio buttons. Each radio button sets the Profiler into a preset configuration. A new configuration is highlighted for profiling Power Usage.

      This will be very handy for finding power consumption optimizations!

  • Added profiler sub-category for Wasm frames (Bug 1780383)
  • Added doc for local profiling on android with screenshots. Here’s the link to the doc. (#4145)
  • Hid the user interface components showing stacks for tracks that don’t have stack samples (#4133)

Search and Navigation

The Mozilla Blog — Mozilla celebrates groundbreaking creators in new docuseries “Firefox Presents”

Different is dope. Firefox has always stood by this. It’s also the mantra of Abby Wren, the woman featured in the pilot episode of our new docuseries, “Firefox Presents.”

Launching Friday, April 15, “Firefox Presents” is a documentary series featuring colorful and inspiring creators who each have a unique journey of finding themselves or their community online. We speak with them about how they are using the internet to overcome obstacles, challenge the status quo and express themselves in a way that encourages and inspires other people to feel welcome and safer online. 

Abby Wren, an alopecia advocate and makeup artist who represents the alopecia and bald communities online through her Tiktok channel. Finding an online community after her diagnosis helped her feel less alone, and she’s now doing the same for others in return.

“There are all these other people who had alopecia that I didn’t know existed before the internet brought us together,” Abby said in our behind-the-scenes interview with her.

Each episode will be released monthly on Firefox’s YouTube channel. Upcoming episodes will feature activist, writer and actor Brandon Kyle Goodman, featured on Netflix’s “Human Resources,” and engineer, entrepreneur and YouTuber Xyla Foxlin. 

The “Firefox Presents” series is produced by Mozilla in collaboration with Long Haul Films, and is executive produced by Steve Flavin (Mozilla) and directed by Melissa Dowler (Long Haul Films).

Firefox browser logo

Get Firefox

Get the browser that protects what’s important

The post Mozilla celebrates groundbreaking creators in new docuseries “Firefox Presents” appeared first on The Mozilla Blog.

The Mozilla Blog — Parents want to keep their kids safe online. But are parental controls the answer?

For our new series Parental Control, we’re digging into the challenges technology poses for parents and exploring all the ways that can empower them. So we looked into digital platforms and found ourselves, as many parents do, in the parental control settings. 

These settings, along with services that promise to shield young people away from “inappropriate” content, can give families comfort in the face of the infinite feed. They let adults limit screen time and restrict mature content (although the way platforms identify what that means is far from perfect). But it is not so simple as setting up the parental control settings and walking away. It’s important for parents to both understand their kids’ behaviors, and explain to them why they’re using parental controls.

The capabilities of these tools, as well as their shortcomings, led us to question the very name of our series: In a world where technology’s hold over everything we do seems uncontrollable, what does parental control even mean?

Jenny Radesky, who studies the intersections of child development, parenting and technology at the University of Michigan, takes issue with the phrase itself.

“Parental mediation is [a better] term, parental engagement is another – and probably better because it implies meaningful discussion or involvement to help kids navigate media, rather than using controlling or restricting approaches,” said Radesky, who has contributed to the American Academy of Pediatrics’ policy agenda on kids’ technology use. 

She pointed to research that suggests letting children manage their own media consumption may be more effective than parental control settings offered by apps. 

In one study called “Coco’s Videos,” researchers designed a video-streaming app for preschoolers. In it, a character named Coco interacts with children as they watch videos. The researchers had the kids use three different versions of the app.

In the neutral version, a child sees a large “home” button after watching a video playlist. That button leads them back to the beginning where they can make a new playlist. 

In the “post-play” version, a child sees the same home button. But this time, there’s a small screen embedded in the top right corner which automatically plays a recommended video. The child can either expand that window to full-screen and keep watching, pause or go back to the home screen. 

In the controlled version, a child is locked out of the app once they’ve finished a video playlist. After three minutes, the app resets and returns to the home screen. 

Researchers found that the post-play version that automatically plays another video, a feature used by platforms such as YouTube and Netflix, “significantly reduced children’s autonomy and likelihood of self-regulation, extended video-viewing time, and led to increases in parent intervention.” Meanwhile, the version that used a lockout mechanism didn’t cut down screen time or the likelihood of parent intervention. 

The study concluded that we don’t need to make additional tools to control excessive media use; we just need to stop creating experiences that encourage it. 

In another study, preschoolers and parents were asked to create a device-based playtime plan together. Researchers observed parent-child interactions and interviewed the parents afterwards. The experts found that children, with parental guidance during the planning phase, moved on to their next activity without their parents having to intervene 93% of the time.

Alex Hiniker, who co-authored both studies, thinks that communication between parents and children about technology can be empowering. But platforms continue to be designed to get as much time and attention as possible. 

“Slapped on top of them are these lockout mechanisms and timers and say, ‘OK, now self-police yourself into not using the super enthralling things that we just put in front of you,’” Hiniker said. “I’m not a big fan of that approach. I don’t really think it’s working for families.”

How well parental control settings and apps work is one question. It’s also worth asking where the balance lies between protecting children online and encroaching on their independence and privacy. 

Parental controls and children’s privacy

Jason Kelley, a strategist for the nonprofit Electronic Frontier Foundation, which advocates for digital rights, worries that parental controls may be normalizing surveillance for children.

“You have to think of strict parental controls in a young person’s mind as essentially a parent sitting and watching them use the internet over their shoulder,” Kelley argued. “This can send a really bad message that safety is only available and possible through surveillance, and that’s simply not true.”

He acknowledged the good intentions behind efforts that seek to rein in social media and other digital platforms that increasingly take up young people’s time. But most parental control tools don’t recognize the different privacy needs of a toddler and that of a teenager, Kelley said, and filtering systems aren’t great at recognizing context. Not only could parental control settings block “thinspiration” posts, they could also restrict posts about body positivity.

“How we protect the mental health of young people online is a reasonable question,” Kelley said. “But there’s also the real question of whether Instagram is worse than, say, ‘Tiger Beat’ magazine. Our culture is set up in a way to make people feel bad about themselves. And Instagram is a reflection of that in some cases. But it’s impossible to eliminate bad things from someone’s online experience.”

Kelley said as a society, we want to instill in younger generations why privacy matters. He also underscored that not all adults act for the best interest of minors. And that risk is clear for marginalized groups. 

The internet has risks, but so do parental controls

It’s important to realize that many kids in the LGBTQI+ community can be made vulnerable by tech monitoring tools, especially in a country where things like conversion therapy are still practiced, said Christopher Wood. He’s the executive director of LGBT Tech, which develops and offers technology resources to support LGBTQI+ communities.

He noted that outside the home, sensitive information about young people can already be exposed to teachers and campus administrators through the school devices they use. Wood said he runs a local LGBTQ+ center in Virginia, where he gets calls from young people getting kicked out of their homes because their families found out their sexual orientation or gender identity — most often through technology.

Recent legal developments, such as Florida’s “don’t say gay” bill and the Texas Supreme Court’s ruling on gender-affirming care for trans teens, heighten privacy concerns. Information exposed by a monitoring tool could land young people, their parents or their teachers in legal trouble, with LGBTQ+ youth at the most risk.

Wood understands the uncertainties families feel about technology and parents’ need to create a safe space for their children on the internet. 

“There’s a push and pull with any child and parent,” he said. “I’m a parent. For me, it’s about creating an opportunity where my child can feel safe to come to me and talk to me if they get into trouble, while also providing the opportunity for them to explore. Technology is only going to become faster, and it will continue to infiltrate more parts of their lives.”

Wood believes that education is key, and not just for kids. 

“Sometimes, I’m like, ‘What are you doing? How are you doing that?” he said of his interactions with his children. “We need to create opportunities for parents to feel educated and feel like they’re not beating their heads against the wall.” 

Researchers like Radesky and Hiniker agree that when it comes to technology and its effects on young people, the onus shouldn’t fall on parents. 

Hiniker said more experts like her are now exploring how platform design can support meaningful relationships within families and give power back to users, including children.

In the “Coco’s Videos” study, Hiniker observed how kids interacted with their parents after watching their videos. In the app, the character Coco reminds the child of their next activity, like sleeping, reading a book or going outside.

“I loved listening to these really sweet moments,” Hiniker recalled. “They’d say, ‘Hey, Mom, it’s time for me to go outside and you need to find my boots for me because it’s raining.’ Kids sort of understood that they’re in control, and it seemed really empowering to them.”

This design is different from our idea of parental controls, Hiniker said, but it could be our best bet in raising children who grow up to have healthy relationships with technology. 

In the end, parents may not be able to find foolproof parental control tools that can shelter their kids from the internet’s imperfections. But families can find comfort in the fact that the best control may be the one young people feel as they learn about the powers of being online – all while knowing that they have whatever support they need offline. 

The internet is a great place for families. It gives us new opportunities to discover the world, connect with others and just generally make our lives easier and more colorful. But it also comes with new challenges and complications for parents raising the next generations. Mozilla wants to help parents make the best online decisions for their kids, whatever that looks like, in our latest series, Parental Control. 

Firefox browser logo

Get Firefox

Get the browser that protects what’s important

The post Parents want to keep their kids safe online. But are parental controls the answer? appeared first on The Mozilla Blog.

Mozilla Privacy Blog — Mozilla submits comments in OSTP consultation on privacy-preserving data sharing

Earlier this month, the US Office of Science and Technology Policy (OSTP) asked stakeholders to contribute to the development of a national strategy for “responsibly harnessing privacy-preserving data sharing and analytics to benefit individuals and society.” This effort offers a much-needed opportunity to advance privacy in online advertising, an industry that has not seen improvement in many years.

In our comments, we set out the work that Mozilla has undertaken over the past decade to shape the evolution of privacy preserving advertising, both in our products, and in how we engage with regulators and standards bodies.

Mozilla has often outlined that the current state of the web is not sustainable, particularly how online advertising works today. The ecosystem is broken. It’s opaque by design, rife with fraud, and does not serve the vast majority of those which depend on it – most importantly, the people who use the open web. The ways in which advertising is conducted today – through pervasive tracking, serial privacy violations, market consolidation, and lack of transparency – are not working and cause more harm than good.

At Mozilla, we’ve been working to drive the industry in a better direction through technical solutions. However, technical work alone can’t address disinformation, discrimination, societal manipulation, privacy violations, and more. A complementary regulatory framework is necessary to mitigate the most egregious practices in the ecosystem and ensure that the outcomes of such practices (discrimination, electoral manipulation, etc.) are untenable under law rather than due to selective product policy enforcement.

Our vision is a web which empowers individuals to make informed choices without their privacy and security being compromised.  There is a real opportunity now to improve the privacy properties of online advertising. We must draw upon the internet’s founding principles of transparency, public participation, and innovation. We look forward to seeing how OSTP’s national strategy progresses this vision.

The post Mozilla submits comments in OSTP consultation on privacy-preserving data sharing appeared first on Open Policy & Advocacy.

This Week In Rust — This Week in Rust 453

Hello and welcome to another issue of This Week in Rust! Rust is a programming language empowering everyone to build reliable and efficient software. This is a weekly summary of its progress and community. Want something mentioned? Tweet us at @ThisWeekInRust or send us a pull request. Want to get involved? We love contributions.

This Week in Rust is openly developed on GitHub. If you find any errors in this week's issue, please submit a PR.

Updates from Rust Community

Official
Foundation
Project/Tooling Updates
Observations/Thoughts
Rust Walkthroughs

Crate of the Week

This week's crate is cargo-semver-checks, a CI-friendly tool to check your library's API.

Thanks to Predrag Gruevski and Matthias Beyer for the (self and other) nominations.

Please submit your suggestions and votes for next week!

Call for Participation

Always wanted to contribute to open-source projects but didn't know where to start? Every week we highlight some tasks from the Rust community for you to pick and get started!

Some of these tasks may also have mentors available, visit the task page for more information.

If you are a Rust project owner and are looking for contributors, please submit tasks here.

Updates from the Rust Project

397 pull requests were merged in the last week

Rust Compiler Performance Triage

Overall it was a mostly good week, with some very significant wins among the secondary benchmarks. Rollups continue to complicate triage process.

Triage done by @pnkfelix. Revision range: 8bd12e8c..50166d5e

Summary:

mean max count
Regressions 😿
(primary)		 N/A N/A 0
Regressions 😿
(secondary)		 2.2% 3.2% 6
Improvements 🎉
(primary)		 -1.8% -21.2% 199
Improvements 🎉
(secondary)		 -2.6% -9.0% 124
All 😿🎉 (primary) -1.8% -21.2% 199

5 Regressions, 4 Improvements, 4 Mixed; 4 of them in rollups 61 artifact comparisons made in total

Full report here

Call for Testing

An important step for RFC implementation is for people to experiment with the implementation and give feedback, especially before stabilization. The following RFCs would benefit from user testing before moving forward:

  • No RFCs issued a call for testing this week.

If you are a feature implementer and would like your RFC to appear on the above list, add the new call-for-testing label to your RFC along with a comment providing testing instructions and/or guidance on which aspect(s) of the feature need testing.

Approved RFCs

Changes to Rust follow the Rust RFC (request for comments) process. These are the RFCs that were approved for implementation this week:

Final Comment Period

Every week, the team announces the 'final comment period' for RFCs and key PRs which are reaching a decision. Express your opinions now.

RFCs
  • No RFCs entered Final Comment Period this week.
Tracking Issues & PRs
New and Updated RFCs

Upcoming Events

Rusty Events between 2022-07-27 - 2022-08-24 🦀

Virtual
Asia
Europe
North America
Oceania

If you are running a Rust event please add it to the calendar to get it mentioned here. Please remember to add a link to the event too. Email the Rust Community Team for access.

Jobs

Please see the latest Who's Hiring thread on r/rust

Quote of the Week

JuSt Be CaReFuL

If there’s one lesson from decades of software engineering, it is the failure of “just be careful” as a strategy. C/C++ programmers still experience memory corruption constantly, no matter how careful they are. Java programmers still frequently see NullPointerExceptions, no matter how careful they are. And so on. One of the reasons that Rust is so successful is that it adds automated checks to prevent many common mistakes.

– Robert Grosse on their blog

Thanks to robin for the suggestion!

Please submit quotes and vote for next week!

This Week in Rust is edited by: nellshamrell, llogiq, cdmistman, ericseppanen, extrawurst, andrewpollack, U007D, kolharsam, joelmarcey, mariannegoldin.

Email list hosting is sponsored by The Rust Foundation

Discuss on r/rust

Mozilla Thunderbird — Thunderbird Time Machine, 2003: A Look Back At Thunderbird 0.1

Let’s take a walk down memory lane to the summer of 2003. Linkin Park, 50 Cent, and Evanescence have top-selling new albums. Apple’s iPod hasn’t even sold 1 million units. Mozilla’s new web browser used to be called Phoenix, but now it’s called Firebird. And a new cross-platform, open-source application called Thunderbird has debuted from the foundations of Mozilla Mail…

Because the entirety of Thunderbird’s releases and corresponding release notes have been preserved, I’ve started a self-guided tour of Thunderbird’s history. Why? A mixture of personal and technical curiosity. I used Thunderbird for a couple years in the mid-2000s, and again more recently, but there are giant gaps in my experience. So I’m revisiting every single major version to discover the nuances between releases; the changes big and small.

(If you ever get the craving to do the same, I’ve found the easiest operating system to use is Windows, preferably inside a virtual machine. Early versions of Thunderbird for Macs were built for PowerPC architecture, while early Linux versions were 32-bit only. Both may cause you headaches with modern PC hardware!)

3-Pane Mail Layout: A Solid Foundation!

Below is my screenshot of Thunderbird 0.1 running on a Windows 11 virtual machine.

The first thing you’re probably thinking is “well, not much has changed!” With respect to the classic 3-pane mail presentation, you’re absolutely right! (Hey, why mess with a good thing?)

A screenshot of Thunderbird 0.1 from 2003, running on modern hardware and Windows 11.

Thousands of changes have been made to the client between Thunderbird 0.1 and Thunderbird 102, both under the hood and cosmetically. But it’s clear that Thunderbird started with a strong foundation. And it remains one of the most flexible, customizable applications you can use.

Something else stands out about that screenshot above: the original Thunderbird logo. Far removed from the modern, flat, circular logo we have today, this original logo simply took the Mozilla Phoenix/Firebird logo and gave it a blue coat of paint:

The original Mozilla Phoenix/Thunderbird logos<figcaption>The original Mozilla Thunderbird (top) and Mozilla Phoenix (bottom) logos</figcaption>

Thunderbird 0.1 Release Notes: “Everything Is New”

Back in 2003, much of what we take for granted in Thunderbird now was actually groundbreaking. Things like UI extensions to extend functionality, and user-modifiable theming were forward-thinking ideas. For a bit of historical appreciation, here are the release notes for Thunderbird 0.1:

  • Customizable Toolbars and Mail 3-pane: Toolbars can be customized the way you want them. Choose View / Toolbars / Customize inside any window. Mozilla Thunderbird also supports a new vertical 3-pane configuration (Tools / Options / General), giving you even more choice in how you want to view your mail.
  • Extensions: UI extensions can be added to Mozilla Thunderbird to customize your experience with specific features and enhancements that you need. Extensions allow you to add features particular to your needs such as offline mail support. A full list of available extensions can be found here.
  • Contacts Manager: A contacts sidebar for mail compose makes it easy and convenient to add address book contacts to emails.
  • Junk Mail Detection: In addition to automatically detecting junk mail using the same method as Mozilla Mail, Thunderbird also sanitizes HTML in mail marked as junk in order to better protect your privacy and give peace of mind when viewing a message identified as junk.
  • New default theme: Mozilla Thunderbird 0.1 sports a crisp, fresh and attractive theme, based on the amazing Qute theme by Arvid Axelsson. This is the same theme used by Mozilla Firebird, giving Thunderbird a similar look and feel. Thunderbird also supports a growing number of downloadable themes which alter the appearance of the client.
  • Stream-lined user interface and simplified Options UI.
  • Integrated spell checker.

Next Time, Inside The Thunderbird Time Machine…

A fictitious entry from a Livejournal page, circa December 2004:

“I had a super productive weekend! Finally finished Half-Life 2 and cannot wait for the sequel! I also upgraded my Dell Inspiron 7000 laptop from Windows 98 to Windows XP, so it’s time to install Firefox 1.0 and Thunderbird 1.0. Looking forward to trying this new open-source software!”

Thunderbird is the leading open-source, cross-platform email and calendaring client, free for business and personal use. We want it to stay secure and become even better. Donations allow us to hire developers, pay for infrastructure, expand our userbase, and continue to improve.

Click here to make a donation. 

The post Thunderbird Time Machine, 2003: A Look Back At Thunderbird 0.1 appeared first on The Thunderbird Blog.

Tiger Oakes — You should use ReadonlyArray in your React state

Preventing a common React bug.

The Mozilla Blog — Firefox Presents: A nerdcore rap artist defying expectations

As far as music subgenres go, nerdcore hip-hop wouldn’t exist in its current form without the internet. Those who create it — self-proclaimed nerds who rap about video games, science fiction or just everyday life as a misfit — record music at home, collaborate via web file transfers and find cheerleaders out of strangers online. 

That’s how Alex Sun Liu, who goes by the stage name LEX the Lexicon Artist, found her people.

Liu learned how to beatbox through beatboxing forums as an overachieving teen in Taiwan. While perfecting violin solos and being top of her class, she traded pointers with beatboxers from Germany and France. 

Now 28, Liu has expanded her repertoire to three nerdcore albums and has toured with like-minded artists across the U.S. She raps about her psychology degree while referencing Descartes. She rhymes about hookups with jokes using hotel chain puns. In a music video for a track titled “Peep Game,” she talks up her skills while lying on a bed of marshmallow chicks. It all sounds very offbeat, and that’s the point.

“Regardless of how weird you are, there’s always going to be a pocket for you to exist in,” Liu said. “If you’re a goth, there will be a goth scene for you. If you love metal, there will be metalheads. If you’re a certain type of queer, there will be a queer community for you.”

LEX the Lexicon Artist smiles at the camera while sitting down.<figcaption>Photo: Nita Hong for Mozilla</figcaption>

She credits the internet for expanding her world outside Taipei, and later, Berkeley, California, where she attended college, saying that “you don’t realize how limiting your locale is until you get on the internet and realize that there are so many people all across the country, all across the world, doing similar things.”

In addition to other Bay Area musicians like spoken word artist Watsky and Daveed Diggs of “Hamilton” fame, Liu is inspired by the South Korean singer Psy.

“He embraced what made him unique and doesn’t conform to what people expect K-pop to be,” Liu said. “He’s middle-aged, not a perfect Ken or Barbie, but an amazing dancer and rapper with incredible charisma and star power. He turned his ‘Gangnam Style’ internet moment into a long-lasting global career.”

While Liu’s not a household name (she holds a day job as a booking manager for a New York City arts venue that “specializes in nerdy entertainment”), her fans are very engaged with her work. They show it through supportive comments on Instagram and YouTube. Liu may not share a fan base with the hip-hop artists who formed her musical inclinations in Taiwan (Eminem, Dr. Dre and 50 Cent), but her verses have found the right corners of rap internet — where listeners not only appreciate a good beat, but also lyrics about embracing your weird. 

She cites “Artist Anthem” as a fan favorite, a track about being a creative person set to poppy melody. Released in 2018 as part of her first album, “Raging Ego,” it starts with a verse introducing herself as an artist who not only writes her own songs but does it well. From there, the song takes a turn: “I am an artist and I hate myself!” 

“I think people enjoy the juxtaposition of goofy and dark in that song, and anyone with some level of an artist mindset – whether they’re a writer, a musician, a visual artist, or any type of creative – can relate to the constant flipping back and forth between ‘I love myself!’ and ‘I hate myself!’” Liu said. 

She said she was drawn to hip-hop artists as a young person because of how they conveyed their unfiltered emotions.  

“My first album was quite self-indulgent. I wanted to express this socially unacceptable part of myself,” Liu explained.

What she thought was “self-involved” resonated with others. 

“A lot of fans have told me that the song is ‘too real,’ and that they almost can’t laugh at it because it’s so relatable,” Liu said. 

Soon, through connections she made on the internet, Liu performed at a nerdcore showcase at South by Southwest in Austin. There she saw people she had only met online, and she went on to tour nationally. 

“I expected touring would be how my life was going to be,” Liu recalled. Then 2020 happened. 

LEX the Lexicon Artist raps on stage holding a mic in front of a crowd.<figcaption>Photo: Nita Hong for Mozilla</figcaption>

A daughter of doctors, including one who “studied his way out of poverty,” Liu excelled academically but always felt she was a performer at heart. Without the joy she got from performing on stage, she began to question her music career. She held marketing positions in the years after college, but it wasn’t until she landed her current role at an arts company when things finally clicked. Now, she takes time off work to perform on the road.

While taping this interview with Firefox during a tour stop in Los Angeles last spring, Liu joked that it was “probably the closest I’ll ever get to being famous.” But she said she doesn’t subscribe to any defined standards for an artist, an Asian American, or for that matter, any human.

“We’re too worried about being perfect and being normal,” Liu said. “I want to show people that you can find your own success by making your own scene, and by being very weird.” 

Firefox is exploring all the ways the internet makes our planet an awesome place. Almost everything we do today ties back to the online world in some way — so, join us in highlighting the funny, weird, inspiring and courageous stories that remind us why we love the world wide web.

LEX the Lexicon Artist holds a phone while looking at the camera, sitting down.

Get the browser that makes a difference

Download Firefox

The post Firefox Presents: A nerdcore rap artist defying expectations appeared first on The Mozilla Blog.

Data@Mozilla — This Week in Data: Python Environment Freshness

(“This Week in Glean Data” is a series of blog posts that the Glean Team at Mozilla is using to try to communicate better about our work. They could be release notes, documentation, hopes, dreams, or whatever: so long as it is inspired by Glean. You can find an index of all TWiG posts online.)

By: Perry McManis and Chelsea Troy

A note on audience: the intended reader for this post is a data scientist or analyst, product owner or manager, or similar who uses Python regularly but has not had the opportunity to work with engineering processes to the degree they may like. Experienced engineers may still benefit from the friendly reminder to keep their environments fresh and up-to-date.

When was the last time you remade your local Python environment? One month ago? Six months ago? 1997?

Wait, please, don’t leave. I know, I might as well have asked you when the last time you cleaned out the food trap in your dishwasher was and I apologize. But this is almost as important. Almost.

If you don’t recall when, go ahead and check when you made your currently most used environment. It might surprise you how long ago it was.

# See this helpful stack overflow post by Timur Shtatland: https://stackoverflow.com/a/69109373
Mac: conda env list -v -v -v | grep -v '^#' | perl -lane 'print $F[-1]' | xargs /bin/ls -lrtd
Linux: conda env list | grep -v '^#' | perl -lane 'print $F[-1]' | xargs ls -lrt1d
Windows: conda env list
# Find the top level directory of your envs, e.g. C:\Users\yourname\miniconda3\envs
Windows: dir /T:C C:\Users\yourname\miniconda3\envs

Don’t feel bad though, if it does surprise you, or the answer is one you’d not admit publicly. Python environments are hard. Not in the everything is hard until you know how way, but in the why doesn’t this work? This worked last week! way. And the impetus is often to just not mess with things. Especially if you have that one environment that you’ve been using for the last 4 years, you know the one you have propped up with popsicle sticks and duct tape? But I’d like to propose that you consider regularly remaking your environments, and you build your own processes for doing so.

It is my opinion that if you can, you should be working in a fresh environment.

Much like the best by date, what is fresh is contextual. But if you start getting that when did I stand this env up? feeling, it’s time. Working in a fresh environment has a few benefits. Firstly, it makes it more likely that other folks will be able to easily duplicate it. Similarly to how providing an accurate forecast becomes increasingly difficult as you go further into the future, as you get further away from the date you completed a task in a changing ecosystem, the less likely it is that task can be successfully completed again.

Perhaps even more relevant is that packages often release security updates, APIs improve, functionality that you originally had to implement yourself may even get an official release. Official releases, especially for higher level programming languages like Python, are often highly optimized. For many researchers, those optimizations are out of the scope of their work, and rightly so. But the included version of that calculation in your favorite stats package will not only have several engineers working on it to make it run as quickly as possible, now you have the benefit of many researchers testing it concurrently with you.

These issues can collide spectacularly in cases where people get stuck trying to replicate your environment due to a deprecated version of a requirement. And if you never update your own environment, it could take someone else bringing it up to you to even notice that one of the packages you are using is no longer available, or an API has been moved from experimental to release, or removed altogether.

There is no best way of making fresh environments, but I have a few suggestions you might consider.

I will preface by saying that my preference is for command line tools, and these suggestions reflect that. Using graphical interfaces is a perfectly valid way to handle your environments, I’m just not that familiar with them, so while I think the ideas of environment freshness still apply, you will have to find your own way with them. And more generally, I would encourage you to develop your own processes anyway. These are more suggestions on where to start, and not all of them need find their way into your routines.

If you are completely unfamiliar with these environments, and you’ve been working in your base environment, I would recommend in the strongest terms possible that you immediately back it up. Python environments are shockingly easy to break beyond repair and tend to do so at the worst possible time in the worst possible way. Think live demo in front of the whole company that’s being simulcast on youtube. LeVar Burton is in the audience. You don’t want to disappoint him, do you? The easiest way to quickly make a backup is to create a new environment through the normal means, confirm it has everything you need in it, and make a copy of the whole install folder of the original.

If you’re not in the habit of making new environments, the next time you need to do an update for a package you use constantly, consider making an entirely new environment for it. As an added bonus this will give you a fallback option in case something goes wrong. If you’ve not done this before, one of the easiest ways is to utilize pip’s freeze function.

pip list --format=freeze > requirements.txt
conda create -n {new env name}
conda activate {new env name}
pip install -r requirements.txt
pip install {package} --upgrade

When you create your requirements.txt file, it’s usually a pretty good idea to go through it. A common gotcha is that you might see local file paths in place of version numbers. That is why we used pip list here. But it never hurts to check.

Take a look at your version numbers, are any of these really out of date? That is something we want to fix, but often some of our important packages have dependencies that require specific versions and we have to be careful not to break those dependencies. But we can work around that while getting the newest version we can by removing those dependencies from our requirements file and installing our most critical packages separately. That way we let pip or conda get the newest versions of everything that will work. For example, if I need pandas, and I know pandas depends on numpy, I can remove both from my requirements document and let pip handle my dependencies for me.

pip install --upgrade pip
pip install -r requirements.txt
pip install pandas

Something you may notice is that this block looks like it should be something that could be packaged up since it’s just a collection of commands. And indeed it can. We can put this in a shell script and with a bit of work, add a command line option, to more or less fire off a new environment for us in one go. This can also be expanded with shell commands for cases where we may need a compiler, tool from another language, a github repo even, etc. Assuming we have a way to run shell scripts,  Let’s call this create_env.sh:

conda deactivate
conda create -n $1
conda activate $1apt install gcc

apt install g++pip install --upgrade pip
pip install pystan==2.19.1.1
python3 -m pip install prophet --no-cache-dir

pip install -r requirements.txt
pip install scikit-learn

git clone https://github.com/mozilla-mobile/fenix.git

cd ./fenix

echo "Finished creating new environment: $1"

And by adding some flag handling, now using bash we can call sh create_env.sh newenv and be ready to go.

It will likely take some experimentation the first time or two. But once you know the steps you need to follow, getting new environment creation down to just a few minutes is as easy as packaging your steps up. And if you want to share, you can send your setup script rather than a list of instructions. Including this in your repository with a descriptive name and a mention in your README.md is a low effort way to help other folks get going with less friction.

There are of course tons of other great ways to package environments, like Docker. I would encourage you to read into them if you are interested in reproducibility beyond the simpler case of just rebuilding your local environment with regularity. There are a huge number of fascinating and immensely powerful tools out there to explore, should you wish to bring even more rigor to your Python working environments.

In the end, the main thing is to work out a fast and repeatable method that enables you to get your environment up and going again quickly from scratch. One that works for you. And then when you get the feeling that your environment has been around for a while, you won’t have to worry about making a new environment being an all-day or even worse, all-week affair. By investing in your own process, you will save yourself loads of time in the long run, and you may even save your colleagues some too. And hopefully, save yourself some frustration, too.

Like anything, the key to working out your process is repetitions. The first time will be hard, though maybe some of the tips here can make it a bit easier. But the second time will be easier. And after a handful, you will have developed a framework that will allow you to make your work more portable, more resilient and less angering, even beyond Python.

Support.Mozilla.Org — Introducing Smith Ellis

Hi everybody,

I’m so happy to introduce our latest addition to the Customer Experience team. Smith Ellis is going to join forces with Tasos and Ryan to develop our support platform. It’s been a while since we got more than 2 engineers on the team, so I’m personally excited to see what we can unlock with more engineers.

Here’s a bit of an intro from Smith:

Hello Mozillians!  I’m Smith Ellis, and I’m joining the Customer Experience team as a Software Engineer. I’m more than happy to be here. I’ve held many technical and management roles in the past and have found that doing work that makes a difference is what makes me happy. My main hobbies are electronics, music, video games, programming, welding and playing with my kids.

I look forward to meeting you and making a difference with Mozilla.

Please join me to congratulate and welcome Smith into our SUMO family!

Anne van Kesteren — Apple

I have often wondered what it would be like to work for Apple and now I have the privilege to find out! There’s a lot of amazing people on the WebKit team I have had the pleasure of meeting and collaborating with. Getting to do a lot more of that will be terrific. Writing this it’s still somewhat difficult to comprehend it’s actually happening.

Apple itself is the sole institution that develops computers end-to-end for the rest of us. It is hard to overstate how exciting it is to be a small part of that for a person that enjoys computing systems. And WebKit too is truly great. With origins in KHTML it’s a web browser engine that’s over two decades old now. But from my eagle-eye perspective the code doesn’t look it. The community cares about refactoring, long-term maintainability, and hackability of the code, as well as making it more accessible to new contributors with the ongoing move to GitHub. A lot of care seems to be placed in how it is evolved: maintaining compatibility, ensuring new code reuses existing primitives, and standards are implemented to the letter. (Or if you’re Antti, you might not read the letters and just make the tests pass.) And I have reason to believe this isn’t just the community. That these values are shared by Apple.

Suffice to say, I am humbled to have been given this opportunity and look forward to learning a lot!

The Rust Programming Language Blog — Announcing Rust 1.62.1

The Rust team has published a new point release of Rust, 1.62.1. Rust is a programming language that is empowering everyone to build reliable and efficient software.

If you have a previous version of Rust installed via rustup, you can get 1.62.1 with:

rustup update stable

If you don't have it already, you can get rustup from the appropriate page on our website, and check out the detailed release notes for 1.62.1 on GitHub.

What's in 1.62.1 stable

Rust 1.62.1 addresses a few recent regressions in the compiler and standard library, and also mitigates a CPU vulnerability on Intel SGX.

Contributors to 1.62.1

Many people came together to create Rust 1.62.1. We couldn't have done it without all of you. Thanks!

Firefox Nightly — These Weeks In Firefox: Issue 120

Highlights

  • Dale Harvey implemented an experimental Quick Actions feature in the Address Bar
Screenshot of the View Source quick action option displayed in the address bar after typing "view" in search

Quick Actions such as “View Source” are available in the address bar when enabled.

Screenshot of sample color-mix notation on used in dev tools

color-mix takes in two color values and mixes them by a given amount.

  • Thanks to evilpie, we now show entries for URLSearchParams instances in the console and debugger
Screenshot of sample URLSearchParams displayed on a browser console

URLSearchParams can now be previewed when desired.

Friends of the Firefox team

Resolved bugs (excluding employees)

Volunteers that fixed more than one bug

  • Itiel
  • Michael Kohler [:mkohler]

New contributors (🌟 = first patch)

Project Updates

Add-ons / Web Extensions

WebExtensions Framework

  •  Extension CSP is now also applied to WebExtensions Workers – Bug 1685627

Developer Tools

  • Raphael added an inactive notice when `border-image*` is used on internal table elements where `border-collapse` is set to `collapse` (Bug 1583910)
  • Julian fixed an issue which was causing blank DevTools when debugging some WebExtension (Bug 1777296)
  • Opening the Browser Console will no longer cause scripts to pause on `debugger` statements (Bug 1766128)
  • Bomsy added a button to remove all breakpoints in the Debugger (Bug 1742774)

    Screenshot of the "Remove all breakpoints" button in dev tools

    Too many breakpoints? Clear them all with the click of a button.

Screenshot of power usage tracks in about:profiling

Inside the profiler, there are new tracks showing power usage, with a tooltip showing the instant power usage.

    • This example comes from this profile: https://share.firefox.dev/3nNc2xSTake care to disable sampling with the “nostacksampling” option to have good measurements (soon a preset will be added to make it easier to make this work best (bug 1778282)).
      This is some huge work done by Florian Quèze.

  • Add a pen icon to indicate profile names are editable (github issue #4099 by first-time contributor Bhavya Joshi 🎉🎆)

    Screenshot of the pen icon next to a profile name in Firefox Profiler

    The pen icon for profile names

  • More label frames: bug 1749521, bug 1776721, bug 1777473. This exposes:
    • more browser internals with understandable names to javascript developers (Style Computation, CSS Parsing, Layout tree destruction)
    • Most javascript APIs implemented in C++ from: String, Object, TypedArray, Number, Date, Intl, RegExp, Map, Set, Reflect, JSON, BigInt, Array.
    • This makes the profiler much more useful for JS developers.
  • Fix the buffer size handling when reaching the configured maximum size. Previously we had a bug where the buffer size would always increase, eventually causing an OOM crash.
  • LUL initialization on Linux has been optimised – but still more to come!

Search and Navigation

  • James Teow has disabled the “Did you mean to go to <site>“ notification bar, which was quite annoying for users having a wildcarded DNS server, resolving any word to an helper page. It can be re-enabled setting `browser.urlbar.dnsResolveSingleWordsAfterSearch` to 1. Bug 1735534
  • Michael Kohler made the Remove button disabled for the default Search Engine, that can’t be removed anyway. Bug 1761513
  • Mandy Cheang is refactoring part of the Search Service to improve maintainability and robustness: Bug 1777278, Bug 1777599, Bug 1777632
  • Daisuke Akatsuka has corrected a regression where autofill could duplicate the protocol. Bug 1778335

IRL (podcast) — The Tech We Won’t Build

Where should tech builders draw the line on AI for military or surveillance? Just because it can be built, doesn’t mean it should be. At what point do we blow the whistle, call out the boss, and tell the world? Find out what it’s like to sound the alarm from inside a big tech company.

Laura Nolan shares the story behind her decision to leave Google in 2018 over their involvement in Project Maven, a Pentagon project which used AI by Google.

Yves Moreau explains why he is calling on academic journals and international publishers to retract papers that use facial recognition and DNA profiling of minority groups.

Yeshimabeit Milner describes how the non-profit Data for Black Lives is pushing back against use of AI powered tools used to surveil and criminalize Black and Brown communities.

Shmyla Khan, describes being on the receiving end of technologies developed by foreign superpowers as a researcher with the Digital Rights Foundation in Pakistan.

IRL is an original podcast from Mozilla, the non-profit behind Firefox. In Season 6, host Bridget Todd shares stories of people who make AI more trustworthy in real life. This season doubles as Mozilla’s 2022 Internet Health Report. Go to the report for show notes, transcripts, and more.

Mike Hommey — Announcing git-cinnabar 0.5.9

Git-cinnabar is a git remote helper to interact with mercurial repositories. It allows to clone, pull and push from/to mercurial remote repositories, using git.

Get it on github.

These release notes are also available on the git-cinnabar wiki.

What’s new since 0.5.8?

  • Updated git to 2.37.1 for the helper.
  • Various python 3 fixes.
  • Fixed stream bundle
  • Added python and py.exe as executables tried on top of python3 and python2.
  • Improved handling of ill-formed local urls.
  • Fixed using old mercurial libraries that don’t support bundlev2 with a server that does.
  • When fsck reports the metadata as broken, prevent further updates to the repo.
  • When issue #207 is detected, mark the metadata as broken
  • Added support for logging redirection to a file
  • Now ignore refs/cinnabar/replace/ refs, and always use the corresponding metadata instead.
  • Various git cinnabar fsck fixes.

The Mozilla Blog — Bridget Todd, the new host of Mozilla’s IRL podcast, on online communities, The X-Files & raising marginalized voices

Here at Mozilla, we are the first to admit the internet isn’t perfect, but we are also quick to point out that the internet is pretty darn magical. The internet opens up doors and opportunities, allows for people to connect with others, and lets everyone find where they belong — their corners of the internet. We all have an internet story worth sharing. In My Corner of the Internet, we talk with people about the online spaces they can’t get enough of, what we should save in Pocket to read later and what sites and forums shaped them.

This month we are announcing the latest season of Mozilla’s podcast, IRL, and its new host, Bridget Todd. Mozilla’s podcast, IRL, is launching Monday, July 18th anywhere you listen to podcasts. We sit down with her to talk about the podcast, how The X-Files AIM chat rooms helped her fall in love with the internet, and how the internet needs to become messy again in this month’s edition of My Corner of the Internet.

What is an internet deep dive that you can’t wait to jump back into?

This is a little bit of a specific answer, but the podcast Dead Eyes. I’m just really obsessed with this story of why Tom Hanks thought this guy had dead eyes. Every little twist and turn, I get stuck on, like Tom Hanks apologizing for getting him fired. It’s one of those stories that shows one of my favorite things about the Internet, how people can revisit and reclaim their experiences in a new light. I think as strange as that story is or how sort of like silly it is initiatives, I think it does reflect how the Internet can play this role in our lives that really empowers us to like rethink our experiences and see them with new, fresh eyes and process them in a different way.

What is the one tab you always regret closing?

I always have too many tabs open — at any given time I have like 40 open. It’s the kind of thing, where you know when you have so many open that you can’t even see them. So, whenever, I accidentally close a tab, I tell myself, you know, maybe you weren’t going to read that long article in The Atlantic, anyway, or like maybe you weren’t actually going to apply for that fellowship and it’s good that the link is no longer just up there. Every time I accidentally close the tab I feel like it’s the universe telling me that it wasn’t meant to be and you’re free now.

What can you not stop talking about on the internet right now? 

I cannot stop talking about all of the ways that people have been historically and traditionally sidelined and marginalized, in general, but also, particularly in conversations centered around technology and the Internet. Yet despite all those ways, we’re still taking up space online, creating conversations online, and building monuments to our experiences using technology. It’s been amazing to cover these stories, on my own technology podcast There Are No Girls On The Internet on iHeartRadio and I’m super excited to do more of that with [the re-launch of Mozilla’s podcast] IRL.

I’m also really proud of the work I am doing with UltraViolet — we’re working to build an internet that is more hospitable for women, queer folks, trans people, people of color and other people who are traditionally marginalized. 

What was the first online community you engaged with?

I’m almost embarrassed to say but The X Files chat rooms on America Online. because I really liked The X Files growing up, and I still do. I had a lot of questions about The X Files — what was happening and what was the truth. I also just like really enjoyed talking about Dana Scully, she’s my favorite character on TV.  I didn’t really have a lot of people to talk to about this — I was watching The X Files pretty much alone in my house — and so the Internet really gave me this way to talk about it. When I first discovered chat rooms on the good old-fashioned America Online, that was really when my love of the Internet took off. So, definitely, The X Files-related AOL chat rooms.

What stories are you most excited to tell with the latest season of IRL coming out this July?

The ones I’m the most excited to talk about are honestly, the stories where tech workers have blown the whistle — stories where they’ve said I’m working on technology that is not good, it’s not safe and I’m not going to do it. I think we forget that oftentimes it’s the rank and file employees at big tech companies who are really putting themselves at great risk to make change by letting the general public in on what’s happening at some of these companies. Those stories, for whatever reason, are stories that really moved me because I feel they take so much courage. It takes so much gumption and personal values to do that kind of thing. Those are the stories that really resonate with me and I’m super excited to chronicle on IRL this season. 

What articles and videos are in your Pocket waiting to be read/watched right now?

An article called on The Daily Beast called, Why Does Hollywood Keep Failing Maya Rudolph? I really love the actress Maya Rudolph and it’s about her new show, called Loot, which I have not watched yet. 

I’m really into pop culture and the intersection of pop culture and race and gender and identity so that’s definitely one that’s been in my pocket. More recently I saved this article that just came out on Wired titled Are You Ready To Be Surveilled Like A Sex Worker? It’s all about FOSTA/SESTA laws and the de-platforming of sex workers in relation to Roe V Wade being overturned. I’m kind of high and low on that, like up and down, escapism and reality, but those are the two that stand out to me.

If you could create your own corner of the internet what would it look like?

My corner of the Internet would be delightfully weird, I yearn to go back to the days, when Instagram was just really unappetizing looking pictures of your lunch and blurry pictures. Back in the day, we would go out to a nightclub and you would bring your digital camera and would take 50 pictures like ‘Oh, these are my friends from this angle from that angle” and then you would upload all of them to Facebook as if somebody wanted to click through every picture.

I miss those days when internet spaces were not curated, there was no expectation that you had to show up in a certain kind of way to get engagement. It felt like oddballs just trying stuff out. If I was in charge of the Internet, it would be a lot weirder, it would be a lot less pretty, a lot less curated and a lot less polished. When we post a picture today we think of all these different things. Back then, I would post photos and I was uglier, I was sweatier, I was less polished but I was free and having much more fun.  

I think my corner of the internet would be a lot more authentic and a lot more true to who we are and what we’re trying to say when we put ourselves out there. 

Bridget Todd is a frequently cited expert, trainer, and speaker on combating disinformation and extremism online, advocating for social media platform accountability, creating safer digital experiences for women and other marginalized people, and celebrating and amplifying marginalized people’s contributions to tech and the internet. 

She created her critically acclaimed podcast. There Are No Girls on the Internet to explore how marginalized people show up online in response to the lack of inclusion in conversations around the internet. 

As Director of Communication for the national gender-justice advocacy organization UltraViolet, Bridget regularly meets with leadership from platforms like Reddit, Twitter, Facebook and TikTok to advocate for and develop policy recommendations to make digital experiences safer and more inclusive. Bridget’s writing and work on technology, race, gender, and culture have been featured in The Atlantic, Newsweek, The Nation, The Daily Show and several other outlets.

The post Bridget Todd, the new host of Mozilla’s IRL podcast, on online communities, The X-Files & raising marginalized voices appeared first on The Mozilla Blog.

The Rust Programming Language Blog — Announcing Rustup 1.25.1

The rustup working group is announcing the release of rustup version 1.25.1. Rustup is the recommended tool to install Rust, a programming language that is empowering everyone to build reliable and efficient software.

If you have a previous version of rustup installed, getting rustup 1.25.1 is as easy as stopping any programs which may be using Rustup (e.g. closing your IDE) and running:

rustup self update

Rustup will also automatically update itself at the end of a normal toolchain update:

rustup update

If you don't have it already, you can get rustup from the appropriate page on our website.

What's new in rustup 1.25.1

This version of rustup fixes a regression introduced in the previous release (1.25.0), which caused some workflows to fail.

Regression in nested Cargo invocations with different toolchains

When you invoke Rust or Cargo installed by rustup, you're not running them directly. Instead, you run rustup "proxy" binaries, whose job is to detect the right toolchain (parsing the +channel CLI argument or using one of the defaults) and run it.

Running these proxies is not instantaneous though, and for example a cargo build invocation might execute several of them (the initial cargo invocation plus one rustc for every dependency), slowing down the build.

To improve performance, rustup 1.25.0 changed the proxies code to set the RUSTC and RUSTDOC environment variables when missing, which instructed Cargo to skip the proxies and invoke the binaries defined in those variables directly. This provided a performance gain when building crates with lots of dependencies.

Unfortunately this change broke some users of rustup, who did something like:

  • The first Cargo invocation (for example an extension, an alias or an integration test) uses toolchain foo, setting the RUSTC and RUSTDOC environment variables pointing to that toolchain.

  • The first invocation calls Cargo again, but this time using toolchain bar (for example cargo +bar build). This does not set the RUSTC and RUSTDOC environment variables pointing to bar, as those variables are already present.

  • The second invocation of Cargo then invokes rustc by using the RUSTC environment variable and skipping the proxy, which results in the foo toolchain being invoked. Previous versions of rustup invoked the proxy instead, which would correctly detect and use the bar toolchain.

Rustup 1.25.1 fixes this regression by reverting the change. The rustup working group is discussing in issue #3035 plans to re-introduce the change in a future release while avoiding breakage.

Thanks

Thanks again to all the contributors who made rustup 1.25.1 possible!

  • Daniel Silverstone (kinnison)
  • Robert Collins (rbtcollins)
  • Joshua Nelson (jyn514)
  • Pietro Albini (pietroalbini)

The Rust Programming Language Blog — Changes in the Core Team

We want to say farewell and thanks to a couple of people who are stepping back from the Core Team:

  • Aidan Hobson Sayers is leaving the Core Team and other roles. Joining the Core Team in 2018, he started out on the project with fixing CI issues and has been a member of the Infrastructure Team since its inception (moving through lead and co-lead). Aidan wants to dedicate more time to working with the Rust community and growing Rust usage at his company, and is looking forward to doing so from the other side of the fence.

  • Ashley Williams will be stepping down from the Core Team and other roles. She became a member of the Core Team in 2018 and has had impact on many parts of the project, from leading the Community team, to setting up PagerDuty for Infrastructure and Crates.io, to the Wasm working group, to Increasing Rust’s Reach and Rustbridge, to her instrumental work creating the Rust Foundation and serving as its first Executive Director. Ashley is leaving her role in the project to focus on her newly founded company.

Many thanks to both of them for their contributions and we look forward to seeing their future efforts with Rust!

Mozilla Performance Blog — Migrating to Browsertime for Performance Testing

Originally, we used a Web Extension for doing performance testing on Firefox in our Raptor test harness. But, we needed to add new features such as visual metrics, so in this post, I’ll briefly describe the steps we took to migrate Raptor to Browsertime.

We now have enabled Browsertime by default in our Raptor harness both locally, and in Continuous Integration (CI) but for some time, we needed to use the flag `–browsertime` to enable it. This work started with Nick Alexander, Rob Wood, and Barret Rennie adding the flag in bug 1566171. From there, others on the Performance team (myself included), began testing Browsertime, preparing the Raptor harness, and building up the infrastructure required for running Browsertime tests in CI.

Our primary motivation for all of this work was obtaining visual metrics. If you’ve never heard of visual metrics before, they can be summed up as performance metrics processed from a video recording of a pageload. You can find more information about these from this article titled Improving Firefox Page Load, by Bas Schouten. Initially, our visual metrics processing system used a two-machine system where one machine would run the test and the other would process the video recordings to obtain the metrics. This worked well for some time until we found some issues with it that were a large point of friction when it came to using our tooling. In Reworking our Visual Metrics Processing System, I describe these issues and how we overcame them. It suffices to say that we now use a single machine in CI, and that those issues were resolved.

After building an MVP, we needed to answer the question: can we catch as many, or more regressions with Browsertime? This is important to answer as we don’t want to swap engines and then find out that we catch less regressions than we did before. There are a couple ways of finding an answer to this and one of the most obvious is back-testing on all previous valid performance alerts to see if Browsertime also sees the changes. But this is unfeasible because it would be extremely time-consuming to go through all of the alerts we have. Instead, we ran our existing pageload tests using Browsertime in CI and then performed a statistical comparison of the changes to the data profile versus the web-extension.

We primarily made use of Levene’s test to compare the two which looked at how the variance differed. We didn’t care about changes to the mean/average of the results because changing the underlying engine that we use was bound to make our metrics change in some way. We only analyzed our high-impact tests here, or those that often had valid performance alerts, and found that in most cases the changes were acceptable on all 4 platforms (Linux, OSX, Windows, Android). At the same time, we saw improvements in most of our warm pageloads because we started using a new testing mode called “chimera” which ran 1 cold (first navigation), and 1 warm pageload (second navigation) for each browser session on each test page. We decided that the benefits of having visual metrics outweighed the risks of removing the web-extension which were few and mainly related to some increases in variance.

With the analysis complete and successful, we began migrating our platforms one at a time and only migrating our pageload tests to start with. After that, we worked through all of our additional tests such as benchmarks, and resource-usage tests. This was a slow process as we didn’t want to overload sheriffs with new performance alerts, and it allowed us to resolve issues that we found along the way. Throughout this time, we worked with Peter Hedenskog on improving Browsertime, and implementing new features there for Firefox. Each migration took approximately 2 weeks from the time we enabled the Browsertime tests to the time we disabled the web-extension tests.

Now, we’ve made it to a point where nearly all of our tests are running on Browsertime! The only remaining tests are power, and memory usage tests as we had no easy way to measure these in Browsertime until recently, in the next few months these should be removed and all of our web-extension code will be also. We’ve made great use of Browsertime for other forms of testing as well now because using a Geckodriver-based engine allows for much more customizable testing versus the web-extension. Furthermore, we’ve started building up our tools for processing these videos which you can find in our mozperftest-tools repository. We use these to make it easier to understand how/why some performance metrics changed in a given test, and some work is ongoing to make it easier for developers to use these tools. For instance, we can make GIFs that make it easy to spot the differences:

Tumblr pageload

Before/After Video of a Tumblr Pageload

Thanks to the whole Performance team for all the effort put into this migration and Peter Hedenskog, creator of Browsertime and Sitespeed.io, for all the help!

 

Mozilla Performance Blog — Reworking our Visual Metrics Processing System

Our visual metrics processing system used to use two separate machines to produce visual metrics from our pageload tests. In this post, I’ll describe how we moved to a single-machine system that also brought about many other benefits for us.

Note: If you’ve never heard of visual metrics before, they can be summed up as performance metrics processed from a video recording of a pageload. You can find more information about these from this article titled Improving Firefox Page Load, by Bas Schouten.

Over the last little while, we’ve been busy moving from a Web Extension-based performance testing system to a Browsertime-based one. I’ve briefly detailed this effort in my post on Migrating Raptor to Browsertime. Originally, our visual metrics processing system in Continuous Integration (CI) used two machines to produce the visual metrics, one for running the test, and the other for processing the video recordings to obtain the visual metrics. This made sense at the time as the effort required to try to run the visual metrics script on the same machine as the task was far too large at the beginning. There were also concerns around speed of execution on the machines, and the installation of dependencies that were not very portable across the various platforms we test.

However, while migrating to Browsertime, it quickly became clear that this system was not going to work. In CI, this system was not built for the existing workflow and we had to implement multiple special-cases to handle Browsertime tests differently. Primarily, our test visualization, and our test retriggering capabilities needed the most work. In the end, those hacks didn’t help much because the developers couldn’t easily find the test results to being with! Local testing wasn’t much better either: dependencies were difficult to install and handle across multiple platforms and python versions, retriggering/rebuilding tasks on Try was no longer possible with --rebuild X, and, in general, running visual metrics was very error-prone.

This was an interesting problem and one that we needed to resolve so developers could easily run visual metrics locally, and in CI with a simple flag, without additional steps, and as similar as possible to the rest of our test harnesses. There were multiple solutions that we thought of trying first such as, building a microservice to process the results, or using a different script to process visual metrics. But each of those came with more issues than we were trying to solve. For instance, the microservice would alleviate the need for two-machines in CI, but then we would have another service outside of Taskcluster that we would need to maintain and this wouldn’t resolve issues with using visual metrics locally.

After looking over the issues and the script again (linked here), I realized that our main issue with using this script in CI, and locally was the ImageMagick dependency. It was not a very portable piece of software, and it frequently caused issues when installing. The other non-Python dependency was FFMPEG but this was easy to install on all of our platforms, and we were already doing this automatically. Installing the Python-dependencies were also never an issue. This meant that if we remove the ImageMagick dependency from the script, we could use it on the same machine in CI, and eliminate the friction of using visual metrics locally. The small risk with doing this was that the script would run a bit slower as we would have more Python-based computations which can be slow compared to other languages. That said, the benefits outweighed the risks here so I moved forward with this solution.

I began by categorizing all the usages of ImageMagick in the script, and then converted all of them, one by one, into a pure-python implementation. I used my own past experience with image processing to do this, and I also had to look through their documentation and their source code to determine how they built each algorithm so I could end up with the same results before and after this rework. This was very important because we didn’t want to rework the script to then receive possibly faulty metrics. At the same time, I even managed to fix an outstanding bug in the hero element calculations by removing the alpha channel that made the edges of the hero element blurry, and this also revealed an issue in the hero element sizing (obtaining sizes of 0) calculations which are produced elsewhere.

Once I finished with the rework, I was immediately able to use it locally, and in CI. This next phase of the rework involved assuring that the metrics before and after the change are exactly the same in all cases. In all cases except for Contentful Speed Index (CSI), the metrics matched perfectly. For CSI, the issue was that we had a different underlying algorithm for Canny-Edge detection and different parameters from what we used to use that caused this mismatch. I worked with Bas Schouten, and the performance team to resolve this issue by finding parameters that matched what we expected from the edge-detection the most, and provided similar but not exact matches in terms of metrics. We agreed upon some parameters, ran some final tests, and then confirmed that this is what we want from CSI. In other words, while I was reworking this metric, we also managed to slightly improve it’s accuracy. In the image below, note how the edges that we used to use (in green) were very noisy and had sporadic pixels defined as edges, while the new method (in red) catches the content much better without so much variance in the edges produced. The old method also has a slight shift in the edges because the Gaussian smoothing kernel used was asymmetric.

Sigma Comparison of Canny-Edge Detection

Red: Edges from the new edge detection method. Green: Edges from the old edge detection method. Left: New method using a sigma of 0.7, right: new method using a sigma of 1.5. In both cases, new method edges are overlaid onto the old edges in Green (same on both sides).

 

Throughout this work, Peter Hedenskog was also very helpful with testing the changes and notifying me about bugs he found as he’s also interested in using this script in Browsertime (thank you!). Once we were all done with testing, and the metrics matched perfectly, or had minor acceptable differences, then we were able to finalize a series of patches and land it so that developers could enjoy using our tools with much less friction. Thank you to Kash Shampur for reviewing all the patches! Today, developers are able to retrigger tasks without issues or special cases, are able to easily find the results of a test in CI, and can easily use visual metrics locally without having to install extra dependencies by themselves. The risk about slower computation turned out to be a non-issue as the script runs just as quickly. This change also resulted in almost halving the number of tasks we have defined for Browsertime tests in CI as we no longer duplicate them for visual metrics which, in theory, should make the Gecko Decision task slightly faster. Overall, reworking this nearly legacy piece of code that is critical to modern performance testing was a great success! You can find the new visual metrics processing script here if you are interested.

Support.Mozilla.Org — What’s up with SUMO – July

Hi everybody,

There is a lot going on in Q2 but we also accomplished many things too! I hope you’re able to celebrate what you’ve contributed and let’s move forward to Q3 with renewed energy and excitement!

Welcome note and shout-outs

  • Welcome kaie, alineee, lisah9333, and Denys. Thanks for joining the KB world.
  • Thanks to Paul, Infinity, Anokhi, Noah, Wes, and many others for supporting Firefox users in the iOS platform.
  • Shout-outs to Kaio Duarte for doing a great job on being a social support moderator.
  • Congratulations to YongHan for getting both the l10n and forum badge in 2022. Keep up the good work!

If you know anyone that we should feature here, please contact Kiki and we’ll make sure to add them in our next edition.

Community news

  • You should be able to watch the scrum meeting without NDA requirement by now. Subscribe to the AirMozilla folder if you haven’t already, so you will get notifications each time we added a new recording.
  • How is our localization community is doing? Check out the result of the SUMO localization audit that we did in Q2. You can also watch the recording of my presentation at the previous community meeting in June.
  • Are you enthusiastic about helping people? We need more contributors for social and mobile support! 
  • Are you a Thunderbird contributor? We need you! 
  • Say hi to Ryan Johnson, our latest addition to the CX team!

Catch up

  • Watch the monthly community call if you haven’t. Learn more about what’s new in June! Reminder: Don’t hesitate to join the call in person if you can. We try our best to provide a safe space for everyone to contribute. You’re more than welcome to lurk in the call if you don’t feel comfortable turning on your video or speaking up. If you feel shy to ask questions during the meeting, feel free to add your questions on the contributor forum in advance, or put them in our Matrix channel, so we can answer them during the meeting.
  • To catch up on product releases update, please watch the recording of the Customer Experience scrum meeting from AirMozilla.
  • Consider subscribe to Firefox Daily Digest to get daily updates about Firefox from across different platforms.
  • Also, check out SUMO Engineering Board to see what the platform team is currently doing.

Community stats

KB

KB pageviews (*)

* KB pageviews number is a total of KB pageviews for /en-US/ only
Month Page views Vs previous month
May 2022 7,921,342 3.19%
Jun 2022 7,787,739 -1.69%

Top 5 KB contributors in the last 90 days: 

KB Localization

Top 10 locales based on total page views

Locale May 2022 pageviews (*) Jun 2022 pageviews (*) Localization progress (per Jul, 11)(**)
de 7.93% 7.94% 97%
zh-CN 6.86% 6.69% 100%
fr 6.22% 6.17% 89%
es 6.28% 5.93% 29%
pt-BR 5.26% 4.80% 52%
ru 4.03% 4.00% 77%
ja 3.75% 3.63% 46%
zh-TW 2.07% 2.26% 4%
It 2.31% 2.20% 100%
pl 1.97% 1.96% 87% 
* Locale pageviews is an overall pageviews from the given locale (KB and other pages)

** Localization progress is the percentage of localized article from all KB articles per locale

Top 5 localization contributors in the last 90 days: 

Forum Support

Forum stats

-TBD-

Top 5 forum contributors in the last 90 days: 

Social Support

Channel Total incoming conv Conv interacted Resolution rate
May 2022 376 222 53.30%
Jun 2022 319 177 53.04%

Top 5 Social Support contributors in the past 2 months: 

  1. Kaio Duarte
  2. Bithiah K
  3. Magno Reis
  4. Christophe Villeneuve
  5. Felipe Koji

Play Store Support

Channel May 2022 Jun 2022
Total priority review Total reviews replied Total priority review Total reviews replied
Firefox for Android 570 474 648 465
Firefox Focus for Android 267 52 236 39
Firefox Klar Android 4 1 3 0

Top 5 Play Store contributors in the past 2 months: 

  • Paul Wright
  • Tim Maks
  • Kaio Duarte
  • Felipe Koji
  • Selim Şumlu

Product updates

To catch up on product releases update, please watch the recording of the Customer Experience scrum meeting from AirMozilla. You can also subscribe to the AirMozilla folder to get notifications each time we added a new recording.

Useful links:

The Rust Programming Language Blog — Announcing Rustup 1.25.0

The rustup working group is happy to announce the release of rustup version 1.25.0. Rustup is the recommended tool to install Rust, a programming language that is empowering everyone to build reliable and efficient software.

If you have a previous version of rustup installed, getting rustup 1.25.0 is as easy as stopping any programs which may be using Rustup (e.g. closing your IDE) and running:

rustup self update

Rustup will also automatically update itself at the end of a normal toolchain update:

rustup update

If you don't have it already, you can get rustup from the appropriate page on our website.

What's new in rustup 1.25.0

This version of Rustup involves a significant number of internal cleanups, both in terms of the Rustup code and its documentation. In addition to a lot of work on the codebase itself, due to the length of time since the last release this one has a record number of contributors and we thank you all for your efforts and time.

One of the biggest changes in 1.25.0 is the new offer on Windows installs to auto-install the Visual Studio 2022 compilers which should simplify the process of getting started for people not used to developing on Windows with the MSVC-compatible toolchains.

A second important change for 1.25.0 is a number of PRs focussed around startup performance for Rustup. While it may not seem all that important to many, Rustup's startup time is a factor in the time it takes to do builds which involve large numbers of crates on systems which do not have large numbers of CPU cores. Hopefully the people for whom this is a common activity will notice an improvement; though there's more opportunity to speed things up still available.

Some, but by no means all, of the rest of this release's highlights includes support for rustup default none to unset the default toolchain, support for Windows arm64, inclusion of rust-gdbgui as a proxy so that platforms which support it can use GDB's gui mode with Rust, and some improvements to rustup-init.sh.

Full details are available in the changelog!

Rustup's documentation is also available in the rustup book.

Thanks

Thanks again to all the contributors who made rustup 1.25.0 possible!

  • 二手掉包工程师 (hi-rustin)
  • Brian Bowman (Seeker14491)
  • Jon Gjengset (jonho)
  • pierwill
  • Daniel Silverstone (kinnison)
  • Robert Collins (rbtcollins)
  • Alan Somers (asomers)
  • Brennan Vincent (umanwizard)
  • Joshua Nelson (jyn514)
  • Eric Huss (ehuss)
  • Will Bush (willbush)
  • Thad Guidry (thadguidry)
  • Alexander Lovchin (alovchin91)
  • zoodirector
  • Takayuki Nakata (giraffate)
  • Yusuke Abe (chansuke)
  • Wyatt Carss (wcarss)
  • Sondre Aasemoen (sondr3)
  • facklambda
  • Chad Dougherty (crd477)
  • Noritada Kobayashi (noritada)
  • Milan (mdaverde)
  • Pat Sier (pjsier)
  • Matt Keeter (mkeeter)
  • Alex Macleod (alexendoo)
  • Sathwik Matsa (sathwikmatsa)
  • Kushal Das (kushaldas)
  • Justus Winter (teythoon)
  • k900
  • Nicolas Ambram (nico-abram)
  • Connor Slade (basicprogrammer10)
  • Yerkebulan Tulibergenov (yerke)
  • Caleb Cartwright (calebcartwright)
  • Matthias Beyer (matthiasbeyer)
  • spacemaniac
  • Alex Touchet (atouchet)
  • Guillaume Gomez (guillaumegomez)
  • Chris Denton (chrisdenton)
  • Thomas Orozco (krallin)
  • cui fliter (cuishuang)
  • Martin Nordholts (enselic)
  • Emil Gardström (emilgardis)
  • Arlo Siemsen (arlosi)

The Mozilla Blog — The journey to Roe and after – a Pocket Collection unveils the stories behind Slate’s 7th season of Slow Burn

With the recent overturn of Roe v. Wade, many of us can’t help but wonder: How did we get here? It didn’t happen overnight — no, it was more of a slow burn.

Just in time for the seventh season of Slate’s Slow Burn, host and executive editor Susan Matthews explores the path to Roe — a time when more Republicans than Democrats supported abortion rights. Her exploration leads her to the forgotten story of the first woman to be convicted of manslaughter for having an abortion, the stories of the unlikely Catholic power couple who helped ignite the pro-life movement and a rookie Supreme Court justice who got assigned the opinion of a lifetime.

We chatted with Matthews to learn more about the stories behind the podcast, their importance especially in this moment and what she’s reading when she’s not reporting.

Slate has done some important reporting on Roe v. Wade and its end. As someone who has played a major part in this reporting, can you give a snapshot of what listeners can expect from this new season of Slow Burn?

One of the things that strikes me the most about this moment is that abortion feels like one of the most “stuck” topics I can think of for Americans right now. That reality was part of why I wanted to go back in time as we awaited this decision — it’s really hard to imagine the conversation on abortion being less stuck than it is now. But in the early 1970s, things were actually changing really rapidly. Abortion was being talked about openly and all over the country for the first time, and in Slow Burn: Roe v. Wade we’re telling some of those specific stories — including the story of the first woman convicted of manslaughter for getting an abortion, the Catholic power couple who jump-started the pro-life movement, the story of a women-backed lawsuit in Connecticut that influenced Roe, and then, of course, the story of the decision itself. 

I tried very hard to approach the storytelling from the perspective of realizing that the people involved in those stories had no idea how the abortion debate in America would turn out, and that helped me follow the inherent surprise of each story. So I guess what I would say is listeners can expect to learn a lot, and I hope they can feel like this is an opportunity to engage in this topic that I hope feels a little less weighed down by the politics of the moment (though there are resonances, to be sure!).

Can you tell us how this season of Slow Burn came to life and what makes it unique?

I’ve edited medical stories, jurisprudence stories and personal essays for a long time. So I am very familiar with the essay that makes the case, rooted in personal experience, for why abortion access is important. I agree that abortion access is important! But what I wanted to do with this podcast was dig into the history of abortion in America from a perspective that tried to investigate what happened and how we got to where we are now, rather than using [personal] narratives to make a simple argument about abortion. When I reported each story, I tried to include as much context as possible — and a lot was different then, more on that in a minute! — but also, in every story, I tried to really drill down into who these people were, what was motivating them to act, what worried them, what inspired them, etc. I think the thrill of Slow Burn is that each season kind of shows the listener that each of these deeply important historical moments were really just things that real people had a hand in — real people were driving the action, real people were reacting to things, real people were having feelings about them. So I hope that what makes this show unique is that we’re digging into the lives of those real people, in all of their complications.

Did anything surprise you in the process of putting this podcast together? Was there anything in your research that stood out to you most?

There are two things: The first is that abortion did not used to be partisan in the way it is now, at all. In 1972, more Republicans than Democrats supported abortion access. We dig into why that was, and we also start to answer the question of how that changed. The other thing that really surprised me was how haphazard a lot of the change was. For example, when New York liberalized its abortion law in 1970 and didn’t include a residency requirement, all of the sudden tons of women all over the country started coming to the state for abortions, and that actually had really severe ramifications that I think, when considering the context, also make sense. 

Another example of this is just that no one — not even the justices — knew that Roe v. Wade was going to be the case that determined abortion rights in America when it first came to the court. It’s finding out all of the little things like that that allows listeners to just think about how many different paths we could have taken outside of the timeline we are actually on.

How has the Pocket collection you’ve created for Slow Burn: Roe vs Wade let fans go deeper into these stories?

One of the most difficult stories we reported this season was the story at the center of Episode 2, which is about the Willkes. The Willkes were a Catholic couple who really helped launch the pro-life movement — they’re perfect fodder for a podcast because we have a lot of audio of them talking. But there was so much news coverage about them as a couple in the decades after the story we’re telling in the show, and I relied on so many other stories to understand them, that it was really gratifying to be able to put all of those resources somewhere for other people to see. I was really invested in portraying the Willkes fairly and I think that building out that story in particular with supporting resources helps listeners get a real understanding of who they were and the influence they had on the pro-life movement over subsequent decades.

How does this Pocket collection support the storytelling in your podcast? Is the content in your collection different that what is shared in Slow Burn: Roe vs Wade?

I would say that our story focuses really specifically on the years leading up to Roe — so really, 1970, 1971, and 1972 — but that obviously a ton has happened with abortion and abortion politics in particular since. And the collection helped us give people a window into the rest of that story, which certainly helps explain how we ended up where we are now.

What articles and videos are in your Pocket waiting to be read/watched right now?

I am dying to read this story from The New York Times about Amber Rose.  The other things in my queue are basically stories that I missed or didn’t have time to read when I was making the show and saved for later. Those include Rebecca Traister’s profile of Dianne Feinstein, the Times op-ed about marrying the wrong person that blew up the internet that I have yet to read, and Margaret Talbot’s long read on Amy Coney Barrett (for obvious reasons).

At Pocket, we’re all about helping people carve out time and space to dig into the stories that matter. Where and when do you catch up on the long reads and podcast episodes you’re excited for?

This is quite nerdy, but I’m a member of the Brooklyn Botanical Gardens and so I try to take long walks on the weekends around there with the podcasts I want to catch up on (Prospect Park works for this purpose too!). Commuting time of any kind — subway to work, car rides — are big for podcasts for me too. I also really like to lounge in bed on Sunday mornings and scroll through whatever I missed during the week.

The post The journey to Roe and after – a Pocket Collection unveils the stories behind Slate’s 7th season of Slow Burn appeared first on The Mozilla Blog.

Cameron Kaiser — Network Solutions screws the pooch again

Floodgap appears to be down because Network Solutions' name servers are timing out and name queries are not reliably resolving (everything's up on this end). There is no ETA. If this continues for much longer I'll figure out an alternative, but between this and the social engineering hack last year, maybe this is a sign I need to move it to a different registrar even though I prepaid a few years ago.

Jody Heavener — Brick by brick: why Docusaurus is a powerful documentation framework

At 2022’s AGConf (1Password’s annual employee conference), every employee received a goodie box to celebrate the event and the company’s successes over the past year. Our theme this year was “space”, so the goodie box included a kit for a Lego rocket ship (very appropriate considering our own CEO is a Lego aficionado).

Building the spaceship brought me back to when I was younger and played endlessly with those little bricks.

For me, though, it wasn’t so much about building the specific items in a kit. Sure, I absolutely loved putting together the houses and planes and cars, but what I was most fascinated by was how I could use tiny bricks to expand my creation and build anything I could dream up. The possibilities were endless, my imagination ran wild, and sometimes – usually through through dumb luck – I built something way cooler than what the kit offered in the first place.

Late last year, I started exploring the React-based documentation framework Docusaurus, and spent a good chunk of time going through the documentation. (Surprise! They use their own product!) I got pretty familiar with how it works under the hood, and the ways in which it can be expanded on. It's also got a bustling community, which is unsurprising since it’s entirely open source.

When I joined 1Password earlier this year, where I would be driving the effort to stand up a developer portal for our new developer offerings, I was excited to learn that we’d chosen Docusaurus v2 as the framework to power it all. I’ve had a chance to really dig in since then, learning as much as I could about this powerful little static site generator.

And it occurred to me recently that, with the way they’ve set it up, I’m reminded of those Lego creations: at its core it’s really just a bunch of individual pieces cleverly interlocked to create something far greater. It’s also built on a foundation designed to be entirely extensible.

So I’d like to look at how Docusaurus is put together, and why it’s so great for the 1Password developer portal.

Plugins all the way down

Plugins are the building blocks of features in a Docusaurus 2 site. Each plugin handles its own individual feature.

Docusaurus has handy plugin lifecycle APIs. When you start up the development server or generate a static bundle, each plugin kicks in and traverses through every stage of the lifecycle. With it, you can pull in data across all plugins simultaneously, register routes, validate configuration, and inject HTML tags, among many other things. Docusaurus leverages these same APIs to build up the overall user-facing functionality of the framework through their own collection of plugins.

Consider the primary use case for Docusaurus: documentation. The @docusaurus/plugin-content-docs plugin powers this central feature for the framework. Its more immediate functionality comes from using the loadContent method to look for potentially localized and versioned sets of documentation on the filesystem, and contentLoaded to provide the structured route data for the core to register and produce HTML files. It also extends Docusaurus’ CLI to allow for tagging a new docs version, and even tells the dev server which files to watch, and in turn run the lifecycles again.

The documentation plugin is obviously a huge part of Docusaurus, but they don’t stop there. Everything from the docs, to blogging and individual pages, all the way down to setting up Google Analytics and generating sitemaps are all powered by plugins.

So, why is this important?

If you’ll allow me to borrow my Lego analogy again: Docusaurus’ plugin APIs mean that, while they provide you with a kit you can set up and build something really cool with, they’ve also provided you with the ability to extend the framework in any direction to build something to suit your exact needs (at least as far as static sites go).

Great examples of this can be found on their community plugins page, where others have built plugins for offline/local search (we even use this today), adding SASS styles loading, and consuming OpenAPI specs to generate full API documentation pages. And it couldn’t be easier to roll your own.

Let’s say you wanted to load in some Google Fonts. Here’s what a plugin that does this by using the injectHtmlTags method might look like:

module.exports = function pluginGoogleFonts(context, options) {
  return {
    name: "plugin-google-fonts",

    injectHtmlTags: () => ({
    // Tell the browser we're going to be loading resources from these origins
      headTags: [
        {
          tagName: "link",
          attributes: {
            rel: "preconnect",
            href: "https://fonts.googleapis.com",
          },
        },
        {
          tagName: "link",
          attributes: {
            rel: "preconnect",
            href: "https://fonts.gstatic.com",
            crossorigin: "anonymous",
          },
        },
        // Load the Lobster font
        {
          tagName: "link",
          attributes: {
            rel: "stylesheet",
            href: "https://fonts.googleapis.com/css2?family=Lobster&display=swap",
          },
        },
      ],
    })
  }
};

With this plugin in place, you can now freely use the Lobster font in your CSS. If you wanted to take it a step further and package this plugin up for distribution, you could even allow it to take an array of font names and weights as options to make it truly dynamic.

In the future, as we expand our developer portal, you’re likely to see us build plugins for things like importing and rendering developer blog posts, highlighting integrations built by our developer community, and a whole lot more.

Need to customize it? Swizzle away.

Plugins aren’t limited to just extending functionality, either. They’re what also delivers the look of the framework. Using the getThemePath method your plugin can tell Docusaurus where to find the React components that make up a theme, selectively overriding components from an existing theme or building your own theme from the ground up.

One of the neatest features of Docusaurus is the ability to Swizzle a component.

[Swizzling] comes from Objective-C and Swift-UI: method swizzling is the process of changing the implementation of an existing selector (method). For Docusaurus, component swizzling means providing an alternative component that takes precedence over the component provided by the theme.

What does this mean in practice? Well, our developer portal currently uses the default Classic theme, but if you check out our footer you’ll notice that it looks nothing like the footer in that theme. We wanted ours to share a consistent look with the one on 1Password.com, so we swizzled the existing Footer component by running the following command:

npm run swizzle @docusaurus/theme-classic Footer -- --eject

This cloned the component out of the Docusaurus package and into our workspace. Now we've got full agency over the look and feel of the site’s footer, while still being able to rely on the rest of the theme’s components, which also includes future updates. We’re going to be swizzling a fair bit this year as the developer portal evolves.

The default Footer component versus our swizzled Footer component to match the 1Password.com style.

The framework ships with the Classic theme, and out of the box it does a fantastic job. As of April 2022 the theme selection is fairly limited for v2 of Docusaurus, with only the Classic theme and some extensions to it available. More are coming, though. One that I’m particularly looking forward to, a Tailwind-powered theme, is also a great example of why I appreciate that they’re an open source project: it started as a community request, grew in popularity, and over time evolved into part of the roadmap.

Markup or Markdown - how about both?

As with every static site generator, it’s expected that Docusaurus would support Markdown - and they took it a step further, using MDX to parse content. MDX allows you to write JSX (React components) alongside your Markdown, allowing seamless native integration with the rest of the React app, which eventually gets all compiled down to HTML. This concept of static site generators interlacing Markdown with another syntax to extend the capabilities of its documentation is not new, but what gets me excited is the power that JSX affords us. You’re not limited to static HTML or shortcodes. Instead you get the full power of JSX components, meaning it’s possible to build fully styled, rich components that you can embed right in your content.

MDX also supports Remark and Rehype plugins, allowing you to augment the syntax and replace content on the fly. What can we do with this? Docusaurus demonstrates this well by creating its own plugins for admonitions, table of contents generation, and creating heading links.

There’s already a huge collection of plugins available for both Remark and Rehype, but if you need something a little more tailored to your specific use case creating these types of plugins is really straightforward, too. Consider this 13-liner that defaults Markdown code blocks to using Shell highlighting:

const visit = require("unist-util-visit");

module.exports = function pluginRemarkShellCode(context, options) {
  return (tree) => {
    visit(tree, (node) => {
      // If the node is a code block, but the language is not set
      if (node.type === "code" && !node.lang) {
        // Set it to Shell
        node.lang = "shell";
      }
    });
  };
};

Using unist-util-visit we can iterate across all nodes and their children to selectively modify the properties or contents of any node that matches our criteria. Now our Markdown files only need to specify language for those code blocks that aren't using Shell.

Fully Open Source

I’ve been heads down in Docusaurus for quite some time now, and it’s proven to be incredibly robust. But beyond the framework itself, I’ve also really appreciated the community behind it. From contributing my own PRs to the core, to getting help from team members themselves and other eager developers in their Discord server, it’s been a pleasure creating with this extraordinary tool.

Go check out the 1Password developer portal, built with Docusaurus. I’m looking forward to showing off the cool things we’ve got planned for it down the road as we use these building blocks to create something really, really cool.

Mozilla Privacy Blog — Mozilla statement as EU Parliament adopts new pro-competition rulebook for Big Tech

The EU Parliament today adopted the ‘Digital Markets Act’, new rules that will empower consumers to easily choose and enjoy independent web browsers. We welcome the new pro-competition approach and call for a comprehensive designation of the full range of Big Tech gatekeepers to ensure the legislation contributes to a fairer and more competitive European digital market.

The DMA will grant consumers more freedom to choose what software they wish to use, while creating the conditions for independent developers to compete fairly with Big Tech. In particular, we see immense benefit in empowering consumer choice through prohibitions that tackle manipulative software designs and introduce safeguards that allow consumers to simply and easily try new apps, delete unwanted apps, switch between apps, change app defaults, and to expect similar functionality and use.

“The browser is a unique piece of software that represents people online. It is a tool that allows individuals to exercise choices about what they do and what happens to them as the navigate across the web. But like other independent web browsers, Mozilla has been harmed by unfair business practices that take away consumer choice, for instance when gatekeepers make it effectively impossible for consumers to enable and keep Firefox as their default web browser, or when they erect artificial operating system barriers that mean we can’t even offer consumers the choice of a privacy- and security-first browser. Ensuring that gatekeepers allocate enough resources to fully and transparently comply with the DMA is the first step towards a more open web and increased competition, allowing users to easily install and keep Firefox as their preferred browser on both desktop and mobile” – added Owen Bennett, Mozilla Senior Policy Manager.

To make the DMA’s promise a reality, swift and effective enforcement of the new law is required. It’s essential that all gatekeepers – and their core platform services – are identified and designated as soon as possible. This is the first test for the European Commission’s enforcement approach, and regulators must set down a marker that Europe means business.

We look forward to contributing to remedies that can ensure independent browsers can compete and offer consumers meaningful choices.

The post Mozilla statement as EU Parliament adopts new pro-competition rulebook for Big Tech appeared first on Open Policy & Advocacy.

IRL (podcast) — Introducing IRL Season 6: AI in Real Life

AI is everywhere now. It’s part of healthcare, social media, maps, and even killer robots. But who has power over AI? And who is shifting that power? Join IRL’s new host, Bridget Todd, as she talks to technology builders and policy folks from around the world who are developing more trustworthy AI that puts people over profits.

IRL is an original podcast from the non-profit Mozilla. For more on our series, visit us here.

The Talospace Project — Firefox 102 on POWER

Firefox 102 is out, not an earth shattering release but with some welcome privacy improvements. It builds out of the box on this Talos II using the PGO-LTO patch from Firefox 101 and the .mozconfigs from Firefox 95.

Firefox 102 is also the basis for the next Extended Support Release, with support for 91ESR (the current version) due to end on September 20 with the release of Firefox 105. Due to a family emergency, I've been out of the country for a bit and haven't been doing much with any projects, let alone the POWER9 JIT (this is why we need more people working on it!). Now that I've been back for a few days and more or less in the swing of things again, it's time to dust it off and forward port the current version to 102 so people doing ESR JIT builds for Fx91 can continue to do so with Fx102. I'll make an announcement and post a patch set when it's ready.

The Mozilla Blog — Killian Wells, CEO of Fragrance House Xyrena, Shares The Joy He Finds In His Corner Of The Internet

Here at Mozilla, we are the first to admit the internet isn’t perfect, but we are also quick to point out that the internet is pretty darn magical. The internet opens up doors and opportunities, allows for people to connect with others, and lets everyone find where they belong — their corners of the internet. We all have an internet story worth sharing. In My Corner of the Internet, we talk with people about the online spaces they can’t get enough of, what we should save in Pocket to read later and what sites and forums shaped them.

With Pride celebrations taking place throughout June, we’re featuring LGBTQ+ leaders this month as part of our My Corner of the Internet series. In the last installment, Killian Wells, CEO of the award-winning fragrance house Xyrena, talks about his love of nerd culture, launching a beauty brand website and Pocketing his favorite blogs. 

What is your favorite corner of the internet?

I’m on Instagram a lot and love throwback accounts like @insta80s90s, @80_deco, @popculturedmemes, and @onlyninetieskidsknow.

What is an internet deep-dive that you can’t wait to jump back into?

A few times a year I fall down an internet rabbit hole of searching for synths/patches used on popular songs, particularly by my favorite producers, like Max Martin and Timbaland. 

What is one tab on your browser you always regret closing?

As a film buff, I’m on IMDb a few times a day, especially when I’m watching a movie or show and need to know where else I’ve seen an actor that looks familiar. I’m on it so often that I’ve developed the very useless talent of being able to name the year a movie was released and its distributor.

Who is an LGBTQ+ person with a very online presence that is a role model for you?

I’m a huge fan of Jeremy Scott. I love his design aesthetic so much! I also saw his documentary a few years ago and really relate to his story in many ways. 

What can you not stop talking about on the internet right now?

We just launched a brand-new site for Xyrena (xyrena.com), so I’m really excited about that. I’m pretty obsessed with the retro design.

What was the first online community that you engaged with?

I’m a big pop culture nerd and collect Funko Pops (the Ad Icons are my favorite) so as a teenager I connected with other funatics. My first business was actually selling Wacky Wobblers online and I got the chance to meet Funko’s founder, Mike Becker, and tour one of their original headquarters. 

What articles and videos are in your Pocket waiting to be read/watched right now?

I keep my favorite blogs in my Pocket, like lunchmeatvhs.com and alternativemovieposters.com

If you could create your own corner of the internet, what would it look like?

It already exists at xyrena.com 

Killian Wells is an Austin-based pop music artist/songwriter/producer turned perfumer and CEO of the award-winning fragrance house Xyrena. Dubbed ‘the bad boy perfumer’ and the ‘Damien Hirst of the perfume world’ by press and critics, Wells’ work is heavily influenced by pop culture from the 80s, 90s, and Y2K. Follow him on Instagram @KillianWells

Save and discover the best articles, stories and videos on the web

Get Pocket

The post Killian Wells, CEO of Fragrance House Xyrena, Shares The Joy He Finds In His Corner Of The Internet appeared first on The Mozilla Blog.

Mozilla Performance Blog — Introducing A New Performance Comparison Tool – PerfCompare

Introduction

I have been a member of the performance test team for the past two and a half years. During that time, the performance team has had many discussions about improving the developer experience when running performance tests. 

The most significant pain points for developers are that:

1) tests are difficult to run, or they don’t know which tests to run, and

2) our tools for comparing these results are complex or confusing.

One of the tools we currently use is called Perfherder. Perfherder does many things, such as alerting on performance improvements and regressions, comparing results of performance tests, and graphing those results.

perfherder alerts view

Perfherder Alerts View

Our performance sheriffs use Perfherder Alerts View to investigate possible improvements or regressions. They first determine if the alert is valid and then identify the culprit revision that caused a regression or improvement.

The sheriff then files a bug in Bugzilla. If it is a regression, a :needinfo flag is added for the author of that commit.

performance regression bug

Example of a performance regression bug

The user workflows for the Compare View and Graphs View are usually one of two scenarios:

Scenario A: An engineer is tagged in a regression (or improvement) by a performance sheriff, as described above. The engineer then needs to determine why the patch caused a regression, which can take many iterations of altering code, running tests, and comparing the results.

or 

Scenario B: A firefox engineer working on a patch will run performance tests to ensure the latest changes have not caused performance to regress. Once tests are complete, the engineer compares the results before and after those changes were made.

The proactive approach, Scenario B, is highly preferable to a reactive one, as in Scenario A.

A primary goal of the performance test team is to simplify and improve the experience of running performance tests and comparing the results.

If we succeed in this, I believe we can improve the performance of Firefox as a whole.

By providing better tools and workflows, we will empower our engineers to include performance testing as a part of their standard development process.

Perfherder User Experience

Perfherder search for revision

Perfherder Compare View – Search for Revision

perfherder compare results

Perfherder Compare View – Comparison Results

We have conducted usability testing and user research to identify developer pain points for Perfherder and attempted to address some of these issues.

However, sometimes these changes created an even more cluttered UI. Some of these changes included adding a User Guide or more documentation to explain how to use the tools, which is a problematic approach.

If we need to write more documentation to explain how to use a tool that should be reasonably straightforward, it suggests that there are fundamental weaknesses in our UX design.

Perfherder was initially developed for a particular use case and has been expanded to include more features and test harnesses over the years.

While it has served its purpose, the time has come to develop a new comparison tool to support the wide range of features that we have added to Perfherder–and some new ones too!

Introducing PerfCompare

PerfCompare performance comparison tool

PerfCompare is a performance comparison tool that aims to replace Perfherder Compare View. This is a task I have wanted to undertake for quite some time, but we had a smaller team and lacked the resources to justify a project of this size.

Although I am an engineer, I am also an artist. I studied art for years, attending a residential arts school my junior year, and (very briefly) pursuing a double major in Computer Engineering and Art.

As a career, however, I knew that software engineering was what I wanted to pursue, while art remained an enjoyable pastime. 

As such, I am very opinionated on matters of aesthetics and design. My goal is to create a tool that is easy to use–but also enjoyable to use!

I want to create something to delight engineers and make them want to run performance tests.

I approached this project with a blank page–literally! When creating initial designs, I sat in front of a white sheet of paper with a black pen. In my mind, I asked, “What does the user want to do?”

I broke this down into the most simple actions, step by step. I wanted to remove anything unnecessary or confusing, and create a clean and aesthetic UI.

Since then, three other engineers have joined the project. Together we have been working to create an MVP to announce and demo at the All Hands in September.

We aim to have a product ready for initial user testing and enroll it in Mozilla’s foxfooding program.

While some of these are still under development, below are some highlights of improvements and features we have implemented so far.

Highlights

  • List of repositories to select from includes only the most common, rather than an exhaustive list of all repositories.
  • Users can search by short hash, long hash, or author email, instead of long hash only.
perfcompare search by email

Search for revision by author email

  • When searching for revisions, the results include not just a revision hash and author email, but also the commit message and timestamp, making it easier to differentiate between them.
  • Users can select up to four revisions to compare instead of just two.
perfcompare selected revisions

Select four revisions to compare

  • We are using icons to represent platform and confidence, with a tooltip that displays more information.
  • Easier-to-read test names.
  • We have removed magnitude of change and repeated header rows for a cleaner UI.
  • Users can edit selected revisions directly from the Results View instead of navigating back to Search View.
perfcompare results view

Example of results view using mock data, showing the platform tooltip

While we still have much to do before PerfCompare is ready for user testing, we have made significant progress and look forward to demonstrating our tool in Hawaii!

If you have any feedback or suggestions, you can find us in the #PerfCompare matrix channel, or join our #PerfCompare User Research channel!

Thank you to everyone who has contributed to our user research so far!

Resources

The Rust Programming Language Blog — RLS Deprecation

The Rust Language Server (RLS) is being deprecated in favor of rust-analyzer. Current users of RLS should migrate to using rust-analyzer instead. Builds of RLS will continue to be released until at least the Rust 1.64 release (2022-09-22), after which no new releases will be made. This timeline may change if any issues arise.

RLS is an implementation of the Language Server Protocol (LSP) which provides enhanced features with any editor that supports the protocol, such as code-checking and refactoring. RLS was introduced by RFC 1317 and development was very active from 2016 through 2019. However, the architecture of RLS has several limitations that can make it difficult to provide low-latency and high-quality responses needed for an interactive environment.

Development of rust-analyzer began near the beginning of 2018 to provide an alternate LSP implementation for Rust. rust-analyzer uses a fundamentally different approach that does not rely on using rustc. In RFC 2912 rust-analyzer was adopted as the official replacement for RLS.

How you migrate to rust-analyzer will depend on which editor you are using. If you are using VSCode, you should uninstall the rust-lang.rust extension and install the official rust-lang.rust-analyzer extension. For other editors, please consult the rust-analyzer manual for instructions on how to install it.

Should you have any issues migrating to rust-analyzer, the Editors and IDEs category on the Rust Users forum is available for help with installation and usage.

We will soon be marking the official rust-lang.rust VSCode extension as deprecated, and will be implementing notifications that will inform users about the transition. After the end of release builds of RLS, we plan to replace the rls executable in official Rust releases with a small LSP implementation that informs the user that RLS is no longer available.

We would like to thank everyone who has worked on RLS and rust-analyzer. These options would not exist without the tremendous effort of all the contributors to these projects.

Spidermonkey Development Blog — SpiderMonkey Newsletter (Firefox 102-103)

SpiderMonkey is the JavaScript engine used in Mozilla Firefox. This newsletter gives an overview of the JavaScript and WebAssembly work we’ve done as part of the Firefox 102 and 103 Nightly release cycles.

👷🏽‍♀️ New features

⚙️ Modernizing JS modules

We’re working on improving our implementation of modules. This includes supporting modules in Workers, adding support for Import Maps, and ESMification (replacing the JSM module system for Firefox internal JS code with standard ECMAScript modules).

⏱️ Profiler support

We’ve collaborated with the performance team to improve support for external profilers such as perf on Linux:

  • The performance team has added support for perf’s jitdump format. This makes it possible to see JS functions in perf profiles. It also lets us annotate assembly code with LIR and CacheIR instruction names.
  • We made changes to the JITs to preserve and use frame pointers for all JIT frames. Profilers such as perf (and other stack unwinders) are now able to reliably unwind through JIT frames by following frame pointers.
  • We’ve simplified and optimized code in the JITs by taking advantage of frame pointers.
  • We fixed an issue with the C++ interpreter’s profiler instrumentation for resumed async functions and generators. This could result in missing frames in the Firefox profiler.

🚀 JS Performance

  • We’ve changed the bytecode we generate for try-finally to support functions with finally blocks in the optimizing JIT. This fixes an old performance cliff.
  • We’ve optimized the code we generate for test expressions.
  • More typed array builtins that use a callback function are now marked as inlinable.
  • We’ve optimized arguments-object allocation for inlined functions.
  • We’ve implemented a new bytecode instruction for closing iterators, to reduce bytecode size for for-of loops.
  • We’ve landed more optimizations for concurrent delazification (disabled by default).

🏎️ WebAssembly Performance

  • We’ve re-enabled code caching with a new serialization system.
  • We’ve landed more optimizations for SIMD instructions.
  • We’ve replaced some uses of splay trees with AVL trees to improve compilation time.
  • We’ve reduced the offset guard memory reservation from 2 GB to 32 MB. This shrinks the amount of virtual memory we reserve for Wasm code by 33%.

📚 Miscellaneous

  • We’ve added a checklist for implementing new JS language features.
  • We’ve imported the latest version of Test262.
  • We’ve improved tracking and assertions for the GC retained heap size.
  • We’ve migrated our string-to-double code to use the modern double-conversion library instead of our old dtoa.c fork.
  • We’ve implemented support for Rooted<Result<V,E>>.
  • We’ve added a command-line argument to the JS shell for setting GC parameters.
  • We’ve started to remove typedefs for various GC types. For example, we now use Rooted<Shape*> instead of the old RootedShape typedef.
  • We’ve improved telemetry for full GCs to be more useful.

Firefox Nightly — These Weeks In Firefox: Issue 119

Highlights

  • Allow the browser toolbox to enable/disable multiprocess behavior on-demand (bug). The user can switch between two modes.
    • Parent process only (fast) – observing only the parent process and useful for debugging Firefox UI only
    • Multiprocess (slower) – observing all chrome as well as content processes

  • The Firefox Profiler can now sort roots in alphabetical order with a flamgraph. This is especially useful when comparing profiles with the JavaScript filter, as this ensures that the order is consistent.

  • Add tracks for power usage (this works on MacOS M1 and Windows 11 especially)

Follow along bug 1774844 to know when the feature will land to Firefox.

  • Subtitles setting panel is now in Firefox 103. Make sure `media.videocontrols.picture-in-picture.display-text-tracks.enabled` is set to true in about:config and subtitles are enabled on the original video.

 

Friends of the Firefox team

Resolved bugs (excluding employees)

Volunteers that fixed more than one bug
  • Itiel
  • Mike Ratcliffe [:miker] [:mratcliffe] [:mikeratcliffe]
  • Rob Lemley [:rjl]
  • Zoltán Szatmáry
New contributors (🌟 = first patch)

Project Updates

Add-ons / Web Extensions

(read-only due to overlapping with our team’s remote workweek meeting)

 

Addon Manager & about:addons

  • InstallTrigger Deprecation: Bug 1772905 landed in Nightly 103 as mentioned two weeks ago: the InstallTrigger global is now fully hidden on nightly and early beta.
    • NOTE: Please link to Bug 1774005 bug reports of website breaking due to websites using InstallTrigger for UserAgent detection (e.g. see Bug 1773873) if you notice newly filed ones during “Firefox :: General” triaging

WebExtensions Framework

  •  As part of the ongoing ManifestVersion 3 (MV3) work:
    • Added to the browserAction (the toolbar extension button) context menu new entries to allow the user to grant/revoke permissions for the current website (currently restricted to MV3 extensions) – Bug 1769793
    • For MV3 extensions, all host permissions are considered optional and not granted (until the user has explicitly granted some to the extension). When the user activates the browserAction button (e.g. by clicking on it), the extension content scripts will be temporarily allowed to be injected in all same-origin frames of the current tab top-level origin if the top-level origin matches at least one of the host permissions listed in the manifest – Bug 1774641

Developer Tools

Browser Toolbox
  • Performance improvements for Browser Toolbox Debugger
    • Source tree in the Debugger panel (bug)
    • Code editor CodeMirror (bug)
    • Reduce number of state updates (bug, bug, bug)

 

  • Performance improvement for Browser Toolbox Console
    • Avoiding cloning Map() objects (bug)

 

  • Mike Ratcliffe is making lots of contribution to DevTools these days and fixed an issue where we would limit the number of indexedDB items you could see in the storage panel (bug)
  • Thanks to arai we now display getter/setter in arrays (bug)

 

WebDriver BiDi

 

Performance Tools (aka Firefox Profiler)

  • Huge documentation improvement. (along with the removal of the documentation of the old panel)
  • The tooltip for markers is now showing the originating thread when we’re in the view for merged tracks.
    Reminder: you can merge tracks by ctrl clicking their names in the timeline (the top part).

  • Local threads are alphabetically sorted at load time, which is nice for threadpools especially, when you’re looking for a specific track in the list.
  • We now have smaller URLs by default, which should avoid the problem where a permalink couldn’t be created with profiles that have a lot of tracks
  • New locale: frisian (fy-NL) (locale with more than 400k speakers, originating in netherlands).
  • As an aside Gerald worked on reducing the timer thread wake-ups to reduce the power usage for this thread.

Privacy/Security

  • Eval has been disallowed in the parent process and system privilege context for some time, but as part of Pwn2Own followup investigation we discovered it was not properly being blocked in .jsm files. After the merge we will be landing this enforcement – if you encounter breakage please cc Tom Ritter on the bug. You can work around it by disabling the prefs.

 

Search and Navigation

 

ESMification status

  • Most of the preparation work has been completed.
  • Documentation and scripts for semi-automatic migration are undergoing testing.
  • Hoping to open up to teams next week.

Mozilla Thunderbird — Are Your Favorite Thunderbird Add-ons Compatible With Thunderbird 102?

Thunderbird 102 is here! Our annual release is full of highly-requested features, beautiful new visuals, and quality-of-life upgrades. We’re also working hard to ensure that your favorite Thunderbird add-ons are compatible. In fact, we expect the majority of add-ons to be updated within a few weeks of Thunderbird 102’s release.

We understand that certain add-ons are invaluable to your workflow. So, to help you decide the best time to update your version of Thunderbird, here’s a simple way to discover if the extensions you currently have installed are compatible with Thunderbird 102.

Install “Addon Compatibility Check For TB 102”

How do you check the compatibility of your add-ons? By installing an add-on! While it doesn’t have the most creative name, “Addon Compatibility Check For TB 102” does its job considerably well.

(This extension works with Thunderbird 68 and newer.)

Addon Compatibility Check For TB 102 icon in Thunderbird 102<figcaption>Click the icon that looks like a puzzle to instantly check add-on compatibility</figcaption>

Once installed, just click the icon that looks like a puzzle piece in your Mail Toolbar. It will instantly check if your installed Thunderbird add-ons will function properly with Thunderbird 102. (The data updates once every 24 hours). It also suggests any known alternatives for extensions that might have been abandoned and no longer receive updates.

Here’s my own result, above. The only “unknown” is uBlock Origin, but that’s because I downloaded it from Github, and not from the main Thunderbird extensions website. (Good news: it does work with 102!)

The Thunderbird Add-on developer community is making a concentrated effort to ensure that all of your favorite add-ons receive updates for Thunderbird 102. We hope this add-on helps you decide when upgrading from Thunderbird 91 is right for you.

The post Are Your Favorite Thunderbird Add-ons Compatible With Thunderbird 102? appeared first on The Thunderbird Blog.

The Rust Programming Language Blog — Announcing Rust 1.62.0

The Rust team is happy to announce a new version of Rust, 1.62.0. Rust is a programming language empowering everyone to build reliable and efficient software.

If you have a previous version of Rust installed via rustup, you can get 1.62.0 with:

rustup update stable

If you don't have it already, you can get rustup from the appropriate page on our website, and check out the detailed release notes for 1.62.0 on GitHub.

If you'd like to help us out by testing future releases, you might consider updating locally to use the beta channel (rustup default beta) or the nightly channel (rustup default nightly). Please report any bugs you might come across!

What's in 1.62.0 stable

cargo add

You can now add new dependencies directly from the command line using cargo add. This command supports specifying features and versions. It can also be used to modify existing dependencies.

For example:

cargo add log
cargo add serde --features derive
cargo add nom@5

See the cargo documentation for more.

#[default] enum variants

You can now use #[derive(Default)] on enums if you specify a default variant. For example, until now you would have to manually write a Default impl for this enum:

#[derive(Default)]
enum Maybe<T> {
    #[default]
    Nothing,

    Something(T),
}

As of now only "unit" variants (variants that have no fields) are allowed to be marked #[default]. More information is available in the RFC for this feature.

Thinner, faster mutexes on Linux

Previously, Mutex, Condvar, and RwLock were backed by the pthreads library on Linux. The pthreads locks support more features than the Rust APIs themselves do, including runtime configuration, and are designed to be used in languages with fewer static guarantees than Rust provides.

The mutex implementation, for example, is 40 bytes and cannot be moved. This forced the standard library to allocate a Box behind the scenes for each new mutex for platforms that use pthreads.

Rust's standard library now ships with a raw futex-based implementation of these locks on Linux, which is very lightweight and doesn't require extra allocation. In 1.62.0 Mutex only needs 5 bytes for its internal state on Linux, though this may change in future versions.

This is part of a long effort to improve the efficiency of Rust's lock types, which includes previous improvements on Windows such as unboxing its primitives. You can read more about that effort in the tracking issue.

Bare metal x86_64 target

It's now easier to build OS-less binaries for x86_64, for example when writing a kernel. The x86_64-unknown-none target has been promoted to Tier 2 and can be installed with rustup.

rustup target add x86_64-unknown-none
rustc --target x86_64-unknown-none my_no_std_program.rs

You can read more about development using no_std in the Embedded Rust book.

Stabilized APIs

The following methods and trait implementations are now stabilized:

Other changes

There are other changes in the Rust 1.62.0 release. Check out what changed in Rust, Cargo, and Clippy.

Contributors to 1.62.0

Many people came together to create Rust 1.62.0. We couldn't have done it without all of you. Thanks!

Data@Mozilla — This Week in Glean: Reviewing a Book – Rust in Action

(“This Week in Glean” is a series of blog posts that the Glean Team at Mozilla is using to try to communicate better about our work. They could be release notes, documentation, hopes, dreams, or whatever: so long as it is inspired by Glean. You can find an index of all TWiG posts online.)

This blog post is going to be a bit different from what you may have read from me in the past. Usually I write about things I am working on or things I have encountered while working that I find interesting. This is still a post about something I find interesting, but instead of directly related to the things I’ve been working on, it’s about something that Mozilla actively encourages me to do: furthering my knowledge and professional development. In this instance, I chose to read a book on Rust to try and increase my knowledge and fill in any gaps in my understanding of a programming language I use almost every day and have come to really enjoy working with. The book in question is Rust in Action by Tim McNamara.

The first thing I would like to call out is the great organization of the material in the book. The first few chapters go over a lot of basic material that was perfect for a beginner to Rust, but which I felt that I was already reasonably familiar with. So, I was able to skim over a few chapters and land at just the right point where I felt comfortable with my knowledge and start reading up on the things I was ready to learn more about. This happened to be right around the end of Part 1 with the bits about lifetimes and borrowing. I have been using Rust long enough to understand a lot of how this works, but learning some of the general strategies to help deal with ownership issues was helpful, especially thinking about wrapping data in types designed to aid in movement issues.

Being an especially data oriented person, I took extra enjoyment out of the “Data in depth” chapter. Having done quite a bit of embedded software development in the past, the explanation of endianness brought back some memories of some not-so-portable code that would break because of this. The rest of the chapter was filled with bit by bit explanations of types and how they represented data in memory, op-codes, and other intricacies of thinking about data at a very low level. Definitely one of my favorite chapters in the book!

I found that the other chapters that stood out to me did so because they explained a topic such as Networking (chapter 8) or Time (chapter 9) in the context of Rust. These were things I had worked with in the past with other languages and that recognition allowed the approaches that were being explained in Rust to really sink in.  Examples of patterns like sending raw TCP data and formatting of timestamps were interrupted with concepts like traits and ways to improve error handling at just the right times to explain Rust approaches to them.

I would definitely recommend this book to anyone who is interested in learning more about Rust, especially how that applies to “systems” programming. I feel like I have come away from this with a better understanding of a few things that were still a little fuzzy and I learned quite a bit about threading in Rust that I really hadn’t been exposed to before. We continue to rely on Rust to write our services in a cross-platform way and there is a lot of information and techniques in this book that I can directly apply to the real-world problems I face working on data and experimentation tools here at Mozilla.

All in all, a really enjoyable book with fun examples to work through. Thanks to the author, and to Mozilla for encouraging me to continually improve myself.

Hacks.Mozilla.Org — Neural Machine Translation Engine for Firefox Translations add-on

Firefox Translations is a website translation add-on that provides an automated translation of web content. Unlike cloud-based alternatives, translation is done locally on the client-side in the user’s computer so that the text being translated does not leave your machine, making it entirely private. The add-on is available for installation on Firefox Nightly, Beta and in General Release.

The add-on utilizes proceedings of project Bergamot which is a collaboration between Mozilla, the University of Edinburgh, Charles University in Prague, the University of Sheffield, and the University of Tartu with funding from the 🇪🇺 European Union’s Horizon 2020 research and innovation programme.

The add-on is powered internally by Bergamot Translator, a Neural Machine Translation engine that performs the actual task of translation. This engine can also be utilized in different contexts, like in this demo website, which lets the user perform free-form translations without using the cloud.

In this article, we will discuss the technical challenges around the development of the translation engine and how we solved them to build a usable Firefox Translations add-on.

Challenges

The translation engine is built on top of marian framework, which is a free Neural Machine Translation framework written in pure C++. The framework has a standalone native application that can provide simple translation functionality. However, two novel features needed to be introduced to the add-on that was not present in the existing native application.

The first was translation of forms, allowing users to input text in their own language and dynamically translate it on-the-fly into the page’s language. The second was estimating the quality of the translations so that low-confidence translations could be automatically highlighted in the page, in order to notify the user of potential errors. This led to the development of the translation engine which is a high level C++ API layer on top of marian.

The resulting translation engine is compiled directly to native code. There were three potential architectural solutions to integrating it into the add-on:

  1. Native integration to Firefox: Bundling the entire translation engine native code into Firefox.
  2. Native messaging: Deploying the translation engine as a native application on the user’s computer and allowing the add-on to exchange messages with it.
  3. Wasm: Porting the translation engine to Wasm and integrating it to the add-on using the developed JS bindings.

We evaluated these solutions on the following factors which we believed were crucial to develop a production ready translation add-on:

  1. Security: The approach of native integration inside the Firefox Web Browser was discarded following Mozilla’s internal security review of the engine code base, which highlighted issues over the number of third-party dependencies of the marian framework.
  2. Scalability and Maintainability: Native messaging would have posed challenges around distributing the code for the project because of the overhead of providing builds compatible with all platforms supported by Firefox. This would have been impractical to scale and maintain.
  3. Platform Support: The underlying marian framework of the translation engine supports translation only on x86/x86_64 architecture based processors. Given the increasing availability of ARM based consumer devices, the native messaging approach would have restricted the reach of the private and local translation technology to a wider audience.
  4. Performance: Wasm runs slower compared to the native code. However, it has potential to execute at near native speed by taking advantage of common hardware capabilities available on a wide range of platforms.

Wasm design as a portable compilation target for programming languages means developing and distributing a single binary running on all platforms. Additionally, Wasm is memory-safe and runs in a sandboxed execution environment, making it secure when parsing and processing Web content. All these advantages coupled with its potential to execute at near native speed gave us motivation to prototype this architectural solution and evaluate whether it meets the performance requirement of the translation add-on.

Prototyping: Porting to Wasm

We chose the Emscripten toolchain for compiling the translation engine to Wasm. The engine didn’t compile to Wasm out of the box and we made few changes to successfully compile and perform translation using the generated Wasm binary, some of which are as follows:

Prototyping to integration

Problems

After having a working translation Wasm binary, we identified a few key problems that needed to be solved to convert the prototype to a usable product.

Scalability

Packaging of all the files for each supported language pair in the Wasm binary meant it was impractical to scale for new language pairs. All the files of each language pair (translating from one language to another and vice versa) in compressed form amount to ~40MB of disk space. As an example, supporting translation of 6 language pairs made the size of the binary ~250 MB.

Demand-based language support

The packaging of files for each supported language pair in the Wasm binary meant that the users will be forced to download all supported language pairs even if they intended to use only a few of them. This is highly inefficient compared to downloading files for language pairs based on the user’s demand.

Performance

We benchmarked the translation engine on three main metrics which we believed were critical from a usability perspective.

  1. Startup time: The time it takes for the engine to be ready for translation. The engine loads models, vocabularies, and optionally a shortlist file contents during this step.
  2. Translation speed: The time taken by the engine to translate a given text after its successful startup, measured in the number of words translated per second aka wps.
  3. Wasm binary size: The disk space of the generated Wasm binary.

The size of the generated Wasm binary, owing to the packaging, became dependent on the number of language pairs supported. The translation engine took an unusually long time (~8 seconds) to startup and was extremely slow in performing translation making it unusable.

As an example, translation from English to German language using corresponding trained models gave only 95 wps on a MacBook Pro (15-inch, 2017), MacOS version 11.6.2, 3.1 GHz Quad-Core Intel Core i7 processor, 16 GB 2133 MHz RAM.

Solution

Scalability, demand-based language support and binary size

As packaging of the files affected the usability of the translation engine on multiple fronts, we decided to solve that problem first. We introduced a new API in the translation engine to pass required files as byte buffers from outside instead of packing them during compile time in Emscripten’s virtual file system.

This allowed the translation engine to scale for new languages without increasing the size of the Wasm binary and enabled the add-on to dynamically download files of only those language pairs that the users were interested in. The final size of the Wasm binary (~6.5 MB) was well within the limits of the corresponding metric.

Startup time optimization

The new API that we developed to solve the packaging problem, coupled with few other optimizations in the marian framework, solved the long startup time problem. Engine’s startup time reduced substantially (~1 second) which was well within the acceptable limits of this performance criteria.

Translation speed optimization

Profiling the translation step in the browser indicated that the General matrix multiply (GEMM) instruction for 8-bit integer operands was the most computational intensive operation, and the exception handling code had a high overhead on translation speed. We focused our efforts to optimize both of them.

  1. Optimizing exception handling code: We replaced try/catch with if/else based implementation in a function that was frequently called during the translation step which resulted in ~20% boost in translation speed.
  2. Optimizing GEMM operation: Deeper investigation on profiling results revealed that the absence of GEMM instruction in Wasm standard was the reason for it to perform so poorly on Wasm.
    1. Experimental GEMM instructions: Purely for performance evaluation of GEMM instruction without getting it standardized in Wasm, we landed two experimental instructions in Firefox Nightly and Release for x86/x86_64 architecture. These instructions improved the translation speed by ~310% and the translation of webpages seemed fast enough for the feature to be usable on these architectures. This feature was protected behind a flag and was exposed only to privileged extensions in Firefox Release owing to its experimental nature. We still wanted to figure out a standard based solution before this could be released as production software but it allowed us to continue developing the extension while we worked with the Firefox WASM team on a better long-term solution.
    2. Non-standard long term solution: In the absence of a concrete timeline regarding the implementation of GEMM instruction in the Wasm standard, we replaced the experimental GEMM instructions with a Firefox specific non-standard long term solution which provided the same or more translation speeds as provided by the experimental GEMM instructions. Apart from privileged extensions, this solution enabled translation functionallity for non-privileged extensions as well as regular content with same translation speeds and enabled translation on ARM64 based platforms, albeit with low speeds. None of this was possible with experimental GEMM instructions.
    3. Native GEMM intrinsics: In an effort to improve translation speeds further, we landed a native GEMM implementation in Firefox Nightly protected behind a flag and exposed as intrinsics. The translation engine would directly call these intrinsics during the translation step whenever it is running in Firefox Nightly on x86/x86_64 architecture based systems. This work increased the translation speeds by 25% and 43% for SSSE3 and AVX2 simd extensions respectively compared to the experimental instructions that we had landed earlier.
  3. Emscripten toolchain upgrade: The most recent effort of updating the Emscripten toolchain to the latest version increased the translation speeds for all platforms by ~15% on Firefox and reduced the size of the Wasm binary further by ~25% (final size ~4.94 MB).

Eventually, we achieved the translation speeds of ~870 wps for translation from English to German language using corresponding trained models on Firefox Release on a MacBook Pro (15-inch, 2017), MacOS version 11.6.2, 3.1 GHz Quad-Core Intel Core i7 processor, 16 GB 2133 MHz RAM.

Future

The translation engine is optimized to run at high translation speeds only for x86/x86_64 processors and we have ideas for improving the situation on ARM. A standardized Wasm GEMM instruction can achieve similar speeds on ARM, providing benefits to emerging class of consumer laptops and mobile devices. We also know that the native Marian engine performs even better with multithreading, but we had to disable multithreaded code in this version of the translation engine. Once SharedArrayBuffer support is broadly enabled, we believe we could re-enable multithreading and even faster translation speeds are possible.

Acknowledgement

I would like to thank Bergamot consortium partners, Mozilla’s Wasm team and my teammates Andre Natal, Evgeny Pavlov for their contributions in developing a mature translation engine. I am thankful to Lonnen along with Mozilla’s Add-on team, Localization team, Q&A team and Mozilla community who supported us and contributed to the development of the Firefox Translations add-on.

This project has received funding from the 🇪🇺European Union’s Horizon 2020 research and innovation programme under grant agreement No 825303.

The post Neural Machine Translation Engine for Firefox Translations add-on appeared first on Mozilla Hacks - the Web developer blog.

Mozilla Thunderbird — Thunderbird 102 Released: A Serious Upgrade To Your Communication

Thunderbird 102 is here! On behalf of the entire Thunderbird team, I’m thrilled to announce the availability of our major yearly release. Thunderbird 102 is loaded with highly-requested features, and we think you’ll be delighted by them.

It features refreshed icons, color folders, and quality-of-life upgrades like the redesigned message header. It ushers in a brand new Address Book to bring you closer than ever to the people you communicate with. Plus useful new tools to help you manage your data, navigate the app faster, and boost your productivity. We’re even bringing Matrix to the party!

[Press friends: we also have a press release with lots of screenshots and GIFs for you right here.]

New icons and color folders<figcaption>New icons and color folders</figcaption>

Thunderbird 102 is available to download right now for our global community of 20M+ Linux, macOS, and Windows users. (Using Thunderbird 91 and don’t need an immediate upgrade? Your client should automatically update within the next few weeks as we gradually roll it out.)

Let’s talk about some of the best new features in Thunderbird 102.

NEW ADDRESS BOOK: YOUR CONTACTS GET A SERIOUS UPGRADE IN THUNDERBIRD 102

You don’t communicate with contacts, you communicate with people. That’s why Thunderbird 102 gives you one of the best tools to better track and interact with all the important people in your life.

<figcaption>New Contact layout in Thunderbird 102</figcaption>
Thunderbird 102: Basic contact view in dark/light modes (Shown here on Linux)<figcaption>Basic contact view in dark/light modes (Shown here on Linux)</figcaption>

One of Thunderbird 102’s headlining features is the new Address Book, which represents the first of many steps towards a new, modernized direction for Thunderbird’s UX/UI. The redesigned Address Book delivers a clean visual presentation and several new information fields to help you better understand who you’re communicating with. It’s also compatible with vCard specs, the industry standard for saving contacts. If your app (like Google Contacts) or device (like an Android smartphone) can export existing contacts to vCard format, Thunderbird can effortlessly import them. Finally, the new Address Book acts as a one-stop launchpad for messaging or event creation with each of your contacts.

New Spaces Toolbar: Get There Faster

Customizing the Spaces Toolbar<figcaption>Thunderbird 102’s Spaces Toolbar</figcaption>

Thunderbird now features a central Spaces Toolbar for fast and easy access to your most important activities inside the application: With a single click, you can navigate between Mail, Address Book, Calendar, Tasks, Chat, and your installed add-ons. And if you feel like adapting Thunderbird to your personal preferences, there’s a button for Settings, too!

New Import / Export Wizard: Move Data With Ease

Thunderbird 102's Import / Export Wizard (shown here on Linux)<figcaption>Thunderbird 102’s Import / Export Wizard (shown here on Linux)</figcaption>

Moving accounts and data in and out of Thunderbird should be a breeze! Previously, this required the installation and use of add-ons, but the Thunderbird team is thrilled to share that it’s now a core, built-in feature. A step-by-step wizard provides a contextual, guided experience for importing your important data. This means that moving from Outlook, SeaMonkey, or another Thunderbird installation is easier than ever.

Redesigned Message Header: Focus On What Matters

The redesigned message header<figcaption>Message Header: TRANSFORM! </figcaption>

One of the core philosophies behind the development of Thunderbird 102 was doing more with less. Now, when reading your email, the redesigned message header allows you to focus on what matters as it highlights important information. It’s also more responsive and easier to navigate. Plus, you can “star” important messages from the header itself, and convert them into a calendar event or a task.

Matrix Chat Support: Decentralized Communication For Everyone

The Thunderbird team loves open source and open standards. That’s why you’ll find out-of-box support for the popular, decentralized chat protocol Matrix in Thunderbird 102. Matrix provides secure messaging, VOIP, and data transfer capabilities and is rapidly expanding its feature-set. 

We hope you love using it as much as we love building it! And there’s still more to come as Thunderbird 102’s development evolves! Complete details of all the changes can be found in the Thunderbird release notes for version 102 and newer.

The post Thunderbird 102 Released: A Serious Upgrade To Your Communication appeared first on The Thunderbird Blog.

The Rust Programming Language Blog — Announcing The RustConf PostConf UnConf

Hello Rust community!

We're excited to announce that the Rust project teams will be hosting an unconference1 the day after RustConf.

RustConf PostConf UnConf promo image

The PostConf Unconf will be dedicated to the Rust project and will be a fantastic opportunity for users, contributors, and maintainers to network and discuss the project's development.

There will be no set agenda; instead, attendees will decide what will be discussed together and can move freely between sessions to find ones in which they can contribute most effectively based on their individual interests and needs.

To get the most out of the unconference, jot down your thoughts ahead of time and bring them ready to share. We will also set up a channel in the RustConf Discord for folks to communicate and make preliminary, informal plans.

If you plan to attend, please register as soon as possible to help us plan appropriately. If we run out of space, project participants will be given preference. Registration is free and open to everyone attending RustConf.

🚨Register Now!🚨

We hope to see you there!

  1. If you are unfamiliar with the notion of an unconference, you may find this informational piece helpful. ↩

Hacks.Mozilla.Org — The JavaScript Specification has a New License

Ecma International recently approved the 2022 standard of ECMAScript. There is something new in this edition that hasn’t been part of prior editions, but this isn’t a new programming feature.

In March of this year, Ecma International accepted a proposal led by Mozilla for a new alternative license. On June 22nd, the first requests to adopt this license were granted to TC39 and applied to the following documents: ECMA-262 (ECMAScript, the official name for JavaScript) and ECMA-402 (the Internationalization API for ECMAScript).

The ECMAScript specification is developed at Ecma International, while other web technologies like HTML and CSS are being developed at W3C. These institutions have different default license agreements, which creates two problems. First, having different licenses increases the overhead of legal review for participants. This can create a speed bump for contributing across different specifications. Second, the default ECMA license contains some restrictions against creating derivative works, in contrast to W3C. These provisions haven’t been a problem in practice, but they nevertheless don’t reflect how we think Open Source should work, especially for something as foundational as JavaScript. Mozilla wants to make it easy for everyone to participate in evolving the Web, so we took the initiative of introducing an alternative license for Ecma International specifications.

What is the alternative license?

The full alternative license text may be found on the Ecma License FAQ. Ecma now provides two licenses, which can be adopted depending on the needs of a given technical committee. The default Ecma International license provides a definitive document and location for work on a given standard, with the intention of preventing forking. The license has provisions that allow inlining a given standard into source text, as well as reproduction in part or full.

The new alternative license seeks to align with the work of the W3C, and the text is largely based on the W3C’s Document and Software License. This license is more permissive regarding derivative works of a standard. This provides a legal framework and an important guarantee that the development of internet infrastructure can continue independent of any organization. By applying the alternative license to a standard as significant as ECMAScript, Ecma International has demonstrated its stewardship of a fundamental building block of the web. In addition, this presents a potential new home for standardization projects with similar licensing requirements.

Standards and Open Source

Standardization arises from the need of multiple implementers to align on a common design. Standardization improves collaboration across the industry, and reduces replicated solutions to the same problem. It also provides a way to gather feedback from users or potential users. Both Standards and Open Source produce technical solutions through collaboration. One notable distinction between standardization and an Open Source project is that the latter often focuses on developing solutions within a single implementation.

Open source has led the way with permissive licensing of projects. Over the years, different licenses such as the BSD, Creative Commons, GNU GPL & co, MIT, and MPL have sought to allow open collaboration with different focuses and goals. Standardizing bodies are gradually adopting more of the techniques of Open Source. In 2015, W3C adopted its Document and Software License, and in doing so moved many of the specifications responsible for the Web such as CSS and HTML. Under this new license, W3C ensured that the ability to build on past work would exist regardless of organizational changes.

Mozilla’s Role

As part of our work to ensure a free and open web, we worked together with Ecma International, and many partners to write a License inspired by the W3C Document and Software License. Our goal was that JavaScript’s status would align with other specifications of the Web. In addition, with this new license available to all TCs at Ecma International, this will provide other organizations to approach standardization with the same perspective.

Changes like this come from the work of many different participants and we thank everyone at TC39 who helped with this effort. In addition, I’d like also thank my colleagues at Mozilla for their excellent work: Zibi Braniecki and Peter Saint-Andre, who supported me in writing the document drafts and the Ecma International discussions; Daniel Nazer, Eric Rescorla, Bobby Holley and Tantek Çelik for their advice and guidance of this project.

The post The JavaScript Specification has a New License appeared first on Mozilla Hacks - the Web developer blog.

Niko Matsakis — Many modes: a GATs pattern

As some of you may know, on May 4th Jack Huey opened a PR to stabilize an initial version of generic associated types. The current version is at best an MVP: the compiler support is limited, resulting in unnecessary errors, and the syntax is limited, making code that uses GATs much more verbose than I’d like. Nonetheless, I’m super excited, since GATs unlock a lot of interesting use cases, and we can continue to smooth out the rough edges over time. However, folks on the thread have raised some strong concerns about GAT stabilization, including asking whether GATs are worth including in the language at all. The fear is that they make Rust the language too complex, and that it would be better to just use them as an internal building block for other, more accessible features (like async functions and [return position impl trait in traits][RPITIT]). In response to this concern, a number of people have posted about how they are using GATs. I recently took some time to deep dive into these comments and to write about some of the patterns that I found there, including a pattern I am calling the “many modes” pattern, which comes from the chumsky parser combinator library. I posted about this pattern on the thread, but I thought I would cross-post my write-up here to the blog as well, because I think it’s of general interest.

General thoughts from reading the examples

I’ve been going through the (many, many) examples that people have posted where they are relying on GATs and look at them in a bit more detail. A few interesting things jumped out at me as I read through the examples:

  • Many of the use-cases involve GATs with type parameters. There has been some discussion of stabilizing “lifetime-only” GATs, but I don’t think that makes sense from any angle. It’s more complex for the implementation and, I think, more confusing for the user. But also, given that the “workaround” for not having GATs tends to be higher-ranked trait bounds (HRTB), and given that those only work for lifetimes, it means we’re losing one of the primary benefits of GATs in practice (note that I do expect to get HRTB for types in the near-ish future).
  • GATs allowed libraries to better hide details from their clients. This is precisely because they could make a trait hierarchy that more directly captured the “spirit” of the trait, resulting in bounds like M: Mode instead of higher-ranked trait bounds (in some cases, the HRTB would have to be over types, like for<X> M: Mode<X>, which isn’t even legal in Rust…yet).

As I read, I felt this fit a pattern that I’ve experienced many times but hadn’t given a name to: when traits are being used to describe a situation that they don’t quite fit, the result is an explosion of where-clauses on the clients. Sometimes you can hide these via supertraits or something, but those complex bounds are still visible in rustdoc, still leak out in error mesages, and don’t generally “stay hidden” as well as you’d like. You’ll see this come up here when I talk about how you would model this pattern in Rust today, but it’s a comon theme across all examples. Issue #95 on the RustAudio crate for example says, “The first [solution] would be to make PortType generic over a 'a lifetime…however, this has a cascading effect, which would force all downstream users of port types to specify their lifetimes”. Pythonesque made a simpler point here, “Without GATs, I ended up having to make an Hkt trait that had to be implemented for every type, define its projections, and then make everything heavily parametric and generic over the various conversions.”

The “many modes” pattern (chumsky)

The first example I looked at closely was the chumsky parsing library. This is leveraging a pattern that I would call the “many modes” pattern. The idea is that you have some “core function” but you want to execute this function in many different modes. Ideally, you’d like to define the modes independently from the function, and you’d like to be able to add more modes later without having to change the function at all. (If you’re familiar with Haskell, monads are an example of this pattern; the monad specifies the “mode” in which some simple sequential function is executed.)

chumsky is a parser combinator library, so the “core function” is a parse function, defined in the Parser trait. Each Parser trait impl contains a function that indicates how to parse some particular construct in the grammar. Normally, this parser function builds up a data structure representing the parsed data. But sometimes you don’t need the full results of the parse: sometimes you might just like to know if the parse succeeds or fails, without building the parsed version. Thus, the “many modes” pattern: we’d like to be able to define our parser and then execute it against one of two modes, emit or check. The emit mode will build the data structure, but check will just check if the parse succeeds.

In the past, chumsky only had one mode, so they always built the data structure. This could take significant time and memory. Adding the “check” mode let’s them skip that, which is a significant performance win. Moreover, the modes are encapsulated within the library traits, and aren’t visible to end-users. Nice!

How did chumsky model modes with GATs?

Chumsky added a Mode trait, encapsulated as part of their internals module. Instead of directly constructing the results from parsing, the Parser impls invoke methods on Mode with closures. This allows the mode to decide which parts of the parsing to execute and which to skip. So, in check mode, the Mode would decide not to execute the closure that builds the output data structure, for example.

Using this approach, the Parser trait does indeed have several ‘entrypoint’ methods, but they are all defaulted and just invoke a common implementation method called go:

pub trait Parser<'a, I: Input + ?Sized, E: Error<I::Token> = (), S: 'a = ()> {
    type Output;
    
    fn parse(&self, input: &'a I) -> Result<Self::Output, E> ... {
        self.go::<Emit>(...)
    }

    fn check(&self, input: &'a I) -> Result<(), E> ... {
        self.go::<Check>(...)
    }
    
    #[doc(hidden)]
    fn go<M: Mode>(&self, inp: &mut InputRef<'a, '_, I, E, S>) -> PResult<M, Self::Output, E>
    where
        Self: Sized;
}

Implementations of Parser just specify the go method. Note that the impls are, presumably, either contained within chumsky or generated by chumsky proc-macros, so the go method doesn’t need to be documented. However, even if go were documented, the trait bounds certainly look quite reasonable. (The type of inp is a bit…imposing, admittedly.)

So how is the Mode trait defined? Just to focus on the GAT, the trait look likes this:

pub trait Mode {
    type Output<T>;
    ...
}

Here, the T represents the result type of “some parser parsed in this mode”. GATs thus allow us to define a Mode that is independent from any particular Parser. There are two impls of Mode (also internal to chumsky):

  • Check, defined like struct Check; impl Mode for Check { type Output<T> = (); ... }. In other words, no matter what parser you use, Check just builds a () result (success or failure is propagated inepdendently of the mode).
  • Emit, defined like struct Emit; impl Mode for Emit { type Output<T> = T; ... }. In Emit mode, the output is exactly what the parser generated.

Note that you could, in theory, produce other modes. For example, a Count mode that not only computes success/failure but counts the number of nodes parsed, or perhaps a mode that computes hashes of the resulting parsed value. Moreover, you could add these modes (and the defaulted methods in Parser) without breaking any clients.

How could you model this today?

I was trying to think how one might model this problem with traits today. All the options I came up with had significant downsides.

Multiple functions on the trait, or multiple traits. One obvious option would be to use multiple functions in the parse trait, or multiple traits:

// Multiple functions
trait Parser { fn parse(); fn check(); }

// Multiple traits
trait Parser: Checker { fn parse(); }
trait Checker { fn check(); }

Both of these approaches mean that defining a new combinator requires writing the same logic twice, once for parse and once for check, but with small variations, which is both annoying and a great opportunity for bugs. It also means that if chumsky ever wanted to define a new mode, they would have to modify every implementation of Parser (a breaking change, to boot).

Mode with a type parameter. You could try defining a the mode trait with a type parameter, like so…

trait ModeFor<T> {
    type Output;
    ...
}

The go function would then look like

fn go<M: ModeFor<Self::Output>>(&self, inp: &mut InputRef<'a, '_, I, E, S>) -> PResult<M, Self::Output, E>
where
    Self: Sized;

In practice, though, this doesn’t really work, for a number of reasons. One of them is that the Mode trait includes methods like combine, which take the output of many parsers, not just one, and combine them together. Good luck writing that constraint with ModeFor. But even ignoring that, lacking HRTB, the signature of go itself is incomplete. The problem is that, given some impl of Parser for some parser type MyParser, MyParser only knows that M is a valid mode for its particular output. But maybe MyParser plans to (internally) use some other parser combinators that produce different kinds of results. Will the mode M still apply to those? We don’t know. We’d have to be able to write a HRTB like for<O> Mode<O>, which Rust doesn’t support yet:

fn go<M: for<O> Mode<O>>(&self, inp: &mut InputRef<'a, '_, I, E, S>) -> PResult<M, Self::Output, E>
where
    Self: Sized;

But even if Rust did support it, you can see that the Mode<T> trait doesn’t capture the user’s intent as closely as the Mode trait from Chumsky did. The Mode trait was defined independently from all parsers, which is what we wanted. The Mode<T> trait is defined relative to some specific parser, and then it falls to the go function to say “oh, I want this to be a mode for all parsers” using a HRTB.

Using just HRTB (which, against, Rust doesn’t have), you could define another trait…

trait Mode: for<O> ModeFor<O> {}

trait ModeFor<O> {}

…which would allow us to write M: Mode on go against, but it’s hard to argue this is simpler than the original GAT variety. This extra ModeFor trait has a “code smell” to it, it’s hard to understand why it is there. Whereas before, you implemented the Mode trait in just the way you think about it, with a single impl that applies to all parsers…

impl Mode for Check {
    type Output<T> = ();
    ...
}

…you now write an impl of ModeFor, where one “instance” of the impl applies to only one parser (which has output type O). It feels indirect:

impl<O> ModeFor<O> for Check {
    type Output = ();
    ...
}

How could you model this with RPITIT?

It’s also been proposed that we should keep GATs, but only as an implementation detail for things like return position impl Trait in traits (RPITIT) or async functions. This implies that we could model the “many modes” pattern with RPITIT. If you look at the Mode trait, though, you’ll see that this simply doesn’t work. Consider the combine method, which takes the results from two parsers and combines them to form a new result:

fn combine<T, U, V, F: FnOnce(T, U) -> V>(
    x: Self::Output<T>,
    y: Self::Output<U>,
    f: F,
) -> Self::Output<V>;

How could we write this in terms of a function that returns impl Trait?

Other patterns

In this post, I went through the chumsky pattern in detail. I’ve not had time to dive quite as deep into other examples, but I’ve been reading through them and trying to extract out patterns. Here are a few patterns I extracted so far:

  • The “generic scopes” pattern (smithay, playground):
    • In the Smithay API, if you have some variable r: R where R: Renderer, you can invoke r.render(|my_frame| ...). This will invoke your callback with some frame my_frame that you can then modify. The thing is that the type of my_frame depends on the type of renderer that you have; moreover, frames often include thread-local data and so should only be accessible to during that callback.
    • I called this the “generic scopes” pattern because, at least from a types POV, it is kind of a generic version of APIs like std::thread::scope. The scope function also uses a callback to give limited access to a variable (the “thread scope”), but in the case of std::thread::scope, the type of that scope is hard-coded to be std::thread::Scope, whereas here, we want the specific type to depend on the renderer.
    • Thanks to GATs, you can express that pretty cleanly, so that the only bound you need is R: Renderer. As with “many modes”, if you tried to express it using features today, you can get part of the way there, but the bounds will be complex and involve HRTB.
  • The “pointer types” pattern:
    • I didn’t dig deep enough into Pythonesque’s hypotheticals, but this comment seemed to be describing a desire to talk about “pointer types” in the abstract, which is definitely a common need; looking at the comits from Veloren that pythonesque also cited, this might be a kind of “pointer types” pattern, but I think I might also call it “many modes”.
  • The “iterable” pattern:
    • In this pattern, you would like a way to say where C: Iterable, meaning that C is a collection with an iter method which fits the signature fn iter(&self) -> impl Iterator<Item = &T>. This is distinct from IntoIterator because it takes &self and thus we can iterate over the same collection many times and concurrently.
    • The most common workaround is to return a Box<dyn> (as in graphene) or a collection (as in metamolectular). Neither is zero-cost, which can be a problem in tight loops, as commented here. You can also use HRTB (as rustc does, which is complex and leaky.

Did I miss something?

Maybe you see a way to express the “many modes” pattern (or one of the other patterns I cited) in Rust today that works well? Let me know by commenting on the thread.

(Since posting this, it occurs to me that one could probably use procedural macros to achieve some similar goals, though I think this approach would also have significant downsides.)

Hacks.Mozilla.Org — Fuzzing rust-minidump for Embarrassment and Crashes – Part 2

This is part 2 of a series of articles on rust-minidump. For part 1, see here.

So to recap, we rewrote breakpad’s minidump processor in Rust, wrote a ton of tests, and deployed to production without any issues. We killed it, perfect job.

And we still got massively dunked on by the fuzzer. Just absolutely destroyed.

I was starting to pivot off of rust-minidump work because I needed a bit of palette cleanser before tackling round 2 (handling native debuginfo, filling in features for other groups who were interested in rust-minidump, adding extra analyses that we’d always wanted but were too much work to do in Breakpad, etc etc etc).

I was still getting some PRs from people filling in the corners they needed, but nothing that needed too much attention, and then @5225225 smashed through the windows and released a bunch of exploding fuzzy rabbits into my office.

I had no idea who they were or why they were there. When I asked they just lowered one of their seven pairs of sunglasses and said “Because I can. Now hold this bunny”. I did as I was told and held the bunny. It was a good bun. Dare I say, it was a true bnnuy: it was libfuzzer. (Huh? You thought it was gonna be AFL? Weird.)

As it turns out, several folks had built out some really nice infrastructure for quickly setting up a decent fuzzer for some Rust code: cargo-fuzz. They even wrote a little book that walks you through the process.

Apparently those folks had done such a good job that 5225225 had decided it would be a really great hobby to just pick up a random rust project and implement fuzzing for it. And then to fuzz it. And file issues. And PRs that fix those issues. And then implement even more fuzzing for it.

Please help my office is drowning in rabbits and I haven’t seen my wife in weeks.

As far as I can tell, the process seems to genuinely be pretty easy! I think their first fuzzer for rust-minidump was basically just:

  • checked out the project
  • run cargo fuzz init (which autogenerates a bunch of config files)
  • write a file with this:
#![no_main]

use libfuzzer_sys::fuzz_target;
use minidump::*;

fuzz_target!(|data: &[u8]| {
    // Parse a minidump like a normal user of the library
    if let Ok(dump) = minidump::Minidump::read(data) {
        // Ask the library to get+parse several streams like a normal user.

        let _ = dump.get_stream::<MinidumpAssertion>();
        let _ = dump.get_stream::<MinidumpBreakpadInfo>();
        let _ = dump.get_stream::<MinidumpCrashpadInfo>();
        let _ = dump.get_stream::<MinidumpException>();
        let _ = dump.get_stream::<MinidumpLinuxCpuInfo>();
        let _ = dump.get_stream::<MinidumpLinuxEnviron>();
        let _ = dump.get_stream::<MinidumpLinuxLsbRelease>();
        let _ = dump.get_stream::<MinidumpLinuxMaps>();
        let _ = dump.get_stream::<MinidumpLinuxProcStatus>();
        let _ = dump.get_stream::<MinidumpMacCrashInfo>();
        let _ = dump.get_stream::<MinidumpMemoryInfoList>();
        let _ = dump.get_stream::<MinidumpMemoryList>();
        let _ = dump.get_stream::<MinidumpMiscInfo>();
        let _ = dump.get_stream::<MinidumpModuleList>();
        let _ = dump.get_stream::<MinidumpSystemInfo>();
        let _ = dump.get_stream::<MinidumpThreadNames>();
        let _ = dump.get_stream::<MinidumpThreadList>();
        let _ = dump.get_stream::<MinidumpUnloadedModuleList>();
    }
});

And that’s… it? And all you have to do is type cargo fuzz run and it downloads, builds, and spins up an instance of libfuzzer and finds bugs in your project overnight?

Surely that won’t find anything interesting. Oh it did? It was largely all bugs in code I wrote? Nice.

cargo fuzz is clearly awesome but let’s not downplay the amount of bafflingly incredible work that 5225225 did here! Fuzzers, sanitizers, and other code analysis tools have a very bad reputation for drive-by contributions.

I think we’ve all heard stories of someone running a shiny new tool on some big project they know nothing about, mass filing a bunch of issues that just say “this tool says your code has a problem, fix it” and then disappearing into the mist and claiming victory.

This is not a pleasant experience for someone trying to maintain a project. You’re dumping a lot on my plate if I don’t know the tool, have trouble running the tool, don’t know exactly how you ran it, etc.

It’s also very easy to come up with a huge pile of issues with very little sense of how significant they are.

Some things are only vaguely dubious, while others are horribly terrifying exploits. We only have so much time to work on stuff, you’ve gotta help us out!

And in this regard 5225225’s contributions were just, bloody beautiful.

Like, shockingly fantastic.

They wrote really clear and detailed issues. When I skimmed those issues and misunderstood them, they quickly clarified and got me on the same page. And then they submitted a fix for the issue before I even considered working on the fix. And quickly responded to review comments. I didn’t even bother asking them to squashing their commits because damnit they earned those 3 commits in the tree to fix one overflow.

Then they submitted a PR to merge the fuzzer. They helped me understand how to use it and debug issues. Then they started asking questions about the project and started writing more fuzzers for other parts of it. And now there’s like 5 fuzzers and a bunch of fixed issues!

I don’t care how good cargo fuzz is, that’s a lot of friggin’ really good work! Like I am going to cry!! This was so helpful??? 😭

That said, I will take a little credit for this going so smoothly: both Rust itself and rust-minidump are written in a way that’s very friendly to fuzzing. Specifically, rust-minidump is riddled with assertions for “hmm this seems messed up and shouldn’t happen but maybe?” and Rust turns integer overflows into panics (crashes) in debug builds (and index-out-of-bounds is always a panic).

Having lots of assertions everywhere makes it a lot easier to detect situations where things go wrong. And when you do detect that situation, the crash will often point pretty close to where things went wrong.

As someone who has worked on detecting bugs in Firefox with sanitizer and fuzzing folks, let me tell you what really sucks to try to do anything with: “Hey so on my machine this enormous complicated machine-generated input caused Firefox to crash somewhere this one time. No, I can’t reproduce it. You won’t be able to reproduce it either. Anyway, try to fix it?”

That’s not me throwing shade on anyone here. I am all of the people in that conversation. The struggle of productively fuzzing Firefox is all too real, and I do not have a good track record of fixing those kinds of bugs. 

By comparison I am absolutely thriving under “Yeah you can deterministically trip this assertion with this tiny input you can just check in as a unit test”.

And what did we screw up? Some legit stuff! It’s Rust code, so I am fairly confident none of the issues were security concerns, but they were definitely quality of implementation issues, and could have been used to at very least denial-of-service the minidump processor.

Now let’s dig into the issues they found!

#428: Corrupt stacks caused infinite loops until OOM on ARM64

Issue

As noted in the background, stackwalking is a giant heuristic mess and you can find yourself going backwards or stuck in an infinite loop. To keep this under control, stackwalkers generally require forward progress.

Specifically, they require the stack pointer to move down the stack. If the stack pointer ever goes backwards or stays the same, we just call it quits and end the stackwalk there.

However, you can’t be so strict on ARM because leaf functions may not change the stack size at all. Normally this would be impossible because every function call at least has to push the return address to the stack, but ARM has the link register which is basically an extra buffer for the return address.

The existence of the link register in conjunction with an ABI that makes the callee responsible for saving and restoring it means leaf functions can have 0-sized stack frames!

To handle this, an ARM stackwalker must allow for there to be no forward progress for the first frame of a stackwalk, and then become more strict. Unfortunately I hand-waved that second part and ended up allowing infinite loops with no forward progress:

// If the new stack pointer is at a lower address than the old,
// then that's clearly incorrect. Treat this as end-of-stack to
// enforce progress and avoid infinite loops.
//
// NOTE: this check allows for equality because arm leaf functions
// may not actually touch the stack (thanks to the link register
// allowing you to "push" the return address to a register).
if frame.context.get_stack_pointer() < self.get_register_always("sp") as u64 {
    trace!("unwind: stack pointer went backwards, assuming unwind complete");
    return None;
}

So if the ARM64 stackwalker ever gets stuck in an infinite loop on one frame, it will just build up an infinite backtrace until it’s killed by an OOM. This is very nasty because it’s a potentially very slow denial-of-service that eats up all the memory on the machine!

This issue was actually originally discovered and fixed in #300 without a fuzzer, but when I fixed it for ARM (32-bit) I completely forgot to do the same for ARM64. Thankfully the fuzzer was evil enough to discover this infinite looping situation on its own, and the fix was just “copy-paste the logic from the 32-bit impl”.

Because this issue was actually encountered in the wild, we know this was a serious concern! Good job, fuzzer!

(This issue specifically affected minidump-processor and minidump-stackwalk)

#407: MinidumpLinuxMaps address-based queries didn’t work at all

Issue

MinidumpLinuxMaps is an interface for querying the dumped contents of Linux’s /proc/self/maps file. This provides metadata on the permissions and allocation state for mapped ranges of memory in the crashing process.

There are two usecases for this: just getting a full dump of all the process state, and specifically querying the memory properties for a specific address (“hey is this address executable?”). The dump usecase is handled by just shoving everything in a Vec. The address usecase requires us to create a RangeMap over the entries.

Unfortunately, a comparison was flipped in the code that created the keys to the RangeMap, which resulted in every correct memory range being discarded AND invalid memory ranges being accepted. The fuzzer was able to catch this because the invalid ranges tripped an assertion when they got fed into the RangeMap (hurray for redundant checks!).

// OOPS
if self.base_address < self.final_address { 
 return None; 
}

Although tests were written for MinidumpLinuxMaps, they didn’t include any invalid ranges, and just used the dump interface, so the fact that the RangeMap was empty went unnoticed!

This probably would have been quickly found as soon as anyone tried to actually use this API in practice, but it’s nice that we caught it beforehand! Hooray for fuzzers!

(This issue specifically affected the minidump crate which technically could affect minidump-processor and minidump-stackwalk. Although they didn’t yet actually do address queries, they may have crashed when fed invalid ranges.)

#381: OOM from reserving memory based on untrusted list length.

Issue

Minidumps have lots of lists which we end up collecting up in a Vec or some other collection. It’s quite natural and more efficient to start this process with something like Vec::with_capacity(list_length). Usually this is fine, but if the minidump is corrupt (or malicious), then this length could be impossibly large and cause us to immediately OOM.

We were broadly aware that this was a problem, and had discussed the issue in #326, but then everyone left for the holidays. #381 was a nice kick in the pants to actually fix it, and gave us a free simple test case to check in.

Although the naive solution would be to fix this by just removing the reserves, we opted for a solution that guarded against obviously-incorrect array lengths. This allowed us to keep the performance win of reserving memory while also making rust-minidump fast-fail instead of vaguely trying to do something and hallucinating a mess.

Specifically, @Swatinem introduced a function for checking that the amount of memory left in the section we’re parsing is large enough to even hold the claimed amount of items (based on their known serialized size). This should mean the minidump crate can only be induced to reserve O(n) memory, where n is the size of the minidump itself.

For some scale:

  • A minidump for Firefox’s main process with about 100 threads is about 3MB.
  • A minidump for a stackoverflow from infinite recursion (8MB stack, 9000 calls) is about 8MB.
  • A breakpad symbol file for Firefox’s main module can be about 200MB.

If you’re symbolicating, Minidumps probably won’t be your memory bottleneck. 😹

(This issue specifically affected the minidump crate and therefore also minidump-processor and minidump-stackwalk.)

The Many Integer Overflows and My Greatest Defeat

The rest of the issues found were relatively benign integer overflows. I claim they’re benign because rust-minidump should already be working under the assumption that all the values it reads out of the minidump could be corrupt garbage. This means its code is riddled with “is this nonsense” checks and those usually very quickly catch an overflow (or at worst print a nonsense value for some pointer).

We still fixed them all, because that’s shaky as heck logic and we want to be robust. But yeah none of these were even denial-of-service issues, as far as I know.

To demonstrate this, let’s discuss the most evil and embarrassing overflow which was definitely my fault and I am still mad about it but in a like “how the heck” kind of way!?

The overflow is back in our old friend the stackwalker. Specifically in the code that attempts to unwind using frame pointers. Even more specifically, when offsetting the supposed frame-pointer to get the location of the supposed return address:

let caller_ip = stack_memory.get_memory_at_address(last_bp + POINTER_WIDTH)?;
let caller_bp = stack_memory.get_memory_at_address(last_bp)?;
let caller_sp = last_bp + POINTER_WIDTH * 2;

If the frame pointer (last_bp) was ~u64::MAX, the offset on the first line would overflow and we would instead try to load ~null. All of our loads are explicitly fallible (we assume everything is corrupt garbage!), and nothing is ever mapped to the null page in normal applications, so this load would reliably fail as if we had guarded the overflow. Hooray!

…but the overflow would panic in debug builds because that’s how debug builds work in Rust!

This was actually found, reported, and fixed without a fuzzer in #251. All it took was a simple guard:

(All the casts are because this specific code is used in the x86 impl and the x64 impl.)

if last_bp as u64 >= u64::MAX - POINTER_WIDTH as u64 * 2 {
    // Although this code generally works fine if the pointer math overflows,
    // debug builds will still panic, and this guard protects against it without
    // drowning the rest of the code in checked_add.
    return None;
}

let caller_ip = stack_memory.get_memory_at_address(last_bp as u64 + POINTER_WIDTH as u64)?;
let caller_bp = stack_memory.get_memory_at_address(last_bp as u64)?;
let caller_sp = last_bp + POINTER_WIDTH * 2;

And then it was found, reported, and fixed again with a fuzzer in #422.

Wait what?

Unlike the infinite loop bug, I did remember to add guards to all the unwinders for this problem… but I did the overflow check in 64-bit even for the 32-bit platforms.

slaps forehead

This made the bug report especially confusing at first because the overflow was like 3 lines away from a guard for that exact overflow. As it turns out, the mistake wasn’t actually as obvious as it sounds! To understand what went wrong, let’s talk a bit more about pointer width in minidumps.

A single instance of rust-minidump has to be able to handle crash reports from any platform, even ones it isn’t natively running on. This means it needs to be able to handle both 32-bit and 64-bit platforms in one binary. To avoid the misery of copy-pasting everything or making everything generic over pointer size, rust-minidump prefers to work with 64-bit values wherever possible, even for 32-bit plaftorms.

This isn’t just us being lazy: the minidump format itself does this! Regardless of the platform, a minidump will refer to ranges of memory with a MINIDUMP_MEMORY_DESCRIPTOR whose base address is a 64-bit value, even on 32-bit platforms!

typedef struct _MINIDUMP_MEMORY_DESCRIPTOR {
  ULONG64                      StartOfMemoryRange;
  MINIDUMP_LOCATION_DESCRIPTOR Memory;
} MINIDUMP_MEMORY_DESCRIPTOR, *PMINIDUMP_MEMORY_DESCRIPTOR;

So quite naturally rust-minidump’s interface for querying saved regions of memory just operates on 64-bit (u64) addresses unconditionally, and 32-bit-specific code casts its u32 address to a u64 before querying memory.

That means the code with the overflow guard was manipulating those values as u64s on x86! The problem is that after all the memory loads we would then go back to “native” sizes and compute caller_sp = last_bp + POINTER_WIDTH * 2. This would overflow a u32 and crash in debug builds. 😿

But here’s the really messed up part: getting to that point meant we were successfully loading memory up to that address. The first line where we compute caller_ip reads it! So this overflow means… we were… loading memory… from an address that was beyond u32::MAX…!?

Yes!!!!!!!!

The fuzzer had found an absolutely brilliantly evil input.

It abused the fact that MINIDUMP_MEMORY_DESCRIPTOR technically lets 32-bit minidumps define memory ranges beyond u32::MAX even though they could never actually access that memory! It could then have the u64-based memory accesses succeed but still have the “native” 32-bit operation overflow!

This is so messed up that I didn’t even comprehend that it had done this until I wrote my own test and realized that it wasn’t actually failing because I foolishly had limited the range of valid memory to the mere 4GB a normal x86 process is restricted to.

And I mean that quite literally: this is exactly the issue that creates Parallel Universes in Super Mario 64.

But hey my code was probably just bad. I know google loves sanitizers and fuzzers, so I bet google breakpad found this overflow ages ago and fixed it:

uint32_t last_esp = last_frame->context.esp;
uint32_t last_ebp = last_frame->context.ebp;
uint32_t caller_eip, caller_esp, caller_ebp;

if (memory_->GetMemoryAtAddress(last_ebp + 4, &caller_eip) &&
    memory_->GetMemoryAtAddress(last_ebp, &caller_ebp)) {
    caller_esp = last_ebp + 8;
    trust = StackFrame::FRAME_TRUST_FP;
} else {
    ...

Ah. Hmm. They don’t guard for any kind of overflow for those uint32_t’s (or the uint64_t’s in the x64 impl).

Well ok GetMemoryAtAddress does actual bounds checks so the load from ~null will generally fail like it does in rust-minidump. But what about the Parallel Universe overflow that lets GetMemoryAtAddress succeed?

Ah well surely breakpad is more principled with integer width than I was–

virtual bool GetMemoryAtAddress(uint64_t address, uint8_t*  value) const = 0;
virtual bool GetMemoryAtAddress(uint64_t address, uint16_t* value) const = 0;
virtual bool GetMemoryAtAddress(uint64_t address, uint32_t* value) const = 0;
virtual bool GetMemoryAtAddress(uint64_t address, uint64_t* value) const = 0;

Whelp congrats to 5225225 for finding an overflow that’s portable between two implementations in two completely different languages by exploiting the very nature of the file format itself!

In case you’re wondering what the implications of this overflow are: it’s still basically benign. Both rust-minidump and google-breakpad will successfully complete the frame pointer analysis and yield a frame with a ~null stack pointer.

Then the outer layer of the stackwalker which runs all the different passes in sequence will see something succeeded but that the frame pointer went backwards. At this point it will discard the stack frame and terminate the stackwalk normally and just calmly output whatever the backtrace was up to that point. Totally normal and reasonable operation.

I expect this is why no one would notice this in breakpad even if you run fuzzers and sanitizers on it: nothing in the code actually does anything wrong. Unsigned integers are defined to wrap, the program behaves reasonably, everything is kinda fine. We only noticed this in rust-minidump because all integer overflows panic in Rust debug builds.

However this “benign” behaviour is slightly different from properly guarding the overflow. Both implementations will normally try to move on to stack scanning when the frame pointer analysis fails, but in this case they give up immediately. It’s important that the frame pointer analysis properly identifies failures so that this cascading can occur. Failing to do so is definitely a bug!

However in this case the stack is partially in a parallel universe, so getting any kind of useful backtrace out of it is… dubious to say the least.

So I totally stand by “this is totally benign and not actually a problem” but also “this is sketchy and we should have the bounds check so we can be confident in this code’s robustness and correctness”.

Minidumps are all corner cases — they literally get generated when a program encounters an unexpected corner case! It’s so tempting to constantly shrug off situations as “well no reasonable program would ever do this, so we can ignore it”… but YOU CAN’T.

You would not have a minidump at your doorstep if the program had behaved reasonably! The fact that you are trying to inspect a minidump means something messed up happened, and you need to just deal with it!

That’s why we put so much energy into testing this thing, it’s a nightmare!

I am extremely paranoid about this stuff, but that paranoia is based on the horrors I have seen. There are always more corner cases.

There are ALWAYS more corner cases. ALWAYS.

 

The post Fuzzing rust-minidump for Embarrassment and Crashes – Part 2 appeared first on Mozilla Hacks - the Web developer blog.

Jody Heavener — Introducing 1Password for Visual Studio Code

In writing software, we’re used to embedding secrets and other configurable values right in the codebase. They might be Stripe keys to power your online shop, webhooks for a custom Slack bot, a Docker username and password for a CI config, AWS credentials, or an API token and host to set up 1Password Connect.

Secrets are used everywhere in our code. Sometimes, though, we forget when we’ve been using real secrets in our work. Maybe there’s a leftover token you dropped in to build that one feature, or maybe you didn’t delete the .env file you set up to test drive the app. Now you’ve got to rotate your secrets because you accidentally committed and pushed sensitive values for the whole world to see. Yikes.

We’ve all been there. That’s why I’m delighted that I get to announce the launch of the all-new 1Password for VS Code extension.

Go ahead, commit your secrets references

With 1Password Secrets Automation, the 1Password Developer Products team introduced the concept of secret references. It starts by storing a sensitive value, such as an API credential or client ID, in 1Password. That item and the field you’d like to get the value from can then be retrieved through a special op:// URL scheme that 1Password’s tooling knows how to parse. It’s made up of three parts: vault, item, and field. This is known as a “secret reference”.

1Password Secret Reference example consisting of vault, item, and field

Now, instead of using a real value in your configs, environment variable files, or anywhere else in the codebase, just drop in the secret reference in VS Code. When you do, you can rest easy knowing that the real value will never accidentally make its way into your codebase.

The best part? Through our suite of tools and integrations, you can work with references in both local and deployed environments.

To help make sure you’re not accidentally leaving secrets in your code, you can move them over to 1Password with just a couple clicks. The extension uses a series of secret detection techniques to look for values that might be sensitive. With these matches, it makes inline suggestions to store them in 1Password, automatically replacing them with secret references.

Example of a secret that would be replaced by the 1Password extension for VS Code

Secret reference integration doesn’t stop there. You can hover a reference to inspect the item and field details, click it to open the item in the desktop app, and even preview the real values of an entire file full of references.

Beyond secret detection suggestions, 1Password for VS Code makes it easy to retrieve items for use in your code, as well as store any bits of code you’d like in 1Password. If you’ve got multiple values you want stored in the same item – perhaps a username, password, and email – it supports that as well. Just select each value and run the “Save in 1Password” command.

VSCode Screenshot

Built using tools available to everyone

I’ll let you in on a little secret: we didn’t plan to build this extension. It wasn’t requested by our developer community, and wasn’t part of any roadmap. Instead this extension began as a side project for myself. I wanted to scratch my own itch and integrate 1Password more closely into my development workflow, and to generally learn more about developing with 1Password.

So you can imagine my excitement when, after a quick demo at an internal call, I was invited to polish it up and get it slated for release.

To my delight, after demoing the extension and then going on vacation, Dave posted a video of the presentation from his CLI launch blog post and it was met with some pretty wild enthusiasm from the developer community. There was even some love for it at our 1Password 8 for Mac Reddit AMA:

Reddit comment from user arnebr asking if the VS Code plugin will be live soon

Although not a goal from the outset, an interesting aspect of this project is that it’s built using only tools available to the public – there’s nothing internal or proprietary powering the features of the extension. We’ve even open-sourced the whole project on our GitHub, so if you want to help iterate on it or report an issue, that’s a great place to start.

VS Code extensions run in a Node environment, and we wanted to interact with the new CLI. So we built and open-sourced an entirely new package for doing exactly this: op-js. It wraps the CLI with a simple-to-use JavaScript interface and ships with TypeScript declarations, making 60+ commands, including those that support biometrics unlock, available to your Node-based application.

Ultimately my hope is that this extension demonstrates some of the neat ways you can extend the power of 1Password by building your own integrations, whether it be for yourself or others. And you should have fun doing it. We’re in early days here, with plenty more developer offerings coming down the line.

I’d love to hear what you think, and we’ll be iterating on the extension as feedback rolls in. Learn more about 1Password for VS Code and our other developer tools by checking out our developer portal. While you’re there, consider joining our Developer Slack workspace, where you’ll find myself and others on the Developer Products team who are eager to hear how you’re incorporating 1Password into your development workflow. And if you’re building something cool, be sure to tag it #BuildWith1Password!

Finally, we owe a tremendous debt of gratitude to Mike Selander, Chris Dunn-Birch, Floris van der Grinten, the incredibly helpful folks over in the VS Code Extension community, and so many more for providing endless help and guidance while working on this project. Thank you!

Pascal Chevrel — MozFR Transvision Reloaded: 1 year later

Just one year ago, the French Mozilla community was living times of major changes: several key historical contributors were leaving the project, our various community portals were no longer updates or broken, our tools were no longer maintained. At the same time a few new contributors were also popping in our IRC channel asking for ways to get involved in the French Mozilla community.

As a result, Kaze decided to organize the first ever community meetup for the French-speaking community in the Paris office (and we will repeat this meetup in June in the brand new Paris office!) .

IMG_0151

This resulted in a major and successful community reboot. Leaving contributors passed on the torch to other members of the community, newer contributors were meeting in real life for the first time. This is how Clarista officially became our events organizer, this is how Théo replaced Cédric as the main Firefox localizer and this is how I became the new developer for Transvision! :)

What is Transvision? Transvision is a web application created by Philippe Dessantes which was helping the French team finding localized/localizable strings in Mozilla repositories.

Summarized like that, it doesn't sound that great, but believe me, it is! Mozilla applications have big gigantic repos, there are tens of thousands of strings in our mercurial repositories, some of them we have translated like a decade ago, when you decide to change a verb for a better one for example, it is important to be able to find all occurrences of this verb you have used in the past to see if they need update too. When somebody spots a typo or a clumsy wording, it's good to be able to check if you didn't make the same translation mistakes in other parts of the Mozilla applications several years ago and of course, it's good to be able to check that in just a few seconds. Basically, Phillippe had built the QA/assistive technology for our localization process that best fitted our team needs and we just couldn't let it die.

During the MozFR meetup, Philippe showed to me how the application worked and we created a github repository where we put the currently running version of the code. I tagged that code as version 1.0.

Over the summer, I familiarized myself with the code which was mostly procedural PHP, several Bash scripts to maintain copies of our mercurial repos and a Python script used to extract the strings. Quickly, I decided that I would follow the old Open Source strategy of Release early, release often. Since I was doing that on the sidelines of my job at Mozilla,  I needed the changes to be small but frequent incremental steps as I didn't know how much time I could devote to this project. Basically, having frequent releases means that I always have the codebase in mind which is good since I can implement an idea quickly, without having to dive into the code to remember it.

One year and 15 releases later, we are now at version 2.5, so here are the features and achievements I am most proud of:

  1. Transvision is alive and kicking :)
  2. We are now a team! Jesús Perez has been contributing code since last December, a couple more people have shown interest in contributing and Philippe is interested in helping again too. We have also a dynamic community of localizers giving feedback, reporting bugs are asking for immrovements
  3. The project is now organized and if some day I need to step down and pass the torch to another maintainer, he should not have difficulties setting the project up and maintaining it. We have a github repo, release notes, bugs, tagged releases, a beta server, unit testing, basic stats to understand what is used in the app and a mostly cleaned up codebase using much more modern PHP and tools (Atoum, Composer). It's not perfect, but I think that for amateur developers, it's not bad at all and the most important thing is that the code keeps on improving!
  4. There are now more than 3000 searches per week done by localizers on Transvision. That was more like 30 per week a year ago. There are searches in more than 70 languages, although 30 locales are doing the bulk of searches and French is still the biggest consumer with 40% of requests.
  5. Some people are using Transvision in ways I hadn't anticipated, for example our documentation localizers use it to find the translation of UI mentioned in help articles they translate for support.mozilla.org, people in QA use it to point to localized strings in Bugzilla

A quick recap of what we have done, feature-wise, in the last 12 months:

  • Completely redesigned the application to look and feel good
  • Locale to Locale searches, English is not necessarily the locale you want to use as the source (very useful to check differences from languages of the same family, for example Occitan/French/Catalan/Spanish...).
  • Hints and warnings for strings that look too long or too short compare to English, potentially bad typography, access keys that don't match your translation...
  • Possibility for anybody to file a bug in Bugzilla with a pointer to the badly translated string (yes we will use it for QA test days within the French community!)
  • Firefox OS strings are now there
  • Search results are a lot more complete and accurate
  • We now have a stable Json/JsonP API, I know that Pontoon uses it to provide translation suggestions, I heard that the Moses project uses it too. (if you use the Transvision API, ping me, I'd like to know!)
  • We can point any string to the right revision controled file in the source and target repos
  • We have a companion add-on called MozTran for heavy users of the tool provided by Goofy, from our Babelzilla friends.

The above list is of course just a highlight of the main features, you can get more details on the changelog.

If you use Transvision, I hope you enjoy it and that it is useful oo you. If you don't use Transvision (yet), give it a try, it may help you in your translation process, especially if your localization process is similar to the French one (targets Firefox Nighty builds first, work directly on the mercurial repo, focus on QA).

This was the first year of the rebirth of Transvision, I hope that the year to come will be just as good as this one. I learnt a lot with this project and I am happy to see it grow both in terms of usage and community, I am also happy that one tool that was created by a specific localization team is now used by so many other teams in the world :)

Pascal Chevrel — My Q2-2014 report

Summary of what I did last quarter (regular l10n-drivers work such as patch reviews, pushes to production, meetings and past projects maintenance excluded) .

Australis release

At the end of April, we shipped Firefox 29 which was our first major redesign of the Firefox user interface since Firefox 4 (released in 2011). The code name for that was Australis and that meant replacing a lot of content on mozilla.org to introduce this new UI and the new features that go with it. That also means that we were able to delete a lot of old content that now had become really obsolete or that was now duplicated on our support site.

Since this was a major UI change, we decided to show an interactive tour of the new UI to both new users and existing users upgrading to the new version. That tour was fully localized in a few weeks time in close to 70 languages, which represents 97.5% of our user base. For the last locales not ready on time, we either decided to show them a partially translated site (some locales had translated almost everything or some of the non-translated strings were not very visible to most users, such as alternative content to images for screen readers) or to let the page fall back to the best language available (like Occitan falling back to French for example).

Mozilla.org was also updated with 6 new product pages replacing a lot of old content as well as updates to several existing pages. The whole site was fully ready for the launch with 60 languages 100% ready and 20 partially ready, all that done in a bit less than 4 weeks, parallel to the webdev integration work.

I am happy to say that thanks to our webdev team, our amazing l10n community and with the help of my colleagues Francesco Lodolo (also Italian localizer) and my intern Théo Chevalier (also French localizer), we were able to not only offer a great upgrading experience for the quasi totality of our user base, we were also able to clean up a lot of old content, fix many bugs and prepare the site from an l10n perspective for the upcoming releases of our products.

Today, for a big locale spanning all of our products and activities, mozilla.org is about 2,000 strings to translate and maintain (+500 since Q1), for a smaller locale, this is about 800 strings (+200 since Q1). This quarter was a significant bump in terms of strings added across all locales but this was closely related to the Australis launch, we shouldn't have such a rise in strings impacting all locales in the next quarters.

Transvision releases

Last quarter we did 2 releases of Transvision with several features targeting out 3 audiences: localizers, localization tools, current and potential Transvision developers.

For our localizers, I worked on a couple of features, one is quick filtering of search results per component for Desktop repositories (you search for 'home' and with one click, you can filter the results for the browser, for mail or for calendar for example). The other one is providing search suggestions when your search yields no results with the best similar matches ("your search for 'lookmark' yielded no result, maybe you were searching for 'Bookmark'?").

For the localization tools community (software or web apps like Pontoon, Mozilla translator, Babelzilla, OmegaT plugins...), I rewrote entirely our old Json API and extended it to provide more services. Our old API was initially created for our own purposes and basically was just giving the possibility to get our search results as a Json feed on our most popular views. Tools started using it a couple of years ago and we also got requests for API changes from those tool makers, therefore it was time to rewrite it entirely to make it scalable. Since we don't want to break anybody's workflow, we now redirect all the old API calls to the new API ones. One of the significant new service to the API is a translation memory query that gives you results and a quality index based on the Levenshtein distance with the searched terms. You can get more information on the new API in our documentation.

I also worked on improving our internal workflow and make it easier for potential developers wanting to hack on Transvision to install and run it locally. That meant that now we do continuous integration with Travis CI (all of our unit tests are ran on each commit and pull request on PHP 5.4 and 5.5 environments), we have made a lot of improvements to our unit tests suite and coverage, we expose to developers peak memory usage and time per request on all views so as to catch performance problems early, and we also now have a "dev" mode that allows getting Transvision installed and running on the PHP development server in a matter of minutes instead of hours for a real production mode. One of the blockers for new developers was the time required to install Transvision locally. Since it is a spidering tool looking for localized strings in Mozilla source repositories, it needed to first clone all the repositories it indexes (mercurial/git/svn) which is about 20GB of data and takes hours even with a fast connection. We are now providing a snapshot of the final extracted data (still 400MB ;)) every 6 hours that is used by the dev install mode.

Check the release notes for 3.3 and 3.4 to see what other features were added by the team (/ex: on demand TMX generation or dynamic Gaia comparison view added by Théo, my intern).

Web dashboard / Langchecker

The main improvement I brought to the web dashboard is probably this quarter the deadline field to all of our .lang files, which allows to better communicate the urgency of projects and for localizers are an extra parameter allowing them to prioritize their work.

Theo's first project for his internship was to build a 'project' view on the web dashboard that we can use to get an overview of the translation of a set of pages/files, this was used for the Australis release (ex: http://l10n.mozilla-community.org/webdashboard/?project=australis_all) but can be used to any other project we want to define , here is an example for the localization of two Android Add-ons I did for the World Cup that we did and tracked with .lang files.

We brought other improvements to our maintenance scripts for example to be able to "bulk activate" a page for all the locales ready, we improved our locamotion import scripts, started adding unit tests etc. Generally speaking, the Web dashboard keeps improving regularly since I rewrote it last quarter and we regularly experiment using it for more projects, especially for projects which don't fit in the usual web/product categories and that also need tracking. I am pretty happy too that now I co-own the dashboard with Francesco who brings his own ideas and code to streamline our processes.

Théo's internship

I mentionned it before, our main French localizer Théo Chevalier, is doing an internship with me and Delphine Lebédel as mentors, this is the internship that ends his 3rd year of engineering (in a 5 years curriculum). He is based in Montain View, started early April and will be with us until late July.

He is basically working on almost all of the projects I, Delphine and Flod work on.

So far, apart from regular work as an l10n-driver, he has worked for me on 3 projects, the Web Dashboard projects view, building TMX files on demand on Transvision and the Firefox Nightly localized page on mozilla.org. This last project I haven't talked about yet and he blogged about it recently, in short, the first page that is shown to users of localized builds of Firefox Nightly can now be localized, and by localized we don't just mean translated, we mean that we have a community block managed by the local community proposing Nightly users to join their local team "on the ground". So far, we have this page in French, Italian, German and Czech, if your locale workflow is to translate mozilla-central first, this is a good tooll for you to reach a potential technical audience to grow your community .

Community

This quarter, I found 7 new localizers (2 French, 1 Marahati, 2 Portuguese/Portugal, 1 Greek, 1 Albanian) to work with me essentially on mozilla.org content. One of them, Nicolas Delebeque, took the lead on the Australis launch and coordinated the French l10n team since Théo, our locale leader for French, was starting his internship at Mozilla.

For Transvision, 4 people in the French community (after all, Transvision was created initially by them ;)) expressed interest or small patches to the project, maybe all the efforts we did in making the application easy to install and hack are starting to pay, we'll probably see in Q3/Q4 :)

I spent some time trying to help rebuild the Portugal community which is now 5 people (instead of 2 before), we recently resurrected the mozilla.pt domain name to actually point to a server, the MozFR one already hosting the French community and WoMoz (having the French community help the Portuguese one is cool BTW). A mailing list for Portugal was created (accessible also as nntp and via google groups) and the #mozilla-portugal IRC channel was created. This is a start, I hope to have time in Q3 to help launch a real Portugal site and help them grow beyond localization because I think that communities focused on only one activity have no room to grow or renew themselves (you also need coding, QA, events, marketing...).

I also started looking at Babelzilla new platform rewrite project to replace the current aging platform (https://github.com/BabelZilla/WTS/) to see if I can help Jürgen, the only Babelzilla dev, with building a community around his project. Maybe some of the experience I gained through Transvision will be transferable to Babelzilla (was a one man effort, now 4 people commit regularly out of 10 committers). We'll see in the next quarters if I can help somehow, I only had time to far to install the app locally.

In terms of events, this was a quiet quarter, apart from our l10n-drivers work week, the only localization event I was in was the localization sprint over a whole weekend in the Paris office. Clarista, the main organizer blogged about it in French, many thanks to her and the whole community that came over, it was very productive, we will definitely do it again and maybe make it a recurring event.

Summary

This quarter was a good balance between shipping, tooling and community building. The beginning of the quarter was really focused on shipping Australis and as usual with big releases, we created scripts and tools that will help us ship better and faster in the future. Tooling and in particular Transvision work which is probably now my main project, took most of my time in the second part of the quarter.

Community building was as usual a constant in my work, the one thing that I find more difficult now in this area is finding time for it in the evening/week end (when most potential volunteers are available for synchronous communication) basically because it conflicts with my family life a bit. I am trying to be more efficient recruiting using asynchronous communication tools (email, forums…) but as long as I can get 5 to 10 additional people per quarter to work with me, it should be fine with scaling our projects.

Pascal Chevrel — Progressively moving from Sublime Text 3 to Atom

I am currently moving from Sublime Text (I am a happy paying customer) to Atom. I love Sublime but unfortunately, the project is dying and I want to invest in tools that are maintained in the long term.

Sublime 3 is still not release after years, the only developper completely disappeared for months and this is not the first time it happens, this is also not an open source project which means that the project will die if this only developper leaves. The support forum is filled with spam and basically, there is no roadmap nor any official commitment to pursue the project.

Of course, Sublime still works fine, but there are already reports of it getting crashy on newer version of Linux, the addons ecosystem is slowing down and in the meantime, there is Atom created by the vibrant GitHub community with regular releases and attractive features (mainly clearly copied from Sublime).

So far I hadn't switched because Atom was not ready for prime time, despite its 1.0 number. It was just not usable on my machine for any serious dev work.

That said, the recent 1.2beta builds did get improvements and many extensions are actually working around core bugs that were making Atom a no go until now. So today, I am trying to use Atom 50% of the time instead of 10% as before.

So as to get a working setup, I did have to install a few must have extensions that just make the app usable. First and foremost, I needed Atom to work on my X1 carbon which boasts a 2560*1440 resolution and well, hiDPI is not Chrome Forte, I couldn't even read the microscopic UI and using CSS to fix that was a pain and not scalable when you switch your screen to a external monitor or for a presentation with a different resolution, one point for Sublime which just respects my OS DPI settings. Fortunately, this extension fixes it all:

HiDPI

The second thing I needed was decent speed, it's still lagging behind Sublime but the recent Atom builds over the summer did make a significant effort in that respect and the boot up time is now acceptable for my daily use.

The third problem which is probably still a blocker for me is that Atom basically sucks at managing non-US Qwerty keyboards, like seriously, it's bad. Most of the shortcuts didn't work on my French keyboard and this is the same problem for most keyboards in the world. Again this seems to be a Chrome engine limitation according to Github Issues and it should be fixed upstream in a few versions of Chrome from now. In the meantime, this package is an acceptable workaround that makes many non-US keyboards work more or less with Atom:

Keyboard Localization

There is one big caveat and one of my day to day blockers, I can't comment out a single line from a keyboard shortcut, if that simple bug were fixed in the package or Atom, I would certainly use it 75% of the time and not 50%.

In terms of UI and color scheme, it took me some time to find something agreable to the eyes (On Sublime, I tended to like Cobalt color scheme) but Atom Light for the UI and Code Dark for color scheme are rather agreable. Of course I am a PHP developper, so my color scheme tends towards having clear syntax in this language. The 1.2beta builds also stopped opening context menu in silly locations on my screen instead of below my cursor, another pain point gone.

The Zen and minimap extensions are just must haves for Sublime defectors like me:

Other extensions I use a lot are available:

In terms of PHP specific extensions, I found several equivalents to what I use in Sublime:

There are other PHP extensions I installed but didn't have much chance with them yet, some require configuration files, others don't seem to work or conflict with the keyboard shortcuts (stuff like php-cs-fixer).

The one extension I really miss from Sublime is Codeintel that autocompletes method names and gives tootips explaining methods from the data extracted from Dockblocks in classes. I really wish this will be ported some day or that an equivalent feature will be created.

I also need to see how to launch simple scripts on saving files, to launch Atoum unit tests for example, hopefully somebody will hack it and I won't have to dig into doing it myself ;)

On the plus side for Atom, Git/GitHub integration in the UI is obviously out of the box, I didn't have to install any extension for that. The UI is also improving regularly and just more pleasing to the eyes than Sublime's, which is a bit too geeky to my taste. There is a real preference panel where you can manage your extensions and many interesting small features for which you don't have to edit configuration files like in Sublime (seriously, setting your font size in a JSON file is not user friendly).

It does have its share of bugs though, for example color syntaxing seems to choke on very large files (like 2MB) and everything is displayed as text (not cool for XML or long PHP arrays for example). There are also sometimes lock ups of the UI, often when I switch to preferences.

But all in all, the experience is getting better over time and I think that I found a credible alternative to Sublime for 2016. I know there are other options, I actually have PHPStorm for example which is super powerful, but just as with Eclipse, Netbeans and other heavy-weight IDEs, I have better things to do in my life than spend 3 months just learning the tool to be able to edit a couple of files and I don't buy the argument that this is a life investment ;)

The one aspect of Atom that I think is still basic is project management, I need something just as simple as Sublime but I may just have overlooked the right extension for that and anyway, it's such a popular request that I have no doubt it will be in core at some point.

That's it, if you haven't tried Atom in a while and like me are worried about Sublime Text future, maybe that post will make you want to give it a try again.

I still love Sublime, even in its beta form it is a solid and complete product and I would be happy to keep on paying for it and get updates, but unfortunately, its days seem to be doomed because all the eggs are in the same basket of a single developer that may have good reasons to have vanished, but I just need work done and there are enough bus factor people in my own job to afford having also this problem with the main tool I use on a daily basis. I'd be happy to be proven wrong and see a Sublime Renaissance, the only dev is obviously incredibly talented and deserves to make a living out of his work, I just think he got overwelmed by the incredible success he had with his product and just can't keep up. At some point, open source or prioritary software, you need a team to scale up and meet your customers satisfaction. I do hope that if he completely gives up on this project to pursue other adventures, he will open source Sublime Text and not let all this coding beauty disappear in the prioritary world limbos forever :)

PS: And yes, I prepared this blog post in Markdown syntax thanks to the built-in Markdown previewing pane in Atom ;)

Pascal Chevrel — My Q4-2013 report

It's the end of the quarter, just list last quarter I wrote down a summary of what I did this quarter for Mozilla for posterity, here it is ;)

Tools and code

Once again, I spent significant time working on tools this quarter. My activity was focused on Transvision, Langchecker and my FirefoxOS minidashboard.

There were 2 releases of Transvision, 2.9 and 3.0, adding many new features and additional support for Gaia repositories. I also created a #transvision IRC channel on Mozilla IRC server. You can now search strings for 3 locales simultaneaously, check all existing translations for an entity, list all potentially wrong varables in your repo or quickly check  all strings that need some extra QA for Firefox OS.

There were also many improvements to langchecker, the tool that I now maintain with my colleague Francesco Lodolo to track and manage progress of translations for projects using our .lang format. Many views were improved and we added features specific to the metadata used on mozilla.org (page activation and home page promos). We also added checks for UTF8 validity of the source files as well as checks for broken or missing python-style replacement variables in translations. We can also know how much of our l10n user base we cover not only per page but also per string on a page, which allows us to decide when we can activate a minor but visible text change on our pages (recently for example, html meta description and title tag changed for SEO reasons on the Firefox download pages).

As for my FirefoxOS mini dashboard (which syndicates data from the l10n dashboard, the langchecker and also contains some manually maintained data sources), it followed Gaia progresses and now tracks Gaia 1.1 and 1.2 in addition to the master branch.

Community building

This quarter I found 8 new localizers for mozilla.org and key Firefox web parts for the following locales: Afrikaans, Arabic, Basque, Bulgarian, Czech, Slovak and Spanish (Mexico). As usual I tried to focus on helping teams that lack manpower and / or maintain Firefox for a minor language.

I also created / revived IRC channels for these locales to help community building:  Catalan (#mozilla-cat), Slovak (#mozilla-sk) and Serbian (#mozilla.sr).

If we can find 5 to 10 localizers working on key content every quarter, we shouldn't have any problem growing with all of our locales together in the years to come, the thing is that we have to be proactive and look for these people and not wait for them to come to us :),

Events

The only event I went to was the Mozilla Summit, it was great and very productive from a localization point of view, I worked with Dwayne who maintains Locamotion, the Pootle instance focused on Mozilla projects for minor locales (Firefox, mozilla.org content, Firefox OS, Firefox health report) and we worked on improving our collaboration. One of the immediate results this quarter is that now we automate imports of strings for mozilla.org from Locamotion, which takes us a few minutes of work per week  and is faster for both Dwayne and ourselves. We are working on how to make it easier for locales on Locamotion to also follow mozilla.org work as this happens at a much quicker pace than product localization.

I also talked and worked with many localizers either on their specific issues (for example Brano and Wlado from the Slovak team asked me for help finding new localizers) or to check what problems a locale has and how to fix them.

Mozilla.org

A lot of work happened on mozilla.org this quarter. The most visible one is that we now have the home page available in 55 languages, versus 28 at the end of the last quarter. This is a steadily growing number, I hope that we can get the page in 70 locales, the most important is of course maintenance over time. The home page received also several updates promoting various topics and we scaled out l10n work to cover that (end of year donation promo, lightbeam, webmaker, addons). The webdev team implemented a way for us (l10n-drivers) to manage the activation of promos on the home page without code changes thanks to metadata in our lang files, that allowed us to never show mixed language content on the home page and activate these promos for locales as they get done.

Key pages in the main menu (products, mission, about, contribute) are all localizable and localized to about the same scale as the main page. Other key download pages (firefox/channels and firefox/installer-help) pages are also now translated anf maintained at large scale. Lightbeam and State of Mozilla sections were proposed as opt-in projects and have many translations (between 15 and 20 languages). I am happy that we were able to scale the localization of mozilla.org in terms of content (more pages, more content for products), number of locales and locale-specific improvements on the mozilla.org platform (better rtl support, better fonts, l10n friendly templates..), and our management tools to allow us to grow.

Another cool l10n feature that happened on the site was the creation of a 'translation bar' proposing you a mozilla.org page in your language if it exists. This was entirely done by a volunteer Mozillian, Kohei Yoshino, many thanks to him for this cool feature that may expand to other Mozilla sites ! Kohei wrote about it on the webdev blog. It is really cool to see improvements brought by volunteers and it is also cool to see that many people in the webdev team are also acquiring an l10n culture and often spot potential problems before myself or flod get to them !

That's all for this quarter, overall an excellent quarter for mozilla.org and tools which improve steadily. On a side note, it is also an excellent quarter for me at a personal level  as my daughter was born last month (which explains why I am less often connected  these days ;) )

Pascal Chevrel — My Q3/Q4-2014 report

I didn't write the Q3 report, so this report covers both Q3 and Q4.

Regular l10n-drivers work such as patch reviews, pushes to production, meetings and past projects maintenance excluded.

Transvision

New release

We released version 3.5 of Transvision and I worked on these features and bug fixes:

  • Improved the setup process for devs, just launch start.sh after cloning the repository and it will set up everything and launch a Transvision instance locally. Setting up Transvision locally for development takes now about 20mn with a fast DSL connection.
  • Fixed a bunch of regressions in our external API that our Galician localizer Enrique Estévez, aka Keko, reported.
  • Worked with Keko on adding a new translation memory export format which allows importing an existing Mozilla localization effort supported by Transvision into OmegaT (an offline localization tool in Java).
  • Worked with my intern Théo on his Locale status view, our numbers are mostly consistent with the official l10n dashboard but with a different aggregation approach that will allow to expose strings by product and category (/ex, expose the status of devtools localization in Firefox, or group Firefox strings with Firefox content hosted on mozilla.org). Still a lot to do on the perf side for this page (http://transvision.mozfr.org/showrepos/) since it relies heavily on regexes on 10s of thousands of strings (page load is about 15s without caching, 4s with caching). Théo will do another internship next summer with us, so stay tuned.
  • Improved extraction of strings to not escape systematically special characters and html at string extraction. That allows searching for strings with html markup or reserved characters in HTML (for example: &BrandShortName or searching for \n codes in strings.
  • All commits to master are now continuously integrated on the beta site via GitHub webhooks (beta site had to be manually updated before)
  • Many smaller improvements to exiting views

Work in progress related to Transvision

  • Created a standalone library to be used initially in Transvision to extract all changesets from hg/git/svn repositories in the same format, so as to be able to dig contribution data in a uniform way
  • Working on an internal directory built dynamically from our logs, coupled with the vcs library it will allow us to provide our core localizers stats to the CBT team via an external API. Also works as a CLI tool for my own use.
  • Working on a Product class for Transvision that allows to query strings per software and not per repository (important for desktop because there are a lot of strings shared between Firefox and Thunderbird).

Overall, I worked a lot on Transvision during Q3 but very little during Q4. There was a good balance of improvements targetting our different users (localizers, Transvision devs, API consumers) and we did some solid improvements to both the code and our workflow to ship faster in 2015. We also had 8 people contributing to the codebase over the semester and several people popping in our IRC channel asking about the project, which is good. I have several big patches with good potential in branches but unfortunately I didn't have time to work on finishing them in Q4 and it seems I won't be able to work on them either in Q1, maybe in Q2.

Langchecker and other tracking tools we use for management

I found a way to generate data about web parts localization over time from Langchecker (Langchecker was not meant to do that, so I had to be creative), so now we generate graphs that show the history of web parts completion per locale: example with Bulgarian showing what happens when you have a new high performer localizer joining Mozilla

I updated the overview status showing the number of untranslated strings per locales with a summary ranking locales status and also showing locales on locamotion it also now counts strings for all projects on the wbedashboard, not just mozilla.org.

I added the number of strings, words and files to the summary view since this is something we got asked for often from other teams.

Google Play copy localization back into a normal l10n process

Historically, the process of publishing updates to the Firefox page on Google Play has been owned by many different people, most if not all of them no longer working for Mozilla. So there was no real formal process to update the copy and get that copy translated. Sylvestre Ledru (Debian core dev that recently joined the Mozilla Release Management team) decided to establish a process with as much automation as possible via the Google Play API to update our copy in English when we ship a new release and I decided to also help on this front to get that content localized and published simultaneaously with English.

So now the process of translating the copy is back to the l10n-drivers team with our own tools (means tracking, dashboards, integration to the translation platforms we use…).

I created a small app to track and QA translations without pushing to Google Play.

And I also created an associated JSON API that release drivers can use.

I am working with Sylvestre on getting release drivers tools automatically update our localized listings along with the en-US copy. We hope to get that working and functionnal in Q1 and be able to always have an updated copy for all of our supported locale for Firefox on Google Play from now on.

Thunderbird web localization migrating to Bedrock (mozilla.org)

This is localization management work that I am doing as a volunteer as Mozilla does not put paid resources on Thunderbird. My colleague Flod is also helping, as well as Théo (my intern and French localizer) and Kohei Yoshino (ex Mozilla Japan webdev and ex mozilla.org localizer, now living in Canada and still contributing to www.mozilla.org at the code level).

Thunderbird web content is still hosted on the old mozilla.org framework, this content is not actively maintained by anybody and that framework will eventually disappear. The l10n process is also broken there. For Thunderbird 10th anniversary, I volunteered to help manage the key content for Thunderbird on our current platform (Bedrock) so as that Thunderbird in-product pages can get updated and translated. If you use Thunderbird, you may have seen the new version of the start page (some people noticed).

A few notes though:

  • If you are a Mozilla localizer but are not interested in translating Thunderbird content, please don't translate it even if you see that on your locale dashboard, help us find a person to do these translations for Thunderbird in you language instead!
  • I am not volunteering to manage Thunderbird product localization, I just don't have enough time for that. If you want to help on that front, please contact the new Thunderbird team leaders, they will be happy to get help on that front!
  • 23 locales have translated the page, thanks to you guys! If your locale is not done and you want to get involved in Mozilla Web localization, please contact me! (pascal AT mozilla DOT com)

Community building

I try to spend as much time as I can actively finding new localizers and growing our localization community with core contributors that help us ship software and projects. It's great to build tools and we are always happy when we can improve by 20% the productivity of a localizer with a new process or tool, but when we find a new core localizer for a language, we get a lot more than a 20% productivity boost ;). After all, we build tools for people to use them.

Here is a list of countries where I found new core localizers:

Portugal:
  • We now have a webmaster for mozilla.pt (hosted on mozfr.org), Joao, a Portuguese volunteer involved in the Rust language
  • Luis Mourão joined the team as a technical localizer
  • Claudio Esperança, is now an svn committer
Bulgaria
  • New Bulgarian web localizer, Miroslav Yovchev. Miroslav did an awesome job with one of his friends getting mozilla.org which was very outdated in Bulgarian in a decent shape. He also corrected many existing pages for consistency of style.
  • I met with my colleague Pavel Ivanov in Portland who is Bulgarian and works on Gaia so as that we can see how we can help grow the Bulgarian community next year, not just in l10n.
Latvia

New Latvian localizer for webparts: Jānis Marks Gailis. Latvian does not have many speakers and historically we had only one localizer (Raivis Deijus) focused on products, so now we can say that we have a team spanning both products and web sites. A third person would be welcome of course :)

Croatia

New Croatian localizer Antun Koncic. Antun used to be the Croatian dictionary packager for Thunderbird and was no longer involved in Mozilla, welcome back then!

Poland
  • We have a new Polish localizer, Piotr Nalepa that is helping us with web content
  • Joanna Mazgaj, long term Polish contributor in aviary.pl is also now a committer for mozilla.org

We could use the help of a couple more people for Polish, especially to translate products and technical content, if you are interested, please leave a comment :)

Events I attended

I went to the Encontro ibérico de tradutores de software libre a linguas minorizadas in Santiago de Compostela, Spain. I met there several of our Mozilla localizers for Catalan, Galician, Basque, Aragonese and Asturian. We talked about processes, tools and community building. Was very interesting and great fun.

Like a thousand more Mozilla employees and many volunteers, I spent a work week in Portland, USA where I met several localizers we had invited. It was the first time I was meeting some of these localizers IRL, in particular people from India and Bangladesh. That was nice.

Privacy Coach Add-on

In parallel to the 10 years of Mozilla, we created and launched a Firefox for Android extension called Privacy Coach teaching users how to set up privacy settings in their browser.

I worked with Margaret Leibovic on that project and I reused the code and process that I had used for the add-ons we had created during the Australis launch earlier this year to translate the add-on while it was being developped. The idea is that it shouldn't be discruptive for either localizers or developers.

We worked with .lang files that were automatically converted to the right set of dtd/properties files for the extension along with the packaging of the locale into the extension (update of install.rdf/chrome.manifest/locale folder) that were integrated into the extenson via pull requests on github during its development. We got 12 locales done which is more than the 8 locales initially planned. Two locales we wanted to get we didn't in the end (Hungarian and Indonesian) so they are part of the locales I will work with in Q1 to get more people.

Various Web projects

  • Firefox Tiles: prepared the project as a .lang one with Mathjazz, prepopulated translations and launched with Mathjazz for all locales
  • Worked with flod on New home page and new contribute pages for mozilla.org
  • 10 years of Mozilla: Flod was the lead on this project and blogged about it in his own Q4 report, basically my main role there was to help find localizers for the project, all the merit of the project from the l10n POV goes to flod :)
  • Removed 9 locales no longer supported in our products from mozilla.org (ak, csb, lg, mn, nso, sah, sw, ta-LK, wo)

Other

  • I created a GitHub organization for our team and moved several of my projects there. I felt that we needed to have a place for us to have all the tools we use for shipping in one common place where we could create projects whenever we need them instead of using our personal accounts.
  • I transferred the womoz.org domain to Flore Allemandou now leading the Women in Mozilla community (I had been paying for it since 2009) and Flore is working on getting the domain owned and paid by Mozilla
  • I helped the French community (with Théo and Natalia from the Marketing team) set up a promotional Firefox OS site focused on end-users (created the github project, did a temporary version of the site until the French community built a design and content to replace it and took care of setting up some continous integration to allow people to publish code and text updates without me merging and pushing to production).

Pascal Chevrel — My Q1-2014 report

Here is what I did in Q1

Tools and code

This quarter we have made 2 releases of Transvision (3.1 and 3.2). For 3.1, most of the credit goes to my colleague (and Italian localizer) Francesco Lodolo who integrated his productization scripts into the app and worked on redesigning the menus and making the app better in responsive mode, my work on 3.1 was limited on performance patches and release management. Transvision 3.2 received a lot more of my code as it integrated one major new feature (planned since I took the project under my umbrella two years ago), which is indexing all translations of our flagship website www.mozilla.org. So now Transvision is no longer a search engine and QA tool for all of our products translations, it also supports websites using the .lang format (next up is indexing websites using gettext). From a developper perspective, this is also the release in which we automate the generation of the documentation of our classes thanks to phpDocumentor. Currently I am working on a better MVC approach and organization of code so as to work on features such as atom feeds, on the fly generation of TMX files and other more flexible export features, it may be in addition or in replacement of the TMX files we generate at extraction time. Caching is also in the pipe, the app is currently very fast and responsive but caching could help us extend more our external API to more time consuming data generation.

A personal achievement is that I finally rewrote my old Web Dashboard (whose code dated back to 2007) and moved that to github. Two new features, direct query of web localization bugs per locale on Bugzilla and display of the translation status for most of the projects hosted on Verbatim, our web localization platform, the latter was a patch by Francesco. The big feature is that the code went from 6000 lines of spaghetti code to 400 clean code, old features that made sense 7 years ago and now useless were removed and the fact that we can now track www.mozilla.org work in Bugzilla per locale allowed cutting a lot of code. Pretty happy that this was totally transparent to the user and that we are already starting to add new features to help us ship software!

On the Langchecker side, the tool we use to track .lang based projects and deploy lang files on repositories, I added per locale json feeds for the status of the locale and it now warns about invalid metadata and can now removes obsolete or invalid tags that we use on mozilla.org to activate a feature for a locale on a page. We also used it to expose strings from our first project based on Pontoon.

Other small tools of mine got updated, my mass bug filing script can now file bugs marked as mozilla confidential, something we needed for Mobile World Congress pages, I whipped up a script using Transvision data to know the distribution of strings across components in Firefox/FirefoxOS which allows specifically to know how many strings we have for devtools (potentially a future Transvision feature) etc.

All in all, I'd say I spent about 30% of my time this quarter hacking on code, which is cool.

Events and community

This quarter I attended one event outside of Paris, Fosdem. As usual it was interesting in terms of networking with European open source hacktivists. I also organized a one day event in the Paris office with April which mostly consisted of workshops on how to contribute to an open source project (I led two workshops on participating to Mozilla localization). I also attended 3 events in the Paris office, one with the French community to welcome half a dozen new volunteers, one with Rosana and Goofy dedicated  to SUMO localization and a last one which was an impromptu meeting between the French community and Brian King from Reps. With a Paris office tailored for meetings, I expect that I will continue to organize / attend to community meetings locally instead of travelling a lot like in the past, that's more compatible with my recent fatherhood :)

This wasn't a big quarter in terms of finding contributors, I can identify only 2 new people that became active and productive in the Mozilla l10n project directly because of me, that was mostly due to the fact that I spent more time on shipping projects and building tools this quarter. I also had some more paperwork than usual to deal with as I have an intern starting in April and I also worked with my colleagues on a tax refund paper for Research and Development in the Paris office.

I spent some specific time helping the Lithuanian community trying to grow in general, we created an IRC channel, opened a mailing list for Lithuania, did some work on etherpads  and contacted other open source projects in Lithuanians, past contributors, would be contributors through the Contribute page... We found some people interested in helping with l10n but more work will be needed to reboot the Lithuanian community (not just in l10n).

I also closed the mozilla.community.europe which had been created after we closed Mozilla Europe and merged its activities with Mozilla, it unfortunately never attracted any activity and was just collecting spam (which I had to delete on a daily basis).

About the Transvision community, the project reached 9 committers in March, is now listed on What Can I Do for Mozilla (which makes it the only active PHP project listed on the site ;)). It is growing slowly but steadily on all fronts (uses, codebase, contributors, features...). We are also working on making it easier to hack (static and automatic documentation, better installation scripts...) but one of the roadblock is the amount of data that needs to be downloaded first to make it useful locally, the sum of our hg/git/svn repos are about 17GB big. Even after extraction of strings from all of our repos, it is probably like 200MB of text, we may evaluate a downloadable snapshot of that in the install process to make it easier for contributors that want to hack on it.

From a tooling community perspective, big thanks to Enrique Estevez, our galician localizer, who is working on upstreaming patches to OmegaT to make it work better with Mozilla formats (.properties and .lang) and who is also writing an Omega T plugin to get translation suggestions from Transvision. Babelzilla also contacted me for similar purposes, that means that at least four Mozilla tools use it or plan to use it directly (Mozilla Translator, OmegaT, Pontoon, Babelzilla). We will put a focus on improving, documenting and versionning our external API this year to make it easier to the tools community to integrate it.

Mozilla.org localization

On Mozilla.org, the big deal this year was like last year the Mobile World Congress event, we updated all of our Firefox OS pages and home page + tabzilla in 17 languages in time before the event and also added some extra content for a dozen of these locales. As usual with big projects, we exposed feature needs and we now can publish promotions in tabzilla per locale, using the same mecanism as for the home page, so as to target only the languages that we were working on for MWC.

MWC was not all though, we also worked on a new Pricacy page for a few locales, we worked on a landing page for Firefox Metro, but unfortunately the product got cancelled, we updated a lot of older content, made patches in our templates to avoid duplicated strings, shipped new locales on the release channel with all o their web content done (Xhosa and Aragonese) and shipped Malay on the Beta and Aurora channels too.

The amount of strings is now around 1500 for a big locale on mozilla.org and about 600 for a small locale. Of course, since we have frequent content changes (promos, minor string updates on tabzilla, download pages, in-product pages...), this is actually more, especially for the bigger locales and Firefox OS locales, but I am happy that we succeed in keeping all of the locales in good shape, especially the small ones, while not slowing down the creation of English content. The site is definitely getting better and more localized month after month, two years after the switch to Django and the total rewrite, this is starting to look really good again.

And now Q2!

Expect Q2 to be more of the same :) One interesting point is that I am taking Theo Chevalier, our main French localizer, for an internship at Mozilla, which means that he will work on all of the cool stuff I have in mind but never get the time to do (like working on growing our l10n Firefox Nightly userbase which is ridiculously low compared to en-US), he will also of course work on the big Q2 project that is the launch of Firefox Australis!

Karl Dubost — Filter your mail in a dated space server side with Sieve

When it comes to sort out your emails, there are many strategies. Since I have been working at W3C, I'm a fan of dated spaces. I apply this strategy to my emails using Sieve.

mailbox stuffed with old letters and papers.

Dated Space?

A dated space is a way to organize the information by date folders. Here is an example on my folders.

organization by dates of folders.

and here the same type of organization for mails.

organization by dates of folders.

Why would you do that?

  • It creates a unique space. The arrow time is irreversible so there is no issue with creating new folders.
  • It limits the number of items by folders. When a folder has too many items, it becomes harder to find what you need.
  • It is actually helpful, because our memory has a lot of anchors in time and we can find easily stuff by remembering when we created it. (a location-based could be an interesting experiment and has useful applications in some circumstances such as photos for example, but that's for another blog post.)
  • OS search engines: Smart folders based on keywords, types, etc. will make it easier to find stuff. And the files can now belong in multiple contexts. Everything everywhere all at once.

Dated Space For Mail With Sieve

Sieve is a language for filtering e-mail messages. It resides on the server side. So it requires that your mail provider accepts that you can manage sieve rules for your own mail.

I filter all my incoming mails indifferently of their nature to a dated space. Everything is mixed. And I restructure my emails folders with smart mailboxes. Then I can suppress folders which are not valid anymore, create new ones, all of this without touching any emails.

Sieve Script

require ["fileinto", "date", "mailbox", "variables"];
# filter emails based on the date and move them to a folder.
# mail from June 2022, will be saved into /2022/06
# set values for month and year
if currentdate :matches "year" "*" { set "year" "${1}"; };
if currentdate :matches "month" "*" { set "month" "${1}"; };
# mv the message into the right mail folder
fileinto :create "${year}/${month}";
stop;

It's a very simple script and works like a charm. Sieve is a very powerful language. It's possible to do all kind of classifications.

Oh… and yes… my Inbox is obviously by the nature of the script always… 0.

Otsukare!

Frédéric Wang — Update on OpenType MATH fonts

I mentioned in a previous post that Igalia organized the Web Engines Hackfest 2022 last week. As usual, fonts were one of the topic discussed. Dominik Röttsches presented COLRv1 color vector fonts in Chrome and OSS (transcript) and we also settled a breakout session on Tuesday morning. Because one issue raised was the availability of OpenType MATH fonts on operating systems, I believe it’s worth giving an update on the latest status…

There are only a few fonts with an OpenType MATH table. Such fonts can be used for math layout e.g. modern TeX engines to render LaTeX, Microsoft Office to render OMML or Web engines to render MathML. Three of such fonts are interesting to consider, so I’m providing a quick overview together with screenshots generated by XeTeX from the LaTeX formula $${\sqrt{\sum_{n=1}^\infty {\frac{10}{n^4}}}} = {\int_0^\infty \frac{2x dx}{e^x-1}} = \frac{\pi^2}{3} \in {\mathbb R}$$:

Recently, Igalia has been in touch with Myles C. Maxfield who has helped with internal discussion at Apple regarding inclusion of STIX Two Math in the list of fonts on macOS. Last week he came back to us announcing it’s now the case on all the betas of macOS 13 Ventura 🎉 ! I just tested it this morning and indeed STIX Two Math is now showing up as expected in the Font Book. Here is the rendering of the last slide of my hackfest presentation in Safari 16.0:

Screenshot of a math formula rendered with STIX Two Math by Safari

Obviously, this is a good news for Chromium and Firefox too. For the former, we are preparing our MathML intent-to-ship and having decent rendering on macOS by default is important. As for the latter, we could in the future finally get rid of hardcoded tables to support the deprecated STIXGeneral set.

Another interesting platform to consider for Chromium is Android. Last week, there has been new activity on the Noto fonts bug and answers seem more encouraging now… So let’s hope we can get a proper math font on Android soon!

Finally, I’m not exactly sure about the situation on Linux and it may be different for the various distributions. STIX and Latin Modern should generally be available as system packages that can be easily installed. It would be nicer if they were pre-installed by default though…

Planet Mozilla

Collected here are the most recent blog posts from all over the Mozilla community. The content here is unfiltered and uncensored, and represents the views of individual community members. Individual posts are owned by their authors -- see original source for licensing information.

Subscribe to Planet

Feeds:

Subscription list:

Last update:

Other Planets

Subscriptions