Shallow Thoughts : : Jun
Akkana's Musings on Open Source Computing and Technology, Science, and Nature.
Sat, 23 Jun 2018
My article on
key bindings in Firefox Quantum by modifying the source tree got
attention from several people who offered helpful suggestions via
Twitter and email on how
to accomplish the same thing using just files in omni.ja,
so it could be done without rebuilding the Firefox source.
That would be vastly better, especially for people who need to
change something like key bindings or browser messages but don't
have a souped-up development machine to build the whole browser.
Brian Carpenter had several suggestions and eventually pointed me to
an old post by Mike Kaply,
Don’t Unpack and Repack omni.ja[r]
that said there were better ways to override specific files.
Unfortunately, Mike Kaply responded that that article was written
for XUL extensions, which are now obsolete, so the article
ought to be removed. That's too bad, because it did sound like a much
nicer solution. I looked into trying it anyway, but the instructions
it points to for
specific files is woefully short on detail on how to map a path
inside omni.ja like chrome://package/type/original-uri.whatever,
to a URL, and the single example I could find was so old that the file
it referenced didn't exist at the same location any more. After a fruitless
half hour or so, I took Mike's warning to heart and decided it wasn't
worth wasting more time chasing something that wasn't expected to work
anyway. (If someone knows otherwise, please let me know!)
But then Paul Wise offered a solution that actually worked, as an
easy to follow sequence of shell commands. (I've changed some of
them very slightly.)
$ tar xf ~/Tarballs/firefox-60.0.2.tar.bz2
# (This creates a "firefox" directory inside the current one.)
$ mkdir omni
$ cd omni
$ unzip -q ../firefox/browser/omni.ja
warning [../firefox-60.0.2/browser/omni.ja]: 34187320 extra bytes at beginning or within zipfile
(attempting to process anyway)
error [../firefox-60.0.2/browser/omni.ja]: reported length of central directory is
-34187320 bytes too long (Atari STZip zipfile? J.H.Holm ZIPSPLIT 1.1
zsh: exit 2 unzip -q ../firefox-60.0.2/browser/omni.ja
$ sed -i 's/or enter address/or just twiddle your thumbs/' chrome/en-US/locale/browser/browser.dtd chrome/en-US/locale/browser/browser.properties
I was a little put off by all the warnings unzip gave, but kept going.
Of course, you can just edit those two files rather than using sed;
but the sed command was Paul's way of being very specific about
the changes he was suggesting, which I appreciated.
Use these flags to repackage omni.ja:
$ zip -qr9XD ../omni.ja *
I had tried that before (without the q since I like to see what zip
and tar commands are doing) and hadn't succeeded. And indeed, when I
listed the two files, the new omni.ja I'd just packaged was
about a third the size of the original:
$ ls -l ../omni.ja ../firefox-60.0.2/browser/omni.ja
-rw-r--r-- 1 akkana akkana 34469045 Jun 5 12:14 ../firefox/browser/omni.ja
-rw-r--r-- 1 akkana akkana 11828315 Jun 17 10:37 ../omni.ja
But still, it's worth a try:
$ cp ../omni.ja ../firefox/browser/omni.ja
Then run the new Firefox. I have a spare profile I keep around for
testing, but Paul's instructions included a nifty way of running with
a brand new profile and it's definitely worth knowing:
$ cd ../firefox
$ MOZILLA_DISABLE_PLUGINS=1 ./firefox -safe-mode -no-remote -profile $(mktemp -d tmp-firefox-profile-XXXXXXXXXX) -offline about:blank
Also note the flags like safe-mode and no-remote, plus disabling
plugins -- all good ideas when testing something new.
And it worked! When I started up, I got the new message, "Search or
just twiddle your thumbs", in the URL bar.
Of course, now I had to test it with my real change. Since I like Paul's
way of using sed to specify exactly what changes to make, here's a
sed version of my Ctrl-W fix:
$ sed -i '/key_close/s/ reserved="true"//' chrome/browser/content/browser/browser.xul
Then run it. To test Ctrl-W, you need a website that includes a text
field you can type in, so -offline isn't an option unless you
happen to have a local web page that includes some text fields.
Google is an easy way to test ... and you might as well re-use that
firefox profile you just made rather than making another one:
$ MOZILLA_DISABLE_PLUGINS=1 ./firefox -safe-mode -no-remote -profile tmp-firefox-profile-* https://google.com
I typed a few words in the google search field that came up, deleted
them with Ctrl-W -- all was good! Thanks, Paul! And Brian, and
everybody else who sent suggestions.
Why are the sizes so different?
I was still puzzled by that threefold difference in size between the
omni.ja I repacked and the original that comes with Firefox.
Was something missing? Paul had the key to that too: use zipinfo
on both versions of the file to see what differed. Turned out
Mozilla's version, after a long file listing, ends with
2650 files, 33947999 bytes uncompressed, 33947999 bytes compressed: 0.0%
while my re-packaged version ends with
2650 files, 33947969 bytes uncompressed, 11307294 bytes compressed: 66.7%
So apparently Mozilla's omni.ja is using no compression at all.
It may be that that makes it start up a little faster; but Quantum
takes so long to start up that any slight difference in uncompressing
omni.ja isn't noticable to me.
I was able to run through this whole procedure on my poor slow netbook,
the one where building Firefox took something like 15 hours ... and
in a few minutes I had a working modified Firefox. And with the sed
command, this is all scriptable, so it'll be easy to re-do whenever
Firefox has a security update. Win!
Update: I have a simple shell script to do this:
Script to modify omni.ja for a custom Firefox.
[ 20:37 Jun 23, 2018
More tech/web |
permalink to this entry |
Thu, 14 Jun 2018
When I first tried switching to Firefox Quantum,
the regression that bothered me most was
Ctrl-W, which I use everywhere as word erase (try it -- you'll get
addicted, like I am). Ctrl-W deletes words in the URL bar;
but if you type Ctrl-W in a text field on a website,
like when editing a bug report or a "Contact" form,
it closes the current tab, losing everything you've just typed.
It's always worked in Firefox in the past; this is a new problem
with Quantum, and after losing a page of typing for about the
20th time, I was ready to give up and find another browser.
A web search found plenty of people online asking about key bindings
like Ctrl-W, but apparently since the deprecation of XUL and XBL
extensions, Quantum no longer offers any way to change or even
just to disable its built-in key bindings.
I wasted a few days chasing a solution inspired by this
of remapping keys only for certain windows using
xdotool getactivewindow; I even went so far as to
write a Python script that intercepts keystrokes, determines the
application for the window where the key was typed, and remaps it if
the application and keystroke match a list of keys to be remapped.
So if Ctrl-W is typed in a Firefox window, Firefox will instead
receive Alt-Backspace. (Why not just type Alt-Backspace, you ask?
Because it's much harder to type, can't be typed from the home
position, and isn't in the same place on every keyboard the way
But sadly, that approach didn't work because it turned out my window
manager, Openbox, acts on programmatically-generated key bindings as
well as ones that are actually typed. If I type a Ctrl-W and it's in
Firefox, that's fine: my Python program sees it, generates an
Alt-Backspace and everything is groovy.
But if I type a Ctrl-W in any other application, the program
doesn't need to change it, so it generates a Ctrl-W, which Openbox
sees and calls the program again, and you have an infinite loop.
I couldn't find any way around this. And admittedly, it's a horrible
hack having a program intercept every keystroke. So I needed to fix
But after spending days searching for a way to customize Firefox's
keys, to no avail, I came to the conclusion that the only way was to
modify the source code and
Firefox from source.
Ironically, one of the snags I hit in building it was that
I'd named my key remapper "pykey.py", and it was still in my
PYTHONPATH; it turns out the Firefox build also has a module called
pykey.py and mine was interfering. But eventually I got the build working.
Firefox Key Bindings
I was lucky: building was the only hard part, because
a very helpful person on Mozilla's #introduction IRC channel
pointed me toward the solution, saving me hours of debugging. Edit
around line 240
reserved="true" from key_closeWindow.
It turned out I needed to remove
the adjacent key_close line as well.
Another file that's related, but more general, is
around line 832; but I didn't need that since the simpler fix worked.
Transferring omni.ja -- or Not
In theory, since browser-sets.inc isn't compiled C++, it seems
like you should be able to make this fix without building the whole
source tree. In an actual Firefox release,
browser-sets.inc is part of omni.ja, and indeed if
you unpack omni.ja you'll see the key_closeWindow and
key_close lines. So it seems like you ought to be able to
regenerate omni.ja without rebuilding all the C++ code.
Unfortunately, in practice omni.ja is more complicated than that.
Although you can unzip it and edit the files, if you zip it back up,
Firefox doesn't see it as valid. I guess that's why they renamed it
.ja: long ago it used to be omni.jar and, like other
.jar files, was a standard zip archive that you could edit.
But the new .ja file isn't documented anywhere I could find,
and all the web discussions I found on how to re-create it amounted
to "it's complicated, you probably don't want to try".
And you'd think that I could take the omni.ja file from my
desktop machine, where I built Firefox, and copy it to my laptop,
replacing the omni.ja file from a released copy of Firefox.
But no -- somehow, it isn't seen, and the old key bindings are still
active. They must be duplicated somewhere else, and I haven't figured
It sure would be nice to have a way to transfer an omni.ja.
Building Firefox on my laptop takes nearly a full day (though
hopefully rebuilding after pulling minor security updates won't be
quite so bad). If anyone knows of a way, please let me know!
[ 16:45 Jun 14, 2018
More tech/web |
permalink to this entry |
Sat, 09 Jun 2018
I did the work to
my own Firefox primarily to fix a couple of serious regressions
that couldn't be fixed any other way. I'll start with the one that's
probably more common (at least, there are many people complaining
about it in many different web forums): the fact that Firefox won't
play sound on Linux machines that don't use PulseAudio.
There's a bug with a long discussion of the problem,
Bug 1345661 - PulseAudio requirement breaks Firefox on ALSA-only systems;
and the discussion in the bug links to another
of the Firefox/PulseAudio problem). Some comments in those
discussions suggest that some near-future version of Firefox may
restore ALSA sound for non-Pulse systems; but most of those comments
are six months old, yet it's still not fixed in the version Mozilla
is distributing now.
In theory, ALSA sound is easy to enable.
pptions in Firefox are controlled through a file called mozconfig.
Create that file at the top level of your build directory, then add to it:
You can see other options with
Of course, like everything else in the computer world, there were
complications. When I typed
mach build, I got:
Assertion failed in _parse_loader_output:
Traceback (most recent call last):
File "/home/akkana/outsrc/gecko-dev/python/mozbuild/mozbuild/mozconfig.py", line 260, in read_mozconfig
parsed = self._parse_loader_output(output)
File "/home/akkana/outsrc/gecko-dev/python/mozbuild/mozbuild/mozconfig.py", line 375, in _parse_loader_output
assert not in_variable
Error loading mozconfig: /home/akkana/outsrc/gecko-dev/mozconfig
Evaluation of your mozconfig produced unexpected output. This could be
triggered by a command inside your mozconfig failing or producing some warnings
or error messages. Please change your mozconfig to not error and/or to catch
errors in executed commands.
... followed by a many-page dump of all my environment variables, twice.
It turned out that was coming from line 449 of
# Lines with a quote not ending in a quote are multi-line.
if has_quote and not value.endswith("'"):
in_variable = name
value = value[:-1] if has_quote else value
I'm guessing this was added because some Mozilla developer sets
a multi-line environment variable that has a quote in it but doesn't
end with a quote. Or something. Anyway, some fairly specific case.
I, on the other hand, have a different specific case: a short
environment variable that includes one or more single quotes,
and the test for their specific case breaks my build.
(In case you're curious why I have quotes in an environment variable:
The prompt-setting code in my .zshrc includes a variable called
PRIMES. In a login shell, this is set to the empty string,
but in subshells, I add ' for each level of shell under the login shell.
So my regular prompt might be (hostname)-, but if I run a
subshell to test something, the prompt will be (hostname')-,
a subshell inside that will be (hostname'')-, and so on.
It's a reminder that I'm still in a subshell and need to exit when
I'm done testing. In theory, I could do that with SHLVL, but SHLVL
doesn't care about login shells, so my normal shells inside X are
all SHLVL=2 while shells on a console or from an ssh are
SHLVL=1, so if I used SHLVL I'd have to have some special case
code to deal with that.
Also, of course I could use a character other than a single-quote.
But in the thirty or so years I've used this, Firefox is the first
program that's ever had a problem with it. And apparently I'm not the
first one to have a problem with this:
was apparently someone else with the same problem. Maybe that will show
up in the release branch eventually.)
Anyway, disabling that line fixed the problem:
# Lines with a quote not ending in a quote are multi-line.
if False and has_quote and not value.endswith("'"):
and after that,
succeeded, I built a new
Firefox, and lo and behond! I can play sound in YouTube videos and
on Xeno-Canto again, without needing an additional browser.
[ 16:49 Jun 09, 2018
More tech/web |
permalink to this entry |
Sun, 03 Jun 2018
With Firefox Quantum, Mozilla has moved away from
letting users configure the browser they way they like.
If I was going to switch to Quantum as my everyday browser,
there were several problems I needed to fix first -- and they all
now require modifying the source code, then building the whole browser
I'll write separately about fixing the specific problems; but first
I had to build Firefox. Although I was a Firefox developer way back in
the day, the development environment has changed completely since then,
so I might as well have been starting from scratch.
Setting up a Firefox build
I started with Mozilla's
build preparation page. There's a script
called bootstrap.py that's amazingly comprehensive. It will
check what's installed on your machine and install what's needed for a
Firefox build -- and believe me, there are a lot of dependencies.
Don't take the "quick" part of the "quick and easy" comment at the
beginning of the script too seriously; I think on my machine, which
already has a fairly full set of build tools, the script was
downloading additional dependencies for 45 minutes or so. But it
was indeed fairly easy: the script asks lots of questions about
optional dependencies, and usually has suggestions, which I mostly
Eventually bootstrap.py finishes loading the dependencies and
gets to the point of wanting to check out the mozilla-unified
repository, and that's where I got into trouble.
The script wants to check out the bleeding edge tip of Mozilla development.
That's what you want if you're a developer checking in to the project.
What I wanted was a copy of the currently released Firefox, but with
a chance to make my own customizations. And that turns out to be difficult.
Getting a copy of the release tree
In theory, once you've checked out mozilla-unified with Mercurial,
assuming you let bootstrap.py enable the recommended "firefoxtree"
hg extension (which I did), you can switch to the release branch
hg pull release
hg up -c release
That didn't work for me: I tried it numerous times over the course of
the day, and every time it died with
"abort: HTTP request error (incomplete response; expected 5328 bytes
got 2672)" after "adding manifests" when it started "adding file changes".
That sent me on a long quest aided by someone in Mozilla's helpful
#introduction channel, where they help people with build issues.
You might think it would be a common thing to want to build a copy of
the released version of Firefox, and update it when a new release
comes out. But apparently not; although #introduction is a friendly
and helpful channel, everyone seemed baffled as to why
didn't work and what the possible alternatives might be.
Bundles and Artifacts
Eventually someone pointed me to the long list of
tarballs and advised me on how to get a release tarball there.
I actually did that, and (skipping ahead briefly) it built and ran;
but I later discovered that "bundles"
aren't actually hg repositories and can't be updated. So once you've
downloaded your 3 gigabytes or so of Mozilla stuff and built it, it's
only good for a week or two until the next Mozilla release, when
you're now hopelessly out of date and have to download a whole nuther
bundle. Bundles definitely aren't the answer, and they aren't well
supported or documented either. I recommend staying away from them.
I should also mention "artifact builds". These sound like a great
idea: a lot of the source is already built for you, so you just build
a little bit of it. However, artifact builds are only available for a
few platforms and branches. If your OS differs in any way from whoever
made the artifact build, or if you're requesting a branch, you're
likely to waste a lot of time (like I did) downloading stuff only to
get mysterious error messages. And even if it works, you won't be able
to update it to keep on top of security fixes. Doesn't seem like a
GitHub to the rescue
Okay, so Mercurial's branch switching doesn't work.
But it turns out you don't have to use Mercurial. There's a
GitHub mirror for Firefox
called gecko-dev, and after cloning it you can use normal
git commands to switch branches:
git clone https://github.com/mozilla/gecko-dev.git
git checkout -t origin/release
You can verify you're on the right branch with
git branch -vv, or if you want to list all branches
and their remotes,
git branch -avv.
Finally: a Firefox release branch that you can actually update!
Once you have a source tree, you can use the all-powerful mach
script to build the current release of Firefox:
Of course that takes forever -- hours and hours, depending on how
fast your machine is.
Running your New Firefox
The build, after it finishes, helpfully tells you to test it with
./mach run, which runs your newly-built firefox with a
special profile, so it doesn't interfere with your running build.
It also prints:
For more information on what to do now, see https://developer.mozilla.org/docs/Developer_Guide/So_You_Just_Built_Firefox
Great! Except there's no information there on how to package or run
your build -- it's just a placeholder page asking people to contribute
to the page.
It turns out that obj-whatever/dist/bin is the directory that
corresponds to the tarball you download from Mozilla, and you can run
/path/to/mozilla-release/obj-whatever/dist/bin/firefox from anywhere.
I tried filing a bug request to have a sub-page created explaining how
to run a newly built Firefox, but it hasn't gotten any response.
Maybe I'll just edit the "So You Just Built" page.
Update, 7 months later:
Nobody ever did respond to my bug, but someone on Mozilla's #introduction
channel for help with builds gave me a blessing to modify the page directly,
which I did.
Specifically, I added:
Your new Firefox executable can be found in: $OBJDIR/dist/bin/firefox. You can run it from there.
If you need to move it, e.g. to another machine, you can run:
This should create an OS-specific package, e.g. a tarball on Linux, which will appear in $OBJDIR/dist. You can also copy the $OBJDIR/dist/bin directory -- be sure to use a copy method that expands soft links -- but the result will be much larger than what you get with mach package.
On Windows you may want to read about
Windows Installer Builds.
Incidentally, my gecko-dev build takes 16G of disk space, of which
9.3G is things it built, which are helpfully segregated in
[ 15:55 Jun 03, 2018
More tech/web |
permalink to this entry |