Shallow Thoughts : : tech
Akkana's Musings on Open Source Computing and Technology, Science, and Nature.
Tue, 12 Feb 2019
A while back, Dave ordered a weather station.
His research pointed to the
Ambient Weather WS-2000 as the best bang for the buck as far as accuracy
(after it's calibrated, which is a time consuming and exacting process
where you compare readings to a known-good mercury thermometer, a process
that I suspect most weather station owners don't bother with).
It comes with a little 7" display console that sits indoors and
reads the radio signal from the outside station as well as a second
thermometer inside, then displays all the current weather data.
It also uses wi-fi to report the data upstream to Ambient and,
optionally, to a weather site such as Wunderground.
(Which we did for a while, but now Wunderground is closing off
their public API, so why give them data if they're not going to
make it easy to share it?)
Having the console readout and the Ambient "dashboard" is all very
nice, but of course, being a data geek, I wanted a way to get the data
myself, so I could plot it, save it or otherwise process it. And
that's where Ambient falls short. The console, though it's already
talking on wi-fi, gives you no way to get the data. They sell a
separate unit called an "Observer" that provides a web page you
can scrape, and we actually ordered one, but it turned out to be
buggy and difficult to use, giving numbers that were substantially
different from what the console showed, and randomly failing to answer,
and we ended up returning the observer for a refund.
The other way of getting the data is online. Ambient provides an API
you can use for that purpose, if you email them for a key. It
mostly works, but it sometimes lags considerably behind real time, and
it seems crazy to have to beg for a key and then get data from a
company website that originated in our own backyard.
What I really wanted to do was read the signal from the weather
station directly. I'd planned for ages to look into how to do that,
but I'm a complete newbie to software defined radio and wasn't
sure where to start. Then one day I noticed an SDR discussion
on the #raspberrypi IRC channel on Freenode where I often hang out.
I jumped in, asked some questions, and got pointed in the right direction
and referred to the friendly and helpful #rtlsdr Freenode channel.
An Inexpensive SDR Dongle
On the experts' advice, I ordered a
Blog R820T2 RTL2832U 1PPM TCXO SMA Software Defined Radio with 2x
Telescopic Antennas on Amazon. This dongle apparently has better
temperature compensation than cheaper alternatives, it came with
a couple of different antenna options, and I was told it should
work well with Linux using a program called
Indeed it did. The command to monitor the weather station is
rtl_433 -f 915M
rtl_433 already knows the protocol for the WS-2000,
so I didn't even need to do any decoding or reverse engineering;
it produces a running account of the periodic signals being
broadcast from the station. rtl_433 also helpfully offers
-F csv options, along with a few other formats.
What a great program!
JSON turned out to be the easiest for me to use; initially I thought
CSV would be more compact, but rtl_433's CSV format includes fields
for every possible quantity a weather station could ever broadcast.
When you think about it, that makes sense: once you're outputting
CSV you can't add a new field in mid-stream, so you'd better be
ready for anything. JSON, on the other hand, lets you report
just whatever the weather station reports, and it's easy to parse
from nearly any programming language.
Testing the SDR Dongle
Full disclosure: at first,
rtl_433 -f 915M wasn't showing
me anything and I still don't know why. Maybe I had a loose connection
on the antenna, or maybe I got unlucky and the weather station picked
the exact wrong time to take a vacation. But while I was testing,
I found another program that was very helpful in testing whether
my SDR dongle was working: rtl_fm, which plays radio stations.
The only trick is finding the right arguments,
since the example from the man page just played static.
Here's what worked for me:
rtl_fm -f 101.1M -M fm -g 20 -s 200k -A fast -r 32k -l 0 -E deemp | play -r 32k -t raw -e s -b 16 -c 1 -V1 -
That command plays the 101.1 FM radio station. (I had to do a web search
to give me some frequencies of local radio stations; it's been
a long time since I listened to normal radio.)
Once I knew the dongle was working, I needed to verify what frequency
the weather station was using for its broadcasts.
What I really wanted was something that would scan frequencies around
915M and tell me what it found. Everyone kept pointing me to a program
called Gqrx. But it turns out Gqrx on Linux requires PulseAudio and
absolutely refuses to work or install without it, even if you have no
interest in playing audio. I didn't want to break my system's sound
(I've never managed to get sound working reliably under PulseAudio),
and although it's supposedly possible to build Gqrx without Pulse,
it's a difficult build: I saw plenty of horror stories, and it
requires Boost, always a sign that a build will be problematic.
I fiddled with it a little but decided it wasn't a good time investment.
I eventually found a scanner that worked:
It let me set limiting frequencies and scan between them, and by
setting it to accumulate, I was able to verify that indeed, the
weather station (or something else!) was sending a signal on 915 MHz.
I guess by then, the original problem had fixed itself, and after that,
rtl_433 started showing me signals from the weather station.
It's not super polished, but it's the only scanner I've found that
works without requiring PulseAudio.
That Puzzling Rainfall Number
One mystery remained to be solved. The JSON I was getting from the
weather station looked like this (I've reformatted it for readablility):
"time" : "2019-01-11 11:50:12",
"model" : "Fine Offset WH65B",
"id" : 60,
"temperature_C" : 2.200,
"humidity" : 94,
"wind_dir_deg" : 316,
"wind_speed_ms" : 0.064,
"gust_speed_ms" : 0.510,
"rainfall_mm" : 90.678,
"uv" : 324,
"uvi" : 0,
"light_lux" : 19344.000,
"battery" : "OK",
"mic" : "CRC"
This on a day when it hadn't rained in ages. What was up with that
"rainfall_mm" : 90.678 ?
I asked on the rtl_433 list and got a prompt and helpful answer:
it's a cumulative number, since some unspecified time in the past
(possibly the last time the battery was changed?) So as long as
I make a note of the rainfall_mm number, any change in
that number means new rainfall.
This being a snowy winter, I haven't been able to test that yet:
the WS-2000 doesn't measure snowfall unless some snow happens to melt
in the rain cup.
Some of the other numbers, like uv and uvi, are in mysterious unknown
units and sometimes give results that make no sense (why doesn't
uv go to zero at night? You're telling me that there's that much UV
in starlight?), but I already knew that was an issue with the Ambient.
It's not rtl_433's fault.
I notice that the numbers are often a bit different from what the
Ambient API reports; apparently they do some massaging of the numbers,
and the console has its own adjustment factors too.
We'll have to do some more calibration with a mercury thermometer
to see which set of numbers is right.
Anyway, cool stuff! It took no time at all to write a simple client
for my WatchWeather
web app that runs rtl_433 and monitors the JSON output.
I already had WatchWeather clients collecting reports from
Raspberry Pi Zero Ws sitting at various places in the house with
temperature/humidity sensors attached; and now my WatchWeather page
can include the weather station itself.
Meanwhile, we donated another
station to the Los Alamos Nature Center,
though it doesn't have the SDR dongle, just the normal Ambient console
reporting to Wunderground.
[ 13:20 Feb 12, 2019
More tech |
permalink to this entry |
Sun, 29 Jul 2018
In my several recent articles about building Firefox from source,
I omitted one minor change I made, which will probably sound a bit silly.
A self-built Firefox thinks its name is "Nightly", so, for example,
the Help menu includes About Nightly.
Somehow I found that unreasonably irritating. It's not
a nightly build; in fact, I hope to build it as seldom as possible,
ideally only after a
git pull when new versions are released.
Yet Firefox shows its name in quite a few places, so you're constantly
faced with that "Nightly". After all the work to build Firefox,
why put up with that?
To find where it was coming from,
I used my recursive grep alias which skips the obj- directory plus
things like object files and metadata. This is how I define it in my .zshrc
(obviously, not all of these clauses are necessary for this Firefox
search), and then how I called it to try to find instances of
"Nightly" in the source:
find . \( -type f -and -not -name '*.o' -and -not -name '*.so' -and -not -name '*.a' -and -not -name '*.pyc' -and -not -name '*.jpg' -and -not -name '*.JPG' -and -not -name '*.png' -and -not -name '*.xcf*' -and -not -name '*.gmo' -and -not -name '.intltool*' -and -not -name '*.po' -and -not -name 'po' -and -not -name '*.tar*' -and -not -name '*.zip' -or -name '.metadata' -or -name 'build' -or -name 'obj-*' -or -name '.git' -or -name '.svn' -prune \) -print0 | xargs -0 grep $* /dev/null
gr Nightly | grep -v '//' | grep -v '#' | grep -v isNightly | grep test | grep -v task | fgrep -v .js | fgrep -v .cpp | grep -v mobile >grep.out
Even with all those exclusions, that still ends up printing
an enormous list. But it turns out all the important hits
are in the browser directory, so you can get away with
running it from there rather than from the top level.
I found a bunch of likely files that all had very similar
"Nightly" lines in them:
Since I didn't know which one was relevant, I changed each of them to
slightly different names, then rebuilt and checked to see which names
I actually saw while running the browser.
It turned out that
is the file that controls the application name in the Help menu
and in Help->About -- though the title of the
About window is still "Nightly" and I haven't found what controls that.
branding/unofficial/locales/en-US/brand.ftl controls the
"Nightly" references in the Edit->Preferences window.
I don't know what all the others do.
There may be other instances of "Nightly" that appear elsewhere in the app,
the other files, but I haven't seen them yet.
Past Firefox building articles:
Building Firefox Quantum;
Building Firefox for ALSA (non PulseAudio) Sound;
Firefox Quantum: Fixing Ctrl W (or other key bindings).
[ 18:23 Jul 29, 2018
More tech/web |
permalink to this entry |
Sat, 07 Jul 2018
A quick followup to my article on
Firefox Files Inside omni.ja:
The steps for modifying the file are fairly easy, but they have to be
done a lot.
First there's the problem of Firefox updates: if a new omni.ja
is part of the update, then your changes will be overwritten, so
you'll have to make them again on the new omni.ja.
But, worse, even aside from updates they don't stay changed.
I've had Ctrl-W mysteriously revert back to its old wired-in
behavior in the middle of a Firefox session. I'm still not clear
how this happens: I speculate that something in Firefox's update mechanism
may allow parts of omni.ja to be overridden, even though I was
told by Mike Kaply, the onetime master of overlays, that they
weren't recommended any more (at least by users, though that doesn't
necessarily mean they're not used for updates).
But in any case, you can be browsing merrily along and suddenly one
of your changes doesn't work any more, even though the change is still
right there in browser/omni.ja. And the only fix I've found
so far is to download a new Firefox and re-apply the changes.
Re-applying them to the current version doesn't work -- they're
already there. And it doesn't help to keep the tarball you originally
downloaded around so you can re-install that; firefox updates every
week or two so that version is guaranteed to be out of date.
All this means that it's crazy not to script the omni changes so
you can apply them easily with a single command. So here's a shell
script that takes the path to the current Firefox, unpacks
browser/omni.ja, makes a couple of simple changes and
re-packs it. I called it
since I used to call my personally modified Firefox build "Kitfox".
Of course, if your changes are different from mine
you'll want to edit the script to change the sed commands.
I hope eventually to figure out how it is that omni.ja changes
stop working, and whether it's an overlay or something else,
and whether there's a way to re-apply fixes without having to
download a whole new Firefox.
If I figure it out I'll report back.
[ 15:01 Jul 07, 2018
More tech/web |
permalink to this entry |
Sat, 23 Jun 2018
My article on
key bindings in Firefox Quantum by modifying the source tree got
attention from several people who offered helpful suggestions via
Twitter and email on how
to accomplish the same thing using just files in omni.ja,
so it could be done without rebuilding the Firefox source.
That would be vastly better, especially for people who need to
change something like key bindings or browser messages but don't
have a souped-up development machine to build the whole browser.
Brian Carpenter had several suggestions and eventually pointed me to
an old post by Mike Kaply,
Don’t Unpack and Repack omni.ja[r]
that said there were better ways to override specific files.
Unfortunately, Mike Kaply responded that that article was written
for XUL extensions, which are now obsolete, so the article
ought to be removed. That's too bad, because it did sound like a much
nicer solution. I looked into trying it anyway, but the instructions
it points to for
specific files is woefully short on detail on how to map a path
inside omni.ja like chrome://package/type/original-uri.whatever,
to a URL, and the single example I could find was so old that the file
it referenced didn't exist at the same location any more. After a fruitless
half hour or so, I took Mike's warning to heart and decided it wasn't
worth wasting more time chasing something that wasn't expected to work
anyway. (If someone knows otherwise, please let me know!)
But then Paul Wise offered a solution that actually worked, as an
easy to follow sequence of shell commands. (I've changed some of
them very slightly.)
$ tar xf ~/Tarballs/firefox-60.0.2.tar.bz2
# (This creates a "firefox" directory inside the current one.)
$ mkdir omni
$ cd omni
$ unzip -q ../firefox/browser/omni.ja
warning [../firefox-60.0.2/browser/omni.ja]: 34187320 extra bytes at beginning or within zipfile
(attempting to process anyway)
error [../firefox-60.0.2/browser/omni.ja]: reported length of central directory is
-34187320 bytes too long (Atari STZip zipfile? J.H.Holm ZIPSPLIT 1.1
zsh: exit 2 unzip -q ../firefox-60.0.2/browser/omni.ja
$ sed -i 's/or enter address/or just twiddle your thumbs/' chrome/en-US/locale/browser/browser.dtd chrome/en-US/locale/browser/browser.properties
I was a little put off by all the warnings unzip gave, but kept going.
Of course, you can just edit those two files rather than using sed;
but the sed command was Paul's way of being very specific about
the changes he was suggesting, which I appreciated.
Use these flags to repackage omni.ja:
$ zip -qr9XD ../omni.ja *
I had tried that before (without the q since I like to see what zip
and tar commands are doing) and hadn't succeeded. And indeed, when I
listed the two files, the new omni.ja I'd just packaged was
about a third the size of the original:
$ ls -l ../omni.ja ../firefox-60.0.2/browser/omni.ja
-rw-r--r-- 1 akkana akkana 34469045 Jun 5 12:14 ../firefox/browser/omni.ja
-rw-r--r-- 1 akkana akkana 11828315 Jun 17 10:37 ../omni.ja
But still, it's worth a try:
$ cp ../omni.ja ../firefox/browser/omni.ja
Then run the new Firefox. I have a spare profile I keep around for
testing, but Paul's instructions included a nifty way of running with
a brand new profile and it's definitely worth knowing:
$ cd ../firefox
$ MOZILLA_DISABLE_PLUGINS=1 ./firefox -safe-mode -no-remote -profile $(mktemp -d tmp-firefox-profile-XXXXXXXXXX) -offline about:blank
Also note the flags like safe-mode and no-remote, plus disabling
plugins -- all good ideas when testing something new.
And it worked! When I started up, I got the new message, "Search or
just twiddle your thumbs", in the URL bar.
Of course, now I had to test it with my real change. Since I like Paul's
way of using sed to specify exactly what changes to make, here's a
sed version of my Ctrl-W fix:
$ sed -i '/key_close/s/ reserved="true"//' chrome/browser/content/browser/browser.xul
Then run it. To test Ctrl-W, you need a website that includes a text
field you can type in, so -offline isn't an option unless you
happen to have a local web page that includes some text fields.
Google is an easy way to test ... and you might as well re-use that
firefox profile you just made rather than making another one:
$ MOZILLA_DISABLE_PLUGINS=1 ./firefox -safe-mode -no-remote -profile tmp-firefox-profile-* https://google.com
I typed a few words in the google search field that came up, deleted
them with Ctrl-W -- all was good! Thanks, Paul! And Brian, and
everybody else who sent suggestions.
Why are the sizes so different?
I was still puzzled by that threefold difference in size between the
omni.ja I repacked and the original that comes with Firefox.
Was something missing? Paul had the key to that too: use zipinfo
on both versions of the file to see what differed. Turned out
Mozilla's version, after a long file listing, ends with
2650 files, 33947999 bytes uncompressed, 33947999 bytes compressed: 0.0%
while my re-packaged version ends with
2650 files, 33947969 bytes uncompressed, 11307294 bytes compressed: 66.7%
So apparently Mozilla's omni.ja is using no compression at all.
It may be that that makes it start up a little faster; but Quantum
takes so long to start up that any slight difference in uncompressing
omni.ja isn't noticable to me.
I was able to run through this whole procedure on my poor slow netbook,
the one where building Firefox took something like 15 hours ... and
in a few minutes I had a working modified Firefox. And with the sed
command, this is all scriptable, so it'll be easy to re-do whenever
Firefox has a security update. Win!
Update: I have a simple shell script to do this:
Script to modify omni.ja for a custom Firefox.
[ 20:37 Jun 23, 2018
More tech/web |
permalink to this entry |
Thu, 14 Jun 2018
When I first tried switching to Firefox Quantum,
the regression that bothered me most was
Ctrl-W, which I use everywhere as word erase (try it -- you'll get
addicted, like I am). Ctrl-W deletes words in the URL bar;
but if you type Ctrl-W in a text field on a website,
like when editing a bug report or a "Contact" form,
it closes the current tab, losing everything you've just typed.
It's always worked in Firefox in the past; this is a new problem
with Quantum, and after losing a page of typing for about the
20th time, I was ready to give up and find another browser.
A web search found plenty of people online asking about key bindings
like Ctrl-W, but apparently since the deprecation of XUL and XBL
extensions, Quantum no longer offers any way to change or even
just to disable its built-in key bindings.
I wasted a few days chasing a solution inspired by this
of remapping keys only for certain windows using
xdotool getactivewindow; I even went so far as to
write a Python script that intercepts keystrokes, determines the
application for the window where the key was typed, and remaps it if
the application and keystroke match a list of keys to be remapped.
So if Ctrl-W is typed in a Firefox window, Firefox will instead
receive Alt-Backspace. (Why not just type Alt-Backspace, you ask?
Because it's much harder to type, can't be typed from the home
position, and isn't in the same place on every keyboard the way
But sadly, that approach didn't work because it turned out my window
manager, Openbox, acts on programmatically-generated key bindings as
well as ones that are actually typed. If I type a Ctrl-W and it's in
Firefox, that's fine: my Python program sees it, generates an
Alt-Backspace and everything is groovy.
But if I type a Ctrl-W in any other application, the program
doesn't need to change it, so it generates a Ctrl-W, which Openbox
sees and calls the program again, and you have an infinite loop.
I couldn't find any way around this. And admittedly, it's a horrible
hack having a program intercept every keystroke. So I needed to fix
But after spending days searching for a way to customize Firefox's
keys, to no avail, I came to the conclusion that the only way was to
modify the source code and
Firefox from source.
Ironically, one of the snags I hit in building it was that
I'd named my key remapper "pykey.py", and it was still in my
PYTHONPATH; it turns out the Firefox build also has a module called
pykey.py and mine was interfering. But eventually I got the build working.
Firefox Key Bindings
I was lucky: building was the only hard part, because
a very helpful person on Mozilla's #introduction IRC channel
pointed me toward the solution, saving me hours of debugging. Edit
around line 240
reserved="true" from key_closeWindow.
It turned out I needed to remove
the adjacent key_close line as well.
Another file that's related, but more general, is
around line 832; but I didn't need that since the simpler fix worked.
Transferring omni.ja -- or Not
In theory, since browser-sets.inc isn't compiled C++, it seems
like you should be able to make this fix without building the whole
source tree. In an actual Firefox release,
browser-sets.inc is part of omni.ja, and indeed if
you unpack omni.ja you'll see the key_closeWindow and
key_close lines. So it seems like you ought to be able to
regenerate omni.ja without rebuilding all the C++ code.
Unfortunately, in practice omni.ja is more complicated than that.
Although you can unzip it and edit the files, if you zip it back up,
Firefox doesn't see it as valid. I guess that's why they renamed it
.ja: long ago it used to be omni.jar and, like other
.jar files, was a standard zip archive that you could edit.
But the new .ja file isn't documented anywhere I could find,
and all the web discussions I found on how to re-create it amounted
to "it's complicated, you probably don't want to try".
And you'd think that I could take the omni.ja file from my
desktop machine, where I built Firefox, and copy it to my laptop,
replacing the omni.ja file from a released copy of Firefox.
But no -- somehow, it isn't seen, and the old key bindings are still
active. They must be duplicated somewhere else, and I haven't figured
It sure would be nice to have a way to transfer an omni.ja.
Building Firefox on my laptop takes nearly a full day (though
hopefully rebuilding after pulling minor security updates won't be
quite so bad). If anyone knows of a way, please let me know!
[ 16:45 Jun 14, 2018
More tech/web |
permalink to this entry |
Sat, 09 Jun 2018
I did the work to
my own Firefox primarily to fix a couple of serious regressions
that couldn't be fixed any other way. I'll start with the one that's
probably more common (at least, there are many people complaining
about it in many different web forums): the fact that Firefox won't
play sound on Linux machines that don't use PulseAudio.
There's a bug with a long discussion of the problem,
Bug 1345661 - PulseAudio requirement breaks Firefox on ALSA-only systems;
and the discussion in the bug links to another
of the Firefox/PulseAudio problem). Some comments in those
discussions suggest that some near-future version of Firefox may
restore ALSA sound for non-Pulse systems; but most of those comments
are six months old, yet it's still not fixed in the version Mozilla
is distributing now.
In theory, ALSA sound is easy to enable.
pptions in Firefox are controlled through a file called mozconfig.
Create that file at the top level of your build directory, then add to it:
You can see other options with
Of course, like everything else in the computer world, there were
complications. When I typed
mach build, I got:
Assertion failed in _parse_loader_output:
Traceback (most recent call last):
File "/home/akkana/outsrc/gecko-dev/python/mozbuild/mozbuild/mozconfig.py", line 260, in read_mozconfig
parsed = self._parse_loader_output(output)
File "/home/akkana/outsrc/gecko-dev/python/mozbuild/mozbuild/mozconfig.py", line 375, in _parse_loader_output
assert not in_variable
Error loading mozconfig: /home/akkana/outsrc/gecko-dev/mozconfig
Evaluation of your mozconfig produced unexpected output. This could be
triggered by a command inside your mozconfig failing or producing some warnings
or error messages. Please change your mozconfig to not error and/or to catch
errors in executed commands.
... followed by a many-page dump of all my environment variables, twice.
It turned out that was coming from line 449 of
# Lines with a quote not ending in a quote are multi-line.
if has_quote and not value.endswith("'"):
in_variable = name
value = value[:-1] if has_quote else value
I'm guessing this was added because some Mozilla developer sets
a multi-line environment variable that has a quote in it but doesn't
end with a quote. Or something. Anyway, some fairly specific case.
I, on the other hand, have a different specific case: a short
environment variable that includes one or more single quotes,
and the test for their specific case breaks my build.
(In case you're curious why I have quotes in an environment variable:
The prompt-setting code in my .zshrc includes a variable called
PRIMES. In a login shell, this is set to the empty string,
but in subshells, I add ' for each level of shell under the login shell.
So my regular prompt might be (hostname)-, but if I run a
subshell to test something, the prompt will be (hostname')-,
a subshell inside that will be (hostname'')-, and so on.
It's a reminder that I'm still in a subshell and need to exit when
I'm done testing. In theory, I could do that with SHLVL, but SHLVL
doesn't care about login shells, so my normal shells inside X are
all SHLVL=2 while shells on a console or from an ssh are
SHLVL=1, so if I used SHLVL I'd have to have some special case
code to deal with that.
Also, of course I could use a character other than a single-quote.
But in the thirty or so years I've used this, Firefox is the first
program that's ever had a problem with it. And apparently I'm not the
first one to have a problem with this:
was apparently someone else with the same problem. Maybe that will show
up in the release branch eventually.)
Anyway, disabling that line fixed the problem:
# Lines with a quote not ending in a quote are multi-line.
if False and has_quote and not value.endswith("'"):
and after that,
succeeded, I built a new
Firefox, and lo and behond! I can play sound in YouTube videos and
on Xeno-Canto again, without needing an additional browser.
[ 16:49 Jun 09, 2018
More tech/web |
permalink to this entry |
Sun, 03 Jun 2018
With Firefox Quantum, Mozilla has moved away from
letting users configure the browser they way they like.
If I was going to switch to Quantum as my everyday browser,
there were several problems I needed to fix first -- and they all
now require modifying the source code, then building the whole browser
I'll write separately about fixing the specific problems; but first
I had to build Firefox. Although I was a Firefox developer way back in
the day, the development environment has changed completely since then,
so I might as well have been starting from scratch.
Setting up a Firefox build
I started with Mozilla's
build preparation page. There's a script
called bootstrap.py that's amazingly comprehensive. It will
check what's installed on your machine and install what's needed for a
Firefox build -- and believe me, there are a lot of dependencies.
Don't take the "quick" part of the "quick and easy" comment at the
beginning of the script too seriously; I think on my machine, which
already has a fairly full set of build tools, the script was
downloading additional dependencies for 45 minutes or so. But it
was indeed fairly easy: the script asks lots of questions about
optional dependencies, and usually has suggestions, which I mostly
Eventually bootstrap.py finishes loading the dependencies and
gets to the point of wanting to check out the mozilla-unified
repository, and that's where I got into trouble.
The script wants to check out the bleeding edge tip of Mozilla development.
That's what you want if you're a developer checking in to the project.
What I wanted was a copy of the currently released Firefox, but with
a chance to make my own customizations. And that turns out to be difficult.
Getting a copy of the release tree
In theory, once you've checked out mozilla-unified with Mercurial,
assuming you let bootstrap.py enable the recommended "firefoxtree"
hg extension (which I did), you can switch to the release branch
hg pull release
hg up -c release
That didn't work for me: I tried it numerous times over the course of
the day, and every time it died with
"abort: HTTP request error (incomplete response; expected 5328 bytes
got 2672)" after "adding manifests" when it started "adding file changes".
That sent me on a long quest aided by someone in Mozilla's helpful
#introduction channel, where they help people with build issues.
You might think it would be a common thing to want to build a copy of
the released version of Firefox, and update it when a new release
comes out. But apparently not; although #introduction is a friendly
and helpful channel, everyone seemed baffled as to why
didn't work and what the possible alternatives might be.
Bundles and Artifacts
Eventually someone pointed me to the long list of
tarballs and advised me on how to get a release tarball there.
I actually did that, and (skipping ahead briefly) it built and ran;
but I later discovered that "bundles"
aren't actually hg repositories and can't be updated. So once you've
downloaded your 3 gigabytes or so of Mozilla stuff and built it, it's
only good for a week or two until the next Mozilla release, when
you're now hopelessly out of date and have to download a whole nuther
bundle. Bundles definitely aren't the answer, and they aren't well
supported or documented either. I recommend staying away from them.
I should also mention "artifact builds". These sound like a great
idea: a lot of the source is already built for you, so you just build
a little bit of it. However, artifact builds are only available for a
few platforms and branches. If your OS differs in any way from whoever
made the artifact build, or if you're requesting a branch, you're
likely to waste a lot of time (like I did) downloading stuff only to
get mysterious error messages. And even if it works, you won't be able
to update it to keep on top of security fixes. Doesn't seem like a
GitHub to the rescue
Okay, so Mercurial's branch switching doesn't work.
But it turns out you don't have to use Mercurial. There's a
GitHub mirror for Firefox
called gecko-dev, and after cloning it you can use normal
git commands to switch branches:
git clone https://github.com/mozilla/gecko-dev.git
git checkout -t origin/release
You can verify you're on the right branch with
git branch -vv, or if you want to list all branches
and their remotes,
git branch -avv.
Finally: a Firefox release branch that you can actually update!
Once you have a source tree, you can use the all-powerful mach
script to build the current release of Firefox:
Of course that takes forever -- hours and hours, depending on how
fast your machine is.
Running your New Firefox
The build, after it finishes, helpfully tells you to test it with
./mach run, which runs your newly-built firefox with a
special profile, so it doesn't interfere with your running build.
It also prints:
For more information on what to do now, see https://developer.mozilla.org/docs/Developer_Guide/So_You_Just_Built_Firefox
Great! Except there's no information there on how to package or run
your build -- it's just a placeholder page asking people to contribute
to the page.
It turns out that obj-whatever/dist/bin is the directory that
corresponds to the tarball you download from Mozilla, and you can run
/path/to/mozilla-release/obj-whatever/dist/bin/firefox from anywhere.
I tried filing a bug request to have a sub-page created explaining how
to run a newly built Firefox, but it hasn't gotten any response.
Maybe I'll just edit the "So You Just Built" page.
Update, 7 months later:
Nobody ever did respond to my bug, but someone on Mozilla's #introduction
channel for help with builds gave me a blessing to modify the page directly,
which I did.
Specifically, I added:
Your new Firefox executable can be found in: $OBJDIR/dist/bin/firefox. You can run it from there.
If you need to move it, e.g. to another machine, you can run:
This should create an OS-specific package, e.g. a tarball on Linux, which will appear in $OBJDIR/dist. You can also copy the $OBJDIR/dist/bin directory -- be sure to use a copy method that expands soft links -- but the result will be much larger than what you get with mach package.
On Windows you may want to read about
Windows Installer Builds.
Incidentally, my gecko-dev build takes 16G of disk space, of which
9.3G is things it built, which are helpfully segregated in
[ 15:55 Jun 03, 2018
More tech/web |
permalink to this entry |
Thu, 31 May 2018
For the last year or so the Firefox development team has been making
life ever harder for users. First they broke all the old extensions
that were based on XUL and XBL, so a lot of customizations no longer
worked. Then they
PulseAudio mandatory on Linux bug (1345661), so on systems
like mine that don't run Pulse, there's no way to get sound in
a web page. Forget YouTube or XenoCanto unless you keep another
browser around for that purpose.
For those reasons I'd been avoiding the Firefox upgrade, sticking to
Debian's firefox-esr ("Extended Support Release"). But when
Debian updated firefox-esr to Firefox 56 ESR late last year, performance
became unusable. Like half a minute between when you hit Page Down
and when the page actually scrolls. It was time to switch browsers.
I'd been hearing about the Firefox variant Pale Moon. It's a fork of
an older Firefox, supposedly with an emphasis on openness and configurability.
I installed the Debian palemoon package. Performance was fine,
similar to Firefox before the tragic firefox-56. It was missing a few
things -- no built-in PDF viewer or Reader mode -- but I don't use
Reader mode that often, and the built-in PDF viewer is an annoyance at
least as often as it's a help. (In Firefox it's fairly random about when
it kicks in anyway, so I'm never sure whether I'll get the PDF viewer
or a Save-as prompt on any given PDF link).
For form and password autofill, for some reason Pale Moon doesn't fill
out fields until you type the first letter. For instance, if I had an
account with name "myname" and a stored password, when I loaded the
page, both fields would be empty, as if there's nothing stored for that
page. But typing an 'm' in the username field makes both username and
password fields fill in. This isn't something Firefox ever did and I
don't particularly like it, but it isn't a major problem.
Then there were some minor irritations, like the fact that profiles
were stored in a folder named ~/.moonchild\ productions/ --
super long so it messed up directory listings, and with a space in the
middle. PaleMoon was also very insistent about using new tabs for
everything, including URLs launched from other programs -- there
doesn't seem to be any way to get it to open URLs in the active tab.
I used it as my main browser for several months, and it basically worked.
But the irritations started to get to me, and I started considering
other options. The final kicker when I saw
bug 86, in which, as far as I can tell, someone working on the
PaleMoon in OpenBSD tries to use system libraries instead of
PaleMoon's patched libraries, and is attacked for it in the bug.
Reading the exchange made me want to avoid PaleMoon for two reasons.
First, the rudeness: a toxic community that doesn't treat contributors
well isn't likely to last long or to have the resources to keep on top
of bug and security fixes. Second, the technical question: if Pale
Moon's code is so quirky that it can't use standard system libraries
and needs a bunch of custom-patched libraries, what does that say
about how maintainable it will be in the long term?
Much has been made in the technical press of the latest Firefox,
called "Quantum", and its supposed speed. I was a bit dubious of that:
it's easy to make your program seem fast after you force everybody
into a few years of working with a program that's degraded its
performance by an order of magnitude, like Firefox had. After
firefox 56, anything would seem fast.
Still, maybe it would at least be fast enough to be usable. But I had
trepidations too. What about all those extensions that don't work any
more? What about sound not working? Could I live with that?
Debian has no current firefox package, so I downloaded the tarball
from mozilla.org, unpacked it,
made a new firefox profile and ran it.
Initial startup performance is terrible -- it takes forever to bring
up the first window, and I often get a "Firefox seems slow to start up"
message at the bottom of the screen, with a link to a page of a bunch
of completely irrelevant hints.
Still, I typically only start Firefox once a day. Once it's up,
performance is a bit laggy but a lot better than firefox-esr 56 was,
I was able to find replacements for most of the really important
But sound, as predicted, didn't work. And there were several other,
worse regressions from older Firefox versions.
As it turned out, the only way to make Firefox Quantum usable for me
was to build a custom version where I could fix the regressions.
To keep articles from being way too long, I'll write about all those
how to build Firefox,
how to fix broken key bindings,
and how to fix the PulseAudio problem.
[ 16:07 May 31, 2018
More tech/web |
permalink to this entry |