Shallow Thoughts : : tech

Akkana's Musings on Open Source Computing and Technology, Science, and Nature.

Tue, 12 Feb 2019

Reading the Output of a Weather Station Using Software Defined Radio

[Weather station] A while back, Dave ordered a weather station. His research pointed to the Ambient Weather WS-2000 as the best bang for the buck as far as accuracy (after it's calibrated, which is a time consuming and exacting process where you compare readings to a known-good mercury thermometer, a process that I suspect most weather station owners don't bother with).

It comes with a little 7" display console that sits indoors and reads the radio signal from the outside station as well as a second thermometer inside, then displays all the current weather data. It also uses wi-fi to report the data upstream to Ambient and, optionally, to a weather site such as Wunderground. (Which we did for a while, but now Wunderground is closing off their public API, so why give them data if they're not going to make it easy to share it?)

[Weather station console] Having the console readout and the Ambient "dashboard" is all very nice, but of course, being a data geek, I wanted a way to get the data myself, so I could plot it, save it or otherwise process it. And that's where Ambient falls short. The console, though it's already talking on wi-fi, gives you no way to get the data. They sell a separate unit called an "Observer" that provides a web page you can scrape, and we actually ordered one, but it turned out to be buggy and difficult to use, giving numbers that were substantially different from what the console showed, and randomly failing to answer, and we ended up returning the observer for a refund.

The other way of getting the data is online. Ambient provides an API you can use for that purpose, if you email them for a key. It mostly works, but it sometimes lags considerably behind real time, and it seems crazy to have to beg for a key and then get data from a company website that originated in our own backyard.

What I really wanted to do was read the signal from the weather station directly. I'd planned for ages to look into how to do that, but I'm a complete newbie to software defined radio and wasn't sure where to start. Then one day I noticed an SDR discussion on the #raspberrypi IRC channel on Freenode where I often hang out. I jumped in, asked some questions, and got pointed in the right direction and referred to the friendly and helpful #rtlsdr Freenode channel.

An Inexpensive SDR Dongle

[Raspberry Pi with SDR dongle] On the experts' advice, I ordered a RTL-SDR Blog R820T2 RTL2832U 1PPM TCXO SMA Software Defined Radio with 2x Telescopic Antennas on Amazon. This dongle apparently has better temperature compensation than cheaper alternatives, it came with a couple of different antenna options, and I was told it should work well with Linux using a program called rtl_433.

Indeed it did. The command to monitor the weather station is

rtl_433 -f 915M

rtl_433 already knows the protocol for the WS-2000, so I didn't even need to do any decoding or reverse engineering; it produces a running account of the periodic signals being broadcast from the station. rtl_433 also helpfully offers -F json and -F csv options, along with a few other formats. What a great program!

JSON turned out to be the easiest for me to use; initially I thought CSV would be more compact, but rtl_433's CSV format includes fields for every possible quantity a weather station could ever broadcast. When you think about it, that makes sense: once you're outputting CSV you can't add a new field in mid-stream, so you'd better be ready for anything. JSON, on the other hand, lets you report just whatever the weather station reports, and it's easy to parse from nearly any programming language.

Testing the SDR Dongle

Full disclosure: at first, rtl_433 -f 915M wasn't showing me anything and I still don't know why. Maybe I had a loose connection on the antenna, or maybe I got unlucky and the weather station picked the exact wrong time to take a vacation. But while I was testing, I found another program that was very helpful in testing whether my SDR dongle was working: rtl_fm, which plays radio stations. The only trick is finding the right arguments, since the example from the man page just played static. Here's what worked for me:

rtl_fm -f 101.1M -M fm -g 20 -s 200k -A fast -r 32k -l 0 -E deemp | play -r 32k -t raw -e s -b 16 -c 1 -V1 -
That command plays the 101.1 FM radio station. (I had to do a web search to give me some frequencies of local radio stations; it's been a long time since I listened to normal radio.)

Once I knew the dongle was working, I needed to verify what frequency the weather station was using for its broadcasts. What I really wanted was something that would scan frequencies around 915M and tell me what it found. Everyone kept pointing me to a program called Gqrx. But it turns out Gqrx on Linux requires PulseAudio and absolutely refuses to work or install without it, even if you have no interest in playing audio. I didn't want to break my system's sound (I've never managed to get sound working reliably under PulseAudio), and although it's supposedly possible to build Gqrx without Pulse, it's a difficult build: I saw plenty of horror stories, and it requires Boost, always a sign that a build will be problematic. I fiddled with it a little but decided it wasn't a good time investment.

I eventually found a scanner that worked: RTLSDR-Scanner. It let me set limiting frequencies and scan between them, and by setting it to accumulate, I was able to verify that indeed, the weather station (or something else!) was sending a signal on 915 MHz. I guess by then, the original problem had fixed itself, and after that, rtl_433 started showing me signals from the weather station. It's not super polished, but it's the only scanner I've found that works without requiring PulseAudio.

That Puzzling Rainfall Number

One mystery remained to be solved. The JSON I was getting from the weather station looked like this (I've reformatted it for readablility):

    "time" : "2019-01-11 11:50:12",
    "model" : "Fine Offset WH65B",
    "id" : 60,
    "temperature_C" : 2.200,
    "humidity" : 94,
    "wind_dir_deg" : 316,
    "wind_speed_ms" : 0.064,
    "gust_speed_ms" : 0.510,
    "rainfall_mm" : 90.678,
    "uv" : 324,
    "uvi" : 0,
    "light_lux" : 19344.000,
     "battery" : "OK",
    "mic" : "CRC"

This on a day when it hadn't rained in ages. What was up with that "rainfall_mm" : 90.678 ?

I asked on the rtl_433 list and got a prompt and helpful answer: it's a cumulative number, since some unspecified time in the past (possibly the last time the battery was changed?) So as long as I make a note of the rainfall_mm number, any change in that number means new rainfall.

This being a snowy winter, I haven't been able to test that yet: the WS-2000 doesn't measure snowfall unless some snow happens to melt in the rain cup.

Some of the other numbers, like uv and uvi, are in mysterious unknown units and sometimes give results that make no sense (why doesn't uv go to zero at night? You're telling me that there's that much UV in starlight?), but I already knew that was an issue with the Ambient. It's not rtl_433's fault.

I notice that the numbers are often a bit different from what the Ambient API reports; apparently they do some massaging of the numbers, and the console has its own adjustment factors too. We'll have to do some more calibration with a mercury thermometer to see which set of numbers is right.

Anyway, cool stuff! It took no time at all to write a simple client for my WatchWeather web app that runs rtl_433 and monitors the JSON output. I already had WatchWeather clients collecting reports from Raspberry Pi Zero Ws sitting at various places in the house with temperature/humidity sensors attached; and now my WatchWeather page can include the weather station itself.

Meanwhile, we donated another weather station to the Los Alamos Nature Center, though it doesn't have the SDR dongle, just the normal Ambient console reporting to Wunderground.

Tags: , , ,
[ 13:20 Feb 12, 2019    More tech | permalink to this entry | comments ]

Sun, 29 Jul 2018

Building Firefox: Changing the App Name

In my several recent articles about building Firefox from source, I omitted one minor change I made, which will probably sound a bit silly. A self-built Firefox thinks its name is "Nightly", so, for example, the Help menu includes About Nightly.

Somehow I found that unreasonably irritating. It's not a nightly build; in fact, I hope to build it as seldom as possible, ideally only after a git pull when new versions are released. Yet Firefox shows its name in quite a few places, so you're constantly faced with that "Nightly". After all the work to build Firefox, why put up with that?

To find where it was coming from, I used my recursive grep alias which skips the obj- directory plus things like object files and metadata. This is how I define it in my .zshrc (obviously, not all of these clauses are necessary for this Firefox search), and then how I called it to try to find instances of "Nightly" in the source:

gr() {
  find . \( -type f -and -not -name '*.o' -and -not -name '*.so' -and -not -name '*.a' -and -not -name '*.pyc' -and -not -name '*.jpg' -and -not -name '*.JPG' -and -not -name '*.png' -and -not -name '*.xcf*' -and -not -name '*.gmo' -and -not -name '.intltool*' -and -not -name '*.po' -and -not -name 'po' -and -not -name '*.tar*' -and -not -name '*.zip' -or -name '.metadata' -or -name 'build' -or -name 'obj-*' -or -name '.git' -or -name '.svn' -prune \) -print0 | xargs -0 grep $* /dev/null

gr Nightly | grep -v '//' | grep -v '#' | grep -v isNightly  | grep test | grep -v task | fgrep -v .js | fgrep -v .cpp | grep -v mobile >grep.out

Even with all those exclusions, that still ends up printing an enormous list. But it turns out all the important hits are in the browser directory, so you can get away with running it from there rather than from the top level.

I found a bunch of likely files that all had very similar "Nightly" lines in them:

Since I didn't know which one was relevant, I changed each of them to slightly different names, then rebuilt and checked to see which names I actually saw while running the browser.

It turned out that browser/branding/unofficial/locales/en-US/brand.dtd is the file that controls the application name in the Help menu and in Help->About -- though the title of the About window is still "Nightly" and I haven't found what controls that.

branding/unofficial/locales/en-US/brand.ftl controls the "Nightly" references in the Edit->Preferences window.

I don't know what all the others do. There may be other instances of "Nightly" that appear elsewhere in the app, the other files, but I haven't seen them yet.

Past Firefox building articles: Building Firefox Quantum; Building Firefox for ALSA (non PulseAudio) Sound; Firefox Quantum: Fixing Ctrl W (or other key bindings).

Tags: , , ,
[ 18:23 Jul 29, 2018    More tech/web | permalink to this entry | comments ]

Sat, 07 Jul 2018

Script to modify omni.ja for a custom Firefox

A quick followup to my article on Modifying Firefox Files Inside omni.ja:

The steps for modifying the file are fairly easy, but they have to be done a lot.

First there's the problem of Firefox updates: if a new omni.ja is part of the update, then your changes will be overwritten, so you'll have to make them again on the new omni.ja.

But, worse, even aside from updates they don't stay changed. I've had Ctrl-W mysteriously revert back to its old wired-in behavior in the middle of a Firefox session. I'm still not clear how this happens: I speculate that something in Firefox's update mechanism may allow parts of omni.ja to be overridden, even though I was told by Mike Kaply, the onetime master of overlays, that they weren't recommended any more (at least by users, though that doesn't necessarily mean they're not used for updates).

But in any case, you can be browsing merrily along and suddenly one of your changes doesn't work any more, even though the change is still right there in browser/omni.ja. And the only fix I've found so far is to download a new Firefox and re-apply the changes. Re-applying them to the current version doesn't work -- they're already there. And it doesn't help to keep the tarball you originally downloaded around so you can re-install that; firefox updates every week or two so that version is guaranteed to be out of date.

All this means that it's crazy not to script the omni changes so you can apply them easily with a single command. So here's a shell script that takes the path to the current Firefox, unpacks browser/omni.ja, makes a couple of simple changes and re-packs it. I called it kitfox-patch since I used to call my personally modified Firefox build "Kitfox".

Of course, if your changes are different from mine you'll want to edit the script to change the sed commands.

I hope eventually to figure out how it is that omni.ja changes stop working, and whether it's an overlay or something else, and whether there's a way to re-apply fixes without having to download a whole new Firefox. If I figure it out I'll report back.

Tags: ,
[ 15:01 Jul 07, 2018    More tech/web | permalink to this entry | comments ]

Sat, 23 Jun 2018

Modifying Firefox Files Inside Omni.ja

My article on Fixing key bindings in Firefox Quantum by modifying the source tree got attention from several people who offered helpful suggestions via Twitter and email on how to accomplish the same thing using just files in omni.ja, so it could be done without rebuilding the Firefox source. That would be vastly better, especially for people who need to change something like key bindings or browser messages but don't have a souped-up development machine to build the whole browser.

Brian Carpenter had several suggestions and eventually pointed me to an old post by Mike Kaply, Don’t Unpack and Repack omni.ja[r] that said there were better ways to override specific files.

Unfortunately, Mike Kaply responded that that article was written for XUL extensions, which are now obsolete, so the article ought to be removed. That's too bad, because it did sound like a much nicer solution. I looked into trying it anyway, but the instructions it points to for Overriding specific files is woefully short on detail on how to map a path inside omni.ja like chrome://package/type/original-uri.whatever, to a URL, and the single example I could find was so old that the file it referenced didn't exist at the same location any more. After a fruitless half hour or so, I took Mike's warning to heart and decided it wasn't worth wasting more time chasing something that wasn't expected to work anyway. (If someone knows otherwise, please let me know!)

But then Paul Wise offered a solution that actually worked, as an easy to follow sequence of shell commands. (I've changed some of them very slightly.)

$ tar xf ~/Tarballs/firefox-60.0.2.tar.bz2
  # (This creates a "firefox" directory inside the current one.)

$ mkdir omni
$ cd omni

$ unzip -q ../firefox/browser/omni.ja
warning [../firefox-60.0.2/browser/omni.ja]:  34187320 extra bytes at beginning or within zipfile
  (attempting to process anyway)
error [../firefox-60.0.2/browser/omni.ja]:  reported length of central directory is
  -34187320 bytes too long (Atari STZip zipfile?  J.H.Holm ZIPSPLIT 1.1
  zipfile?).  Compensating...
zsh: exit 2     unzip -q ../firefox-60.0.2/browser/omni.ja

$ sed -i 's/or enter address/or just twiddle your thumbs/' chrome/en-US/locale/browser/browser.dtd chrome/en-US/locale/browser/

I was a little put off by all the warnings unzip gave, but kept going.

Of course, you can just edit those two files rather than using sed; but the sed command was Paul's way of being very specific about the changes he was suggesting, which I appreciated.

Use these flags to repackage omni.ja:

$ zip -qr9XD ../omni.ja *

I had tried that before (without the q since I like to see what zip and tar commands are doing) and hadn't succeeded. And indeed, when I listed the two files, the new omni.ja I'd just packaged was about a third the size of the original:

$ ls -l ../omni.ja ../firefox-60.0.2/browser/omni.ja
-rw-r--r-- 1 akkana akkana 34469045 Jun  5 12:14 ../firefox/browser/omni.ja
-rw-r--r-- 1 akkana akkana 11828315 Jun 17 10:37 ../omni.ja

But still, it's worth a try:

$ cp ../omni.ja ../firefox/browser/omni.ja

Then run the new Firefox. I have a spare profile I keep around for testing, but Paul's instructions included a nifty way of running with a brand new profile and it's definitely worth knowing:

$ cd ../firefox

$ MOZILLA_DISABLE_PLUGINS=1 ./firefox -safe-mode -no-remote -profile $(mktemp -d tmp-firefox-profile-XXXXXXXXXX) -offline about:blank

Also note the flags like safe-mode and no-remote, plus disabling plugins -- all good ideas when testing something new.

And it worked! When I started up, I got the new message, "Search or just twiddle your thumbs", in the URL bar.

Fixing Ctrl-W

Of course, now I had to test it with my real change. Since I like Paul's way of using sed to specify exactly what changes to make, here's a sed version of my Ctrl-W fix:

$ sed -i '/key_close/s/ reserved="true"//' chrome/browser/content/browser/browser.xul

Then run it. To test Ctrl-W, you need a website that includes a text field you can type in, so -offline isn't an option unless you happen to have a local web page that includes some text fields. Google is an easy way to test ... and you might as well re-use that firefox profile you just made rather than making another one:

$ MOZILLA_DISABLE_PLUGINS=1 ./firefox -safe-mode -no-remote -profile tmp-firefox-profile-*

I typed a few words in the google search field that came up, deleted them with Ctrl-W -- all was good! Thanks, Paul! And Brian, and everybody else who sent suggestions.

Why are the sizes so different?

I was still puzzled by that threefold difference in size between the omni.ja I repacked and the original that comes with Firefox. Was something missing? Paul had the key to that too: use zipinfo on both versions of the file to see what differed. Turned out Mozilla's version, after a long file listing, ends with

2650 files, 33947999 bytes uncompressed, 33947999 bytes compressed:  0.0%
while my re-packaged version ends with
2650 files, 33947969 bytes uncompressed, 11307294 bytes compressed:  66.7%

So apparently Mozilla's omni.ja is using no compression at all. It may be that that makes it start up a little faster; but Quantum takes so long to start up that any slight difference in uncompressing omni.ja isn't noticable to me.

I was able to run through this whole procedure on my poor slow netbook, the one where building Firefox took something like 15 hours ... and in a few minutes I had a working modified Firefox. And with the sed command, this is all scriptable, so it'll be easy to re-do whenever Firefox has a security update. Win!

Update: I have a simple shell script to do this: Script to modify omni.ja for a custom Firefox.

Tags: ,
[ 20:37 Jun 23, 2018    More tech/web | permalink to this entry | comments ]

Thu, 14 Jun 2018

Firefox Quantum: Fixing Ctrl W (or other key bindings)

When I first tried switching to Firefox Quantum, the regression that bothered me most was Ctrl-W, which I use everywhere as word erase (try it -- you'll get addicted, like I am). Ctrl-W deletes words in the URL bar; but if you type Ctrl-W in a text field on a website, like when editing a bug report or a "Contact" form, it closes the current tab, losing everything you've just typed. It's always worked in Firefox in the past; this is a new problem with Quantum, and after losing a page of typing for about the 20th time, I was ready to give up and find another browser.

A web search found plenty of people online asking about key bindings like Ctrl-W, but apparently since the deprecation of XUL and XBL extensions, Quantum no longer offers any way to change or even just to disable its built-in key bindings.

I wasted a few days chasing a solution inspired by this clever way of remapping keys only for certain windows using xdotool getactivewindow; I even went so far as to write a Python script that intercepts keystrokes, determines the application for the window where the key was typed, and remaps it if the application and keystroke match a list of keys to be remapped. So if Ctrl-W is typed in a Firefox window, Firefox will instead receive Alt-Backspace. (Why not just type Alt-Backspace, you ask? Because it's much harder to type, can't be typed from the home position, and isn't in the same place on every keyboard the way W is.)

But sadly, that approach didn't work because it turned out my window manager, Openbox, acts on programmatically-generated key bindings as well as ones that are actually typed. If I type a Ctrl-W and it's in Firefox, that's fine: my Python program sees it, generates an Alt-Backspace and everything is groovy. But if I type a Ctrl-W in any other application, the program doesn't need to change it, so it generates a Ctrl-W, which Openbox sees and calls the program again, and you have an infinite loop. I couldn't find any way around this. And admittedly, it's a horrible hack having a program intercept every keystroke. So I needed to fix Firefox somehow.

But after spending days searching for a way to customize Firefox's keys, to no avail, I came to the conclusion that the only way was to modify the source code and rebuild Firefox from source.

Ironically, one of the snags I hit in building it was that I'd named my key remapper "", and it was still in my PYTHONPATH; it turns out the Firefox build also has a module called and mine was interfering. But eventually I got the build working.

Firefox Key Bindings

I was lucky: building was the only hard part, because a very helpful person on Mozilla's #introduction IRC channel pointed me toward the solution, saving me hours of debugging. Edit browser/base/content/ around line 240 and remove reserved="true" from key_closeWindow. It turned out I needed to remove reserved="true" from the adjacent key_close line as well.

Another file that's related, but more general, is nsXBLWindowKeyHandler.cpp around line 832; but I didn't need that since the simpler fix worked.

Transferring omni.ja -- or Not

In theory, since isn't compiled C++, it seems like you should be able to make this fix without building the whole source tree. In an actual Firefox release, is part of omni.ja, and indeed if you unpack omni.ja you'll see the key_closeWindow and key_close lines. So it seems like you ought to be able to regenerate omni.ja without rebuilding all the C++ code.

Unfortunately, in practice omni.ja is more complicated than that. Although you can unzip it and edit the files, if you zip it back up, Firefox doesn't see it as valid. I guess that's why they renamed it .ja: long ago it used to be omni.jar and, like other .jar files, was a standard zip archive that you could edit. But the new .ja file isn't documented anywhere I could find, and all the web discussions I found on how to re-create it amounted to "it's complicated, you probably don't want to try".

And you'd think that I could take the omni.ja file from my desktop machine, where I built Firefox, and copy it to my laptop, replacing the omni.ja file from a released copy of Firefox. But no -- somehow, it isn't seen, and the old key bindings are still active. They must be duplicated somewhere else, and I haven't figured out where.

It sure would be nice to have a way to transfer an omni.ja. Building Firefox on my laptop takes nearly a full day (though hopefully rebuilding after pulling minor security updates won't be quite so bad). If anyone knows of a way, please let me know!

Tags: , ,
[ 16:45 Jun 14, 2018    More tech/web | permalink to this entry | comments ]

Sat, 09 Jun 2018

Building Firefox for ALSA (non PulseAudio) Sound

I did the work to built my own Firefox primarily to fix a couple of serious regressions that couldn't be fixed any other way. I'll start with the one that's probably more common (at least, there are many people complaining about it in many different web forums): the fact that Firefox won't play sound on Linux machines that don't use PulseAudio.

There's a bug with a long discussion of the problem, Bug 1345661 - PulseAudio requirement breaks Firefox on ALSA-only systems; and the discussion in the bug links to another discussion of the Firefox/PulseAudio problem). Some comments in those discussions suggest that some near-future version of Firefox may restore ALSA sound for non-Pulse systems; but most of those comments are six months old, yet it's still not fixed in the version Mozilla is distributing now.

In theory, ALSA sound is easy to enable. Build pptions in Firefox are controlled through a file called mozconfig. Create that file at the top level of your build directory, then add to it:

ac_add_options --enable-alsa
ac_add_options --disable-pulseaudio

You can see other options with ./configure --help

Of course, like everything else in the computer world, there were complications. When I typed mach build, I got:

Assertion failed in _parse_loader_output:
Traceback (most recent call last):
  File "/home/akkana/outsrc/gecko-dev/python/mozbuild/mozbuild/", line 260, in read_mozconfig
    parsed = self._parse_loader_output(output)
  File "/home/akkana/outsrc/gecko-dev/python/mozbuild/mozbuild/", line 375, in _parse_loader_output
    assert not in_variable
Error loading mozconfig: /home/akkana/outsrc/gecko-dev/mozconfig

Evaluation of your mozconfig produced unexpected output.  This could be
triggered by a command inside your mozconfig failing or producing some warnings
or error messages. Please change your mozconfig to not error and/or to catch
errors in executed commands.

mozconfig output:

... followed by a many-page dump of all my environment variables, twice.

It turned out that was coming from line 449 of python/mozbuild/mozbuild/

   # Lines with a quote not ending in a quote are multi-line.
    if has_quote and not value.endswith("'"):
        in_variable = name
        value = value[:-1] if has_quote else value

I'm guessing this was added because some Mozilla developer sets a multi-line environment variable that has a quote in it but doesn't end with a quote. Or something. Anyway, some fairly specific case. I, on the other hand, have a different specific case: a short environment variable that includes one or more single quotes, and the test for their specific case breaks my build.

(In case you're curious why I have quotes in an environment variable: The prompt-setting code in my .zshrc includes a variable called PRIMES. In a login shell, this is set to the empty string, but in subshells, I add ' for each level of shell under the login shell. So my regular prompt might be (hostname)-, but if I run a subshell to test something, the prompt will be (hostname')-, a subshell inside that will be (hostname'')-, and so on. It's a reminder that I'm still in a subshell and need to exit when I'm done testing. In theory, I could do that with SHLVL, but SHLVL doesn't care about login shells, so my normal shells inside X are all SHLVL=2 while shells on a console or from an ssh are SHLVL=1, so if I used SHLVL I'd have to have some special case code to deal with that.

Also, of course I could use a character other than a single-quote. But in the thirty or so years I've used this, Firefox is the first program that's ever had a problem with it. And apparently I'm not the first one to have a problem with this: bug 1455065 was apparently someone else with the same problem. Maybe that will show up in the release branch eventually.)

Anyway, disabling that line fixed the problem:

   # Lines with a quote not ending in a quote are multi-line.
    if False and has_quote and not value.endswith("'"):
and after that, mach build succeeded, I built a new Firefox, and lo and behond! I can play sound in YouTube videos and on Xeno-Canto again, without needing an additional browser.

Tags: , ,
[ 16:49 Jun 09, 2018    More tech/web | permalink to this entry | comments ]

Sun, 03 Jun 2018

Building Firefox Quantum

With Firefox Quantum, Mozilla has moved away from letting users configure the browser they way they like. If I was going to switch to Quantum as my everyday browser, there were several problems I needed to fix first -- and they all now require modifying the source code, then building the whole browser from scratch.

I'll write separately about fixing the specific problems; but first I had to build Firefox. Although I was a Firefox developer way back in the day, the development environment has changed completely since then, so I might as well have been starting from scratch.

Setting up a Firefox build

I started with Mozilla's Linux build preparation page. There's a script called that's amazingly comprehensive. It will check what's installed on your machine and install what's needed for a Firefox build -- and believe me, there are a lot of dependencies. Don't take the "quick" part of the "quick and easy" comment at the beginning of the script too seriously; I think on my machine, which already has a fairly full set of build tools, the script was downloading additional dependencies for 45 minutes or so. But it was indeed fairly easy: the script asks lots of questions about optional dependencies, and usually has suggestions, which I mostly followed.

Eventually finishes loading the dependencies and gets to the point of wanting to check out the mozilla-unified repository, and that's where I got into trouble.

The script wants to check out the bleeding edge tip of Mozilla development. That's what you want if you're a developer checking in to the project. What I wanted was a copy of the currently released Firefox, but with a chance to make my own customizations. And that turns out to be difficult.

Getting a copy of the release tree

In theory, once you've checked out mozilla-unified with Mercurial, assuming you let enable the recommended "firefoxtree" hg extension (which I did), you can switch to the release branch with:

hg pull release
hg up -c release

That didn't work for me: I tried it numerous times over the course of the day, and every time it died with "abort: HTTP request error (incomplete response; expected 5328 bytes got 2672)" after "adding manifests" when it started "adding file changes".

That sent me on a long quest aided by someone in Mozilla's helpful #introduction channel, where they help people with build issues. You might think it would be a common thing to want to build a copy of the released version of Firefox, and update it when a new release comes out. But apparently not; although #introduction is a friendly and helpful channel, everyone seemed baffled as to why hg up didn't work and what the possible alternatives might be.

Bundles and Artifacts

Eventually someone pointed me to the long list of "bundle" tarballs and advised me on how to get a release tarball there. I actually did that, and (skipping ahead briefly) it built and ran; but I later discovered that "bundles" aren't actually hg repositories and can't be updated. So once you've downloaded your 3 gigabytes or so of Mozilla stuff and built it, it's only good for a week or two until the next Mozilla release, when you're now hopelessly out of date and have to download a whole nuther bundle. Bundles definitely aren't the answer, and they aren't well supported or documented either. I recommend staying away from them.

I should also mention "artifact builds". These sound like a great idea: a lot of the source is already built for you, so you just build a little bit of it. However, artifact builds are only available for a few platforms and branches. If your OS differs in any way from whoever made the artifact build, or if you're requesting a branch, you're likely to waste a lot of time (like I did) downloading stuff only to get mysterious error messages. And even if it works, you won't be able to update it to keep on top of security fixes. Doesn't seem like a good bet.

GitHub to the rescue

Okay, so Mercurial's branch switching doesn't work. But it turns out you don't have to use Mercurial. There's a GitHub mirror for Firefox called gecko-dev, and after cloning it you can use normal git commands to switch branches:

git clone
cd gecko-dev/
git checkout -t origin/release

You can verify you're on the right branch with git branch -vv, or if you want to list all branches and their remotes, git branch -avv.

Finally: a Firefox release branch that you can actually update!

Building Firefox

Once you have a source tree, you can use the all-powerful mach script to build the current release of Firefox:

./mach build

Of course that takes forever -- hours and hours, depending on how fast your machine is.

Running your New Firefox

The build, after it finishes, helpfully tells you to test it with ./mach run, which runs your newly-built firefox with a special profile, so it doesn't interfere with your running build. It also prints:

For more information on what to do now, see

Great! Except there's no information there on how to package or run your build -- it's just a placeholder page asking people to contribute to the page.

It turns out that obj-whatever/dist/bin is the directory that corresponds to the tarball you download from Mozilla, and you can run /path/to/mozilla-release/obj-whatever/dist/bin/firefox from anywhere.

I tried filing a bug request to have a sub-page created explaining how to run a newly built Firefox, but it hasn't gotten any response. Maybe I'll just edit the "So You Just Built" page.

Update, 7 months later:
Nobody ever did respond to my bug, but someone on Mozilla's #introduction channel for help with builds gave me a blessing to modify the page directly, which I did.

Specifically, I added:

Your new Firefox executable can be found in: $OBJDIR/dist/bin/firefox. You can run it from there.

If you need to move it, e.g. to another machine, you can run:
./mach package
This should create an OS-specific package, e.g. a tarball on Linux, which will appear in $OBJDIR/dist. You can also copy the $OBJDIR/dist/bin directory -- be sure to use a copy method that expands soft links -- but the result will be much larger than what you get with mach package. On Windows you may want to read about Windows Installer Builds.

Incidentally, my gecko-dev build takes 16G of disk space, of which 9.3G is things it built, which are helpfully segregated in obj-x86_64-pc-linux-gnu.

Tags: ,
[ 15:55 Jun 03, 2018    More tech/web | permalink to this entry | comments ]

Thu, 31 May 2018

Trying Firefox Variants: From Firefox ESR to Pale Moon to Quantum

For the last year or so the Firefox development team has been making life ever harder for users. First they broke all the old extensions that were based on XUL and XBL, so a lot of customizations no longer worked. Then they made PulseAudio mandatory on Linux bug (1345661), so on systems like mine that don't run Pulse, there's no way to get sound in a web page. Forget YouTube or XenoCanto unless you keep another browser around for that purpose.

For those reasons I'd been avoiding the Firefox upgrade, sticking to Debian's firefox-esr ("Extended Support Release"). But when Debian updated firefox-esr to Firefox 56 ESR late last year, performance became unusable. Like half a minute between when you hit Page Down and when the page actually scrolls. It was time to switch browsers.

Pale Moon

I'd been hearing about the Firefox variant Pale Moon. It's a fork of an older Firefox, supposedly with an emphasis on openness and configurability.

I installed the Debian palemoon package. Performance was fine, similar to Firefox before the tragic firefox-56. It was missing a few things -- no built-in PDF viewer or Reader mode -- but I don't use Reader mode that often, and the built-in PDF viewer is an annoyance at least as often as it's a help. (In Firefox it's fairly random about when it kicks in anyway, so I'm never sure whether I'll get the PDF viewer or a Save-as prompt on any given PDF link).

For form and password autofill, for some reason Pale Moon doesn't fill out fields until you type the first letter. For instance, if I had an account with name "myname" and a stored password, when I loaded the page, both fields would be empty, as if there's nothing stored for that page. But typing an 'm' in the username field makes both username and password fields fill in. This isn't something Firefox ever did and I don't particularly like it, but it isn't a major problem.

Then there were some minor irritations, like the fact that profiles were stored in a folder named ~/.moonchild\ productions/ -- super long so it messed up directory listings, and with a space in the middle. PaleMoon was also very insistent about using new tabs for everything, including URLs launched from other programs -- there doesn't seem to be any way to get it to open URLs in the active tab.

I used it as my main browser for several months, and it basically worked. But the irritations started to get to me, and I started considering other options. The final kicker when I saw Pale Moon bug 86, in which, as far as I can tell, someone working on the PaleMoon in OpenBSD tries to use system libraries instead of PaleMoon's patched libraries, and is attacked for it in the bug. Reading the exchange made me want to avoid PaleMoon for two reasons. First, the rudeness: a toxic community that doesn't treat contributors well isn't likely to last long or to have the resources to keep on top of bug and security fixes. Second, the technical question: if Pale Moon's code is so quirky that it can't use standard system libraries and needs a bunch of custom-patched libraries, what does that say about how maintainable it will be in the long term?

Firefox Quantum

Much has been made in the technical press of the latest Firefox, called "Quantum", and its supposed speed. I was a bit dubious of that: it's easy to make your program seem fast after you force everybody into a few years of working with a program that's degraded its performance by an order of magnitude, like Firefox had. After firefox 56, anything would seem fast.

Still, maybe it would at least be fast enough to be usable. But I had trepidations too. What about all those extensions that don't work any more? What about sound not working? Could I live with that?

Debian has no current firefox package, so I downloaded the tarball from, unpacked it, made a new firefox profile and ran it.

Initial startup performance is terrible -- it takes forever to bring up the first window, and I often get a "Firefox seems slow to start up" message at the bottom of the screen, with a link to a page of a bunch of completely irrelevant hints. Still, I typically only start Firefox once a day. Once it's up, performance is a bit laggy but a lot better than firefox-esr 56 was, certainly usable.

I was able to find replacements for most of the really important extensions (the ones that control things like cookies and javascript). But sound, as predicted, didn't work. And there were several other, worse regressions from older Firefox versions.

As it turned out, the only way to make Firefox Quantum usable for me was to build a custom version where I could fix the regressions. To keep articles from being way too long, I'll write about all those issues separately: how to build Firefox, how to fix broken key bindings, and how to fix the PulseAudio problem.

Tags: ,
[ 16:07 May 31, 2018    More tech/web | permalink to this entry | comments ]