Shallow Thoughts : : web

Akkana's Musings on Open Source Computing and Technology, Science, and Nature.

Sat, 08 Feb 2020

Displaying Quotes on a Kiosk -- and Javascript Memory Leaks

The LWV had a 100th anniversary celebration earlier this week. In New Mexico, that included a big celebration at the Roundhouse. One of our members has collected a series of fun facts that she calls "100-Year Minutes". You can see them at lwvnm.org. She asked me if it would be possible to have them displayed somehow during our display at the Roundhouse.

Of course! I said. "Easy, no problem!" I said.

Famous last words.

There are two parts: first, display randomly (or sequentially) chosen quotes with large text in a fullscreen window. Second, set up a computer (the obvious choice is a Raspberry Pi) run the kiosk automatically. This article only covers the first part; I'll write about the Raspberry Pi setup separately.

A Simple Plaintext Kiosk Python Script

When I said "easy" and "no problem", I was imagining writing a little Python program: get text, scale it to the screen, loop. I figured the only hard part would be the scaling. the quotes aren't all the same length, but I want them to be easy to read, so I wanted each quote displayed in the largest font that would let the quote fill the screen.

Indeed, for plaintext it was easy. Using GTK3 in Python, first you set up a PangoCairo layout (Cairo is the way you draw in GTK3, Pango is the font/text rendering library, and a layout is Pango's term for a bunch of text to be rendered). Start with a really big font size, ask PangoCairo how large the layout would render, and if it's so big that it doesn't fit in the available space, reduce the font size and try again. It's not super elegant, but it's easy and it's fast enough. It only took an hour or two for a working script, which you can see at quotekiosk.py.

But some of the quotes had minor HTML formatting. GtkWebkit was orphaned several years ago and was never available for Python 3; the only Python 3 option I know of for displaying HTML is Qt5's QtWebEngine, which is essentially a fully functioning browser window.

Which meant that it seeming made more sense to write the whole kiosk as a web page, with the resizing code in JavaScript. I say "seemingly"; it didn't turn out that way.

JavaScript: Resizing Text to Fit Available Space

The hard part about using JavaScript was the text resizing, since I couldn't use my PangoCairo resizing code.

Much web searching found lots of solutions that resize a single line to fit the width of the screen, plus a lot of hand-waving suggestions that didn't work. I finally found a working solution in a StackOverflow thread: Fit text perfectly inside a div (height and width) without affecting the size of the div. The only one of the three solutions there that actually worked was the jQuery one. It basically does the same thing my original Python script did: check element.scrollHeight and if it overflows, reduce the font size and try again.

I used the jquery version for a little while, but eventually rewrote it to pure javascript so I wouldn't have to keep copying jquery-min.js around.

JS Timers on Slow Machines

There are two types of timers in Javascript: setTimeout, which schedules something to run once N seconds from now, and setInterval, which schedules something to run repeatedly every N seconds. At first I thought I wanted setInterval, since I want the kiosk to keep running, changing its quote every so often.

I coded that, and it worked okay on my laptop, but failed miserably on the Raspberry Pi Zero W. The Pi, even with a lightweight browser like gpreso (let alone chromium), takes so long to load a page and go through the resize-and-check-height loop that by the time it has finally displayed, it's about ready for the timer to fire again. And because it takes longer to scale a big quote than a small one, the longest quotes give you the shortest time to read them.

So I switched to setTimeout instead. Choose a quote (since JavaScript makes it hard to read local files, I used Python to read all the quotes in, turn them into a JSON list and write them out to a file that I included in my JavaScript code), set the text color to the background color so you can't see all the hacky resizing, run the resize loop, set the color back to the foreground color, and only then call setTimeout again:

function newquote() {
    // ... resizing and other slow stuff here

    setTimeout(newquote, 30000);
}

// Display the first page:
newquote();

That worked much better on the Raspberry Pi Zero W, so I added code to resize images in a similar fashion, and added some fancy CSS fade effects that it turned out the Pi was too slow to run, but it looks nice on a modern x86 machine. The full working kiosk code is quotekioska>).

Memory Leaks in JavaScript's innerHTML

I ran it for several hours on my development machine and it looked great. But when I copied it to the Pi, even after I turned off the fades (which looked jerky and terrible on the slow processor), it only ran for ten or fifteen minutes, then crashed. Every time. I tried it in several browsers, but they all crashed after running a while.

The obvious culprit, since it ran fine for a while then crashed, was a memory leak. The next step was to make a minimal test case.

I'm using innerHTML to change the kiosk content, because it's the only way I know of to parse and insert a snippet of HTML that may or may not contain paragraphs and other nodes. This little test page was enough to show the effect:

<h1>innerHTML Leak</h1>

<p id="thecontent">
</p>

<script type="text/javascript">
var i = 0;
function changeContent() {
    var s = "Now we're at number " + i;
    document.getElementById("thecontent").innerHTML = s;
    i += 1;

    setTimeout(changeContent, 2000);
}

changeContent();
</script>

Chromium has a nice performance recording tool that can show you memory leaks. (Firefox doesn't seem to have an equivalent, alas.)

[Chrome performance graph showing innerHTML node leak] To test a leak, go to More Tools > Developer Tools and choose the Performance tab. Load your test page, then click the Record button. Run it for a while, like a couple of minutes, then stop it and you'll see a graph like this (click on the image for a full-size version).

Both the green line, Nodes, and the blue line, JS Heap, are going up. But if you run it for longer, say, ten minutes, the garbage collector eventually runs and the JS Heap line drops back down. The Nodes line never does: the node count just continues going up and up and up no matter how long you run it.

So it looks like that's the culprit: setting innerHTML adds a new node (or several) each time you call it, and those nodes are never garbage collected. No wonder it couldn't run for long on the poor Raspberry Pi Zero with 512Gb RAM (the Pi 3 with 1Gb didn't fare much better).

It's weird that all browsers would have the same memory leak; maybe something about the definition of innerHTML causes it. I'm not enough of a Javascript expert to know, and the experts I was able to find didn't seem to know anything about either why it happened, or how to work around it.

Python html2text

So I gave up on JavaScript and went back to my original Python text kiosk program. After reading in an HTML snippet, I used the Python html2text module to convert the snippet to text, then displayed it. I added image resizing using GdkPixbuf and I was good to go.

quotekiosk.py ran just fine throughout the centennial party, and no one complained about the formatting not being fancy enough. A happy ending, complete with cake and lemonade. But I'm still curious about that JavaScript leak, and whether there's a way to work around it. Anybody know?

Tags: , , ,
[ 18:48 Feb 08, 2020    More tech/web | permalink to this entry | comments ]

Sat, 01 Feb 2020

Migrate a sqlite3 Flask App to Postgresql

The New Mexico legislature is in session again, which means the New Mexico Bill Tracker I wrote last year is back in season. But I guess the word has gotten out, because this year, I started seeing a few database errors. Specifically, "sqlite3.OperationalError: database is locked".

It turns out that even read queries on an sqlite3 database inside flask and sqlalchemy can sometimes keep the database open indefinitely. Consider something like:

    userbills = user.get_bills()    # this does a read query

    # Do some slow operations that don't involve the database at all
    for bill in userbills:
        slow_update_involving_web_scraping(bill)

    # Now bills are all updated; add and commit them.
    # Here's where the write operations start.
    for bill in userbills:
        db.session.add(bill)
    db.session.commit()

I knew better than to open a write query that might keep the database open during all those long running operations. But apparently, when using sqlite3, even the initial query of the database to get the user's bill list opens the database and keeps it open ... until when? Can you close it manually, then reopen it when you're ready? Does it help to call db.session.commit() after the read query? No one seems to know, and it's not obvious how to test to find out.

I've suspected for a long time that sqlite was only a temporary solution. While developing the billtracker, I hit quite a few difficulties where the answer turned out to be "well, this would be easy in a real database, but sqlite doesn't support that". I figured I'd eventually migrate to postgresql. But I'm such a database newbie that I'd been putting it off.

And rightly so. It turns out that migrating an existing database from sqlite3 to postgresql isn't something that gets written about much; I really couldn't find any guides on it. Apparently everybody but me just chooses the right database to begin with? Anyway, here are the steps on Debian. Obviously, install postgresql first.

Create a User and a Database

Postgresql has its own notion of users, which you need to create. At least on Debian, the default is that if you create a postgres user named martha, then the Linux user martha on the same machine can access databases that the postgres user martha has access to. This is controlled by the "peer" auth method, which you can read about in the postgresql documentation on pg_hba.conf.

First su to the postgres Linux user and run psql:

$ sudo su - postgres
$ psql

Inside psql, create a postgresql user with the same name as your flask user, and create a database for that user:

CREATE USER myflaskuser WITH PASSWORD 'password';
ALTER ROLE myflaskuser SET client_encoding TO 'utf8';
ALTER ROLE myflaskuser SET default_transaction_isolation TO 'read committed';
ALTER ROLE myflaskuser SET timezone TO 'UTC';

CREATE DATABASE dbname;
GRANT ALL PRIVILEGES ON DATABASE dbname TO myflaskuser;

If you like, you can also create a user yourusername and give it access to the same database, to make debugging easier.

With the database created, the next step is to migrate the old data from the sqlite database.

pgloader (if you have a very recent pgloader)

Using sqlalchemy in my flask app meant that I could use flask db upgrade to create the database schema in any database I chose. It does a lovely job of creating an empty database. Unfortunately, that's no help if you already have an existing database full of user accounts.

Some people suggested exporting data in either SQL or CSV format, then importing it into postgresql. Bad idea. There are many incompatibilities between the two databases: identifiers that work in sqlite but not in postgresql (like "user", which is a reserved word in postgres but a common table name in flask-based apps), capitalization of column names, incompatible date formats, and probably many more.

A program called pgloader takes care of many (but not all) of the incompatibilities. Create a file -- I'll call it migrate.pgloader -- like this:

load database
    from 'latest-sqlite-file.db'
    into postgresql:///new_db_name

with include drop, quote identifiers, create tables, create indexes, reset sequences

set work_mem to '16MB', maintenance_work_mem to '512 MB';

Then, from a Linux user who has access to the database (e.g. the myflaskuser you created earlier), run pgloader migrate.pgloader.

That worked nicely on my Ubuntu 19.10 desktop, which has pgloader 3.6.1. It failed utterly on the server, which is running Debian stable and pgloader 3.3.2. Building the latest pgloader from source didn't work on Debian either; it's based on Common Lisp, and the older CL on Debian dumped me into some kind of assembly debugger when I tried to build pgloader. Rather than build CL from source too, I looked for another option.

On an Older OS: Use pgloader Remotely

Postgresql can take commands from remote machines. So you can configure postgresql to accept remote connections, then run the migration from a machine with a new enough pgloader version.

There are two files to edit. The location of postgresql's configuration directory varies with version, so do a locate pg_hba.conf to find it. In that directory, first edit pg_hba.conf and add these lines to the end to allow net socket connections from IP4 and IP6:

host  all     all   0.0.0.0/0     md5
host  all     all   ::/0          md5

In the same directory, edit postgresql.conf and search for listen_addr. Comment out the localhost line if it's uncommented, and add this to allow connections from anywhere, not just localhost:

listen_addresses = '*'

Then restart the database with

service postgresql restart

Modify the migrate.pgloader file from the previous section so the "into" line looks like

    into postgresql://username:password@host/dbname
The username there is the postgres username, if you made that different from the Unix username. You need to use a password because postgres is no longer using peer auth (see that postgres documentation file I linked earlier).

Assuming this You're done with the remote connection part. If you don't need remote database connections for your app, you can now edit postgresql.conf, comment out that listen_addresses = '*' line, and restart the database again with service postgresql restart. Don't remove the two lines you added in pg_hba.conf; flask apparently needs them.

You're ready for the migration. Make sure you have the latest copy of the server's sqlite database, then, from your desktop, run:

pgloader migrate.pgloader

Migrate Autoincrements to Sequences

But that's not enough. If you're using any integer primary keys that autoincrement -- a pretty common database model -- postgresql doesn't understand that. Instead, it has sequence objects. You need to define a sequence, tie it to a table, and tell postgresql that when it adds a new object to the table, the default value of id is the maximum number in the corresponding sequence. Here's how to do that for the table named "user":

CREATE SEQUENCE user_id_seq OWNED by "user".id;
ALTER TABLE "user" ALTER COLUMN id SET default nextval('user_id_seq');
SELECT setval(pg_get_serial_sequence('user', 'id'), coalesce(max(id)+1,1), false) FROM "user";

Note the quotes around "user" because otherwise user is a postgresql reserved word. Repeat these three lines for every table in your database, except that you don't need the quotes around any table name except user.

Incidentally, I've been told that using autoincrement/sequence primary keys isn't best practice, because it can be a bottleneck if lots of different objects are being created at once. I used it because all the models I was following when I started with flask worked that way, but eventually I plan try to switch to using some other unique primary key.

Update: Turns out there was another problem with the sequences, and it was pretty annoying. I ended up with a bunch of indices with names like "idx_15517_ix_user_email" when they should have been "ix_user_email". The database superficially worked fine, but it havoc ensues if you ever need to do a flask/sqlalchemy/alembic migration, since sqlalchemy doesn't know anything about those indices with the funny numeric names. It's apparently possible to rename indices in postgresql, but it's a tricky operation that has to be done by hand for each index.

Now the database should be ready to test.

Test

Your flask app probably has something like this in config.py:

    SQLALCHEMY_DATABASE_URI = os.environ.get('DATABASE_URL') or \
        'sqlite:///' + os.path.join(basedir, 'dbname.db')

If so, you can export DATABSE_URL=postgresql:///dbname and then test it as you usually would. If you normally test on a local machine and not on the server, remember you can tell flask's test server to accept connections from remote machines with flask run --host=0.0.0.0

Database Backups

You're backing up your database, right? That's easier in sqlite where you can just copy the db file.

From the command line, you can back up a postgresql database with: pg_dump dbname > dbname-backup.pg You can do that from Python in a subprocess:

    with open(backup_file, 'w') as fp:
        subprocess.call(["pg_dump", dbname], stdout=fp)

Verify You're Using The New Database

I had some problems with that DATABASE_URL setting; I'd never used it so I didn't realize that it wasn't in the right place and didn't actually work. So I ran through my migration steps, changed DATABASE_URL, thought I was done, and realized later that the app was still running off sqlite3.

It's better to know for sure what your app is running. For instance, you can add a route to routes.py that prints details like that.

You can print app.config["SQLALCHEMY_DATABASE_URI"]. That's enough in theory, but I wanted to know for sure. Turns out str(db.session.get_bind()) will print the connection the flask app's database is actually using. So I added a route that prints both, plus some other information about the running app.

Whew! I was a bit surprised that migrating was as tricky as it was, and that there wasn't more documentation for it. Happy migrations, everyone.

Tags: , , , , ,
[ 12:34 Feb 01, 2020    More tech/web | permalink to this entry | comments ]

Thu, 09 Jan 2020

Updating a Persistent Window from Javascript Part 2: A Clever Hack

I wrote about various ways of managing a persistent popup window from Javascript, eventually settling on a postMessage() solution that turned out not to work in QtWebEngine. So I needed another solution.

Data URI

First I tried using a data: URI. In that scheme, you encode a page's full content into the URL. For instance: try this in your browser: data:text/html,Hello%2C%20World!

So for a longer page, you can do something like:

    var htmlhead = '<html>\n'
        + '<head>\n'
        + '<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">\n'
        + '<link rel="stylesheet" type="text/css" href="stylesheet.css">\n'
        + '</head>\n'
        + '\n'
        + '<body>\n'
        + '<div id="mydiv">\n';
     var htmltail = '</div>\n'
        + '</body>\n'
        + '</html>\n';

    var encodedDataURI = encodeURI(htmlhead + noteText + htmltail);

    var notewin = window.open('data:text/html,' + encodedDataURI, "notewindow",
                              "width=800,height=500");

Nice and easy -- and it even works from file: URIs!

Well, sort of works. It turns out it has a problem related to the same-origin problems I saw with postMessage. A data: URI is always opened with an origin of about:blank; and two about:blank origin pages can't talk to each other.

But I don't need them to talk to each other if I'm not using postMessage, do I? Yes, I do. The problem is that stylesheet I included in htmlhead above:

<link rel="stylesheet" type="text/css" href="stylesheet.css">\n'
All browsers I tested refuse to open the stylesheet in the about:blank popup. This seems strange: don't people use stylesheets from other domains fairly often? Maybe it's a behavior special to null (about:blank) origin pages. But in any case, I couldn't find a way to get my data: URI popup to load a stylesheet. So unless I hard-code all the styles I want for the notes page into the Javascript that opens the popup window (and I'd really rather not do that), I can't use data: as a solution.

Clever hack: Use the Same Page, Loaded in a Different Way

That's when I finally came across Remy Sharp's page, Creating popups without HTML files. Remy first explores the data: URI solution, and rejects it because of the cross-origin problem, just as I did. But then he comes up with a clever hack. It's ugly, as he acknowledges ... but it works.

The trick is to create the popup with the URL of the parent page that created it, but with a named anchor appended: parentPage.html#popup. Then, in the Javascript, check whether #popup is in the URL. If not, we're in the parent page and still need to call window.open to create the popup. If it is there, then the JS code is being executed in the popup. In that case, rewrite the page as needed. In my case, since I want the popup to show only whatever is in the div named #notes, and the slide content is all inside a div called #page, I can do this:

function updateNoteWindow() {
    if (window.location.hash.indexOf('#notes') === -1) {
        window.open(window.location + '#notes', 'noteWin',
                    'width=300,height=300');
        return;
    }

    // If here, it's the popup notes window.
    // Remove the #page div
    var pageDiv = document.getElementById("page");
    pageDiv.remove();

    // and rename the #notes div so it will be displayed in a different place
    var notesDiv = document.getElementById("notes");
    notesDiv.id = "fullnotes";
}

It works great, even in file: URIs, and even in QtWebEngine. That's the solution I ended up using.

Tags: , ,
[ 19:44 Jan 09, 2020    More tech/web | permalink to this entry | comments ]

Sun, 05 Jan 2020

Updating a Persistent Window from Javascript Part 1: postMessage

I'm trying to update my htmlpreso HTML presentation slide system to allow for a separate notes window.

Up to now, I've just used display mirroring. I connect to the projector at 1024x768, and whatever is on the first (topmost/leftmost) 1024x768 pixels of my laptop screen shows on the projector. Since my laptop screen is wider than 1024 pixels, I can put notes to myself to the right of the slide, and I'll see them but the audience won't.

That works fine, but I'd like to be able to make the screens completely separate, so I can fiddle around with other things while still displaying a slide on the projector. But since my slides are in HTML, and I still want my presenter notes, that requires putting the notes in a separate window, instead of just to the right of each slide.

The notes for each slide are in a <div id="notes"> on each page. So all I have to do is pop up another browser window and mirror whatever is in that div to the new window, right? Sure ... except this is JavaScript, so nothing is simple. Every little thing is going to be multiple days of hair-tearing frustration, and this was no exception.

I should warn you up front that I eventually found a much simpler way of doing this. I'm documenting this method anyway because it seems useful to be able to communicate between two windows, but if you just want a simple solution for the "pop up notes in another window" problem, stay tuned for Part 2.

Step 0: Give Up On file:

Normally I use file: URLs for presentations. There's no need to run a web server, and in fact, on my lightweight netbook I usually don't start apache2 by default, only if I'm actually working on web development.

But most of the methods of communicating between windows don't work in file URLs, because of the "same-origin policy". That policy is a good security measure: it ensures that a page from innocent-url.com can't start popping up windows with content from evilp0wnU.com without you knowing about it. I'm good with that. The problem is that file: URLs have location.origin of null, and every null-origin window is considered to be a different origin -- even if they're both from the same directory. That makes no sense to me, but there seems to be no way around it. So if I want notes in a separate window, I have to run a web server and use http://localhost.

Step 1: A Separate Window

The first step is to pop up the separate notes window, or get a handle to it if it's already up.

JavaScript offers window.open(), but there's a trick: if you just call notewin = window.open("notewin.html", "notewindow") you'll actually get a new tab, not a new window. If you actually want a window, the secret code for that is to give it a size:

  notewin = window.open("notewin.html", "notewindow",
                        "width=800,height=500");

There's apparently no way to just get a handle to an existing window. The only way is to call window.open(), pop up a new window if it wasn't there before, or reloads it if it's already there.

I saw some articles implying that passing an empty string "" as the first argument would return a handle to an existing window without changing it, but it's not true: in Firefox and Chromium, at least, that makes the existing window load about:blank instead of whatever page it already has. So just give it the same page every time.

Step 2: Figure Out When the Window Has Loaded

There are several ways to change the content in the popup window from the parent, but they all have one problem: if you update the content right away after calling window.open, whatever content you put there will be overwritten immediately when the popup reloads its notewin.html page (or even about:blank). So you need to wait until the popup is finished loading.

That sounds suspiciously easy. Assuming you have a function called updateNoteWinContent(), just do this:

// XXX This Doesn't work:
notewin.addEventListener('load', updateNoteWinContent, false);

Except it turns out the "load" event listener isn't called on reloads, at least not in popups. So this will work the first time, when the note window first pops up, but never after that.

I tried other listeners, like "DOMContentLoaded" and "readystatechange", but none of them are called on reload. Why not? Who knows? It's possible this is because the listener gets set too early, and then is wiped out when the page reloads, but that's just idle speculation.

For a while, I thought I was going to have to resort to an ugly hack: sleep for several seconds in the parent window to give the popup time to load: await new Promise(r => setTimeout(r, 3000)); (requires declaring the calling function as async). This works, but ... ick. Fortunately, there's a better way.

Step 2.5: Simulate onLoad with postMessage

What finally worked was a tricky way to use postMessage() in reverse. I'd already experimented with using postMessage() from the parent window to the popup, but it didn't work because the popup was still loading and wasn't ready for the content.

What works is to go the other way. In the code loaded by the popup (notewin.html in this example), put some code at the end of the page that calls

window.opener.postMessage("Loaded");

Then in the parent, handle that message, and don't try to update the popup's content until you've gotten the message:

function receiveMessageFromPopup(event) {
    console.log("Parent received a message from the notewin:", event.data);
    // Optionally, check whether event.data == "Loaded"
    // if you want to support more than one possible message.

    // Update the "notes" div in the popup notewin:
    var noteDiv = notewin.document.getElementById("notes");
    noteDiv.innerHTML = "

Here is some content.

"; } window.addEventListener("message", receiveMessageFromPopup, false);

Here's a complete working test: Test of Persistent Popup Window.

In the end, though, this didn't solve my presentation problem. I got it all debugged and working, only to discover that postMessage doesn't work in QtWebEngine, so I couldn't use it in my slide presentation app. Fortunately, I found a couple of other ways: stay tuned for Part 2.

(Update: Part 2: A Clever Hack.)

Debugging Multiple Windows: Separate Consoles

A note on debugging: One thing that slowed me down was that JS I put in the popup didn't seem to be running: I never saw its console.log() messages. It took me a while to realize that each window has its own web console, both in Firefox and Chromium. So you have to wait until the popup has opened before you can see any debugging messages for it. Even then, the popup window doesn't have a menu, and its context menu doesn't offer a console window option. But it does offer Inspect element, which brings up a Developer Tools window where you can click on the Console tab to see errors and debugging messages.

Tags: , ,
[ 20:29 Jan 05, 2020    More tech/web | permalink to this entry | comments ]

Sun, 29 Jul 2018

Building Firefox: Changing the App Name

In my several recent articles about building Firefox from source, I omitted one minor change I made, which will probably sound a bit silly. A self-built Firefox thinks its name is "Nightly", so, for example, the Help menu includes About Nightly.

Somehow I found that unreasonably irritating. It's not a nightly build; in fact, I hope to build it as seldom as possible, ideally only after a git pull when new versions are released. Yet Firefox shows its name in quite a few places, so you're constantly faced with that "Nightly". After all the work to build Firefox, why put up with that?

To find where it was coming from, I used my recursive grep alias which skips the obj- directory plus things like object files and metadata. This is how I define it in my .zshrc (obviously, not all of these clauses are necessary for this Firefox search), and then how I called it to try to find instances of "Nightly" in the source:

gr() {
  find . \( -type f -and -not -name '*.o' -and -not -name '*.so' -and -not -name '*.a' -and -not -name '*.pyc' -and -not -name '*.jpg' -and -not -name '*.JPG' -and -not -name '*.png' -and -not -name '*.xcf*' -and -not -name '*.gmo' -and -not -name '.intltool*' -and -not -name '*.po' -and -not -name 'po' -and -not -name '*.tar*' -and -not -name '*.zip' -or -name '.metadata' -or -name 'build' -or -name 'obj-*' -or -name '.git' -or -name '.svn' -prune \) -print0 | xargs -0 grep $* /dev/null
}

gr Nightly | grep -v '//' | grep -v '#' | grep -v isNightly  | grep test | grep -v task | fgrep -v .js | fgrep -v .cpp | grep -v mobile >grep.out

Even with all those exclusions, that still ends up printing an enormous list. But it turns out all the important hits are in the browser directory, so you can get away with running it from there rather than from the top level.

I found a bunch of likely files that all had very similar "Nightly" lines in them:

Since I didn't know which one was relevant, I changed each of them to slightly different names, then rebuilt and checked to see which names I actually saw while running the browser.

It turned out that browser/branding/unofficial/locales/en-US/brand.dtd is the file that controls the application name in the Help menu and in Help->About -- though the title of the About window is still "Nightly" and I haven't found what controls that.

branding/unofficial/locales/en-US/brand.ftl controls the "Nightly" references in the Edit->Preferences window.

I don't know what all the others do. There may be other instances of "Nightly" that appear elsewhere in the app, the other files, but I haven't seen them yet.

Past Firefox building articles: Building Firefox Quantum; Building Firefox for ALSA (non PulseAudio) Sound; Firefox Quantum: Fixing Ctrl W (or other key bindings).

Tags: , , ,
[ 18:23 Jul 29, 2018    More tech/web | permalink to this entry | comments ]

Sat, 07 Jul 2018

Script to modify omni.ja for a custom Firefox

A quick followup to my article on Modifying Firefox Files Inside omni.ja:

The steps for modifying the file are fairly easy, but they have to be done a lot.

First there's the problem of Firefox updates: if a new omni.ja is part of the update, then your changes will be overwritten, so you'll have to make them again on the new omni.ja.

But, worse, even aside from updates they don't stay changed. I've had Ctrl-W mysteriously revert back to its old wired-in behavior in the middle of a Firefox session. I'm still not clear how this happens: I speculate that something in Firefox's update mechanism may allow parts of omni.ja to be overridden, even though I was told by Mike Kaply, the onetime master of overlays, that they weren't recommended any more (at least by users, though that doesn't necessarily mean they're not used for updates).

But in any case, you can be browsing merrily along and suddenly one of your changes doesn't work any more, even though the change is still right there in browser/omni.ja. And the only fix I've found so far is to download a new Firefox and re-apply the changes. Re-applying them to the current version doesn't work -- they're already there. And it doesn't help to keep the tarball you originally downloaded around so you can re-install that; firefox updates every week or two so that version is guaranteed to be out of date.

All this means that it's crazy not to script the omni changes so you can apply them easily with a single command. So here's a shell script that takes the path to the current Firefox, unpacks browser/omni.ja, makes a couple of simple changes and re-packs it. I called it kitfox-patch since I used to call my personally modified Firefox build "Kitfox".

Of course, if your changes are different from mine you'll want to edit the script to change the sed commands.

I hope eventually to figure out how it is that omni.ja changes stop working, and whether it's an overlay or something else, and whether there's a way to re-apply fixes without having to download a whole new Firefox. If I figure it out I'll report back.

Tags: ,
[ 15:01 Jul 07, 2018    More tech/web | permalink to this entry | comments ]

Sat, 23 Jun 2018

Modifying Firefox Files Inside Omni.ja

My article on Fixing key bindings in Firefox Quantum by modifying the source tree got attention from several people who offered helpful suggestions via Twitter and email on how to accomplish the same thing using just files in omni.ja, so it could be done without rebuilding the Firefox source. That would be vastly better, especially for people who need to change something like key bindings or browser messages but don't have a souped-up development machine to build the whole browser.

Brian Carpenter had several suggestions and eventually pointed me to an old post by Mike Kaply, Don’t Unpack and Repack omni.ja[r] that said there were better ways to override specific files.

Unfortunately, Mike Kaply responded that that article was written for XUL extensions, which are now obsolete, so the article ought to be removed. That's too bad, because it did sound like a much nicer solution. I looked into trying it anyway, but the instructions it points to for Overriding specific files is woefully short on detail on how to map a path inside omni.ja like chrome://package/type/original-uri.whatever, to a URL, and the single example I could find was so old that the file it referenced didn't exist at the same location any more. After a fruitless half hour or so, I took Mike's warning to heart and decided it wasn't worth wasting more time chasing something that wasn't expected to work anyway. (If someone knows otherwise, please let me know!)

But then Paul Wise offered a solution that actually worked, as an easy to follow sequence of shell commands. (I've changed some of them very slightly.)

$ tar xf ~/Tarballs/firefox-60.0.2.tar.bz2
  # (This creates a "firefox" directory inside the current one.)

$ mkdir omni
$ cd omni

$ unzip -q ../firefox/browser/omni.ja
warning [../firefox-60.0.2/browser/omni.ja]:  34187320 extra bytes at beginning or within zipfile
  (attempting to process anyway)
error [../firefox-60.0.2/browser/omni.ja]:  reported length of central directory is
  -34187320 bytes too long (Atari STZip zipfile?  J.H.Holm ZIPSPLIT 1.1
  zipfile?).  Compensating...
zsh: exit 2     unzip -q ../firefox-60.0.2/browser/omni.ja

$ sed -i 's/or enter address/or just twiddle your thumbs/' chrome/en-US/locale/browser/browser.dtd chrome/en-US/locale/browser/browser.properties

I was a little put off by all the warnings unzip gave, but kept going.

Of course, you can just edit those two files rather than using sed; but the sed command was Paul's way of being very specific about the changes he was suggesting, which I appreciated.

Use these flags to repackage omni.ja:

$ zip -qr9XD ../omni.ja *

I had tried that before (without the q since I like to see what zip and tar commands are doing) and hadn't succeeded. And indeed, when I listed the two files, the new omni.ja I'd just packaged was about a third the size of the original:

$ ls -l ../omni.ja ../firefox-60.0.2/browser/omni.ja
-rw-r--r-- 1 akkana akkana 34469045 Jun  5 12:14 ../firefox/browser/omni.ja
-rw-r--r-- 1 akkana akkana 11828315 Jun 17 10:37 ../omni.ja

But still, it's worth a try:

$ cp ../omni.ja ../firefox/browser/omni.ja

Then run the new Firefox. I have a spare profile I keep around for testing, but Paul's instructions included a nifty way of running with a brand new profile and it's definitely worth knowing:

$ cd ../firefox

$ MOZILLA_DISABLE_PLUGINS=1 ./firefox -safe-mode -no-remote -profile $(mktemp -d tmp-firefox-profile-XXXXXXXXXX) -offline about:blank

Also note the flags like safe-mode and no-remote, plus disabling plugins -- all good ideas when testing something new.

And it worked! When I started up, I got the new message, "Search or just twiddle your thumbs", in the URL bar.

Fixing Ctrl-W

Of course, now I had to test it with my real change. Since I like Paul's way of using sed to specify exactly what changes to make, here's a sed version of my Ctrl-W fix:

$ sed -i '/key_close/s/ reserved="true"//' chrome/browser/content/browser/browser.xul

Then run it. To test Ctrl-W, you need a website that includes a text field you can type in, so -offline isn't an option unless you happen to have a local web page that includes some text fields. Google is an easy way to test ... and you might as well re-use that firefox profile you just made rather than making another one:

$ MOZILLA_DISABLE_PLUGINS=1 ./firefox -safe-mode -no-remote -profile tmp-firefox-profile-* https://google.com

I typed a few words in the google search field that came up, deleted them with Ctrl-W -- all was good! Thanks, Paul! And Brian, and everybody else who sent suggestions.

Why are the sizes so different?

I was still puzzled by that threefold difference in size between the omni.ja I repacked and the original that comes with Firefox. Was something missing? Paul had the key to that too: use zipinfo on both versions of the file to see what differed. Turned out Mozilla's version, after a long file listing, ends with

2650 files, 33947999 bytes uncompressed, 33947999 bytes compressed:  0.0%
while my re-packaged version ends with
2650 files, 33947969 bytes uncompressed, 11307294 bytes compressed:  66.7%

So apparently Mozilla's omni.ja is using no compression at all. It may be that that makes it start up a little faster; but Quantum takes so long to start up that any slight difference in uncompressing omni.ja isn't noticable to me.

I was able to run through this whole procedure on my poor slow netbook, the one where building Firefox took something like 15 hours ... and in a few minutes I had a working modified Firefox. And with the sed command, this is all scriptable, so it'll be easy to re-do whenever Firefox has a security update. Win!

Update: I have a simple shell script to do this: Script to modify omni.ja for a custom Firefox.

Tags: ,
[ 20:37 Jun 23, 2018    More tech/web | permalink to this entry | comments ]

Thu, 14 Jun 2018

Firefox Quantum: Fixing Ctrl W (or other key bindings)

When I first tried switching to Firefox Quantum, the regression that bothered me most was Ctrl-W, which I use everywhere as word erase (try it -- you'll get addicted, like I am). Ctrl-W deletes words in the URL bar; but if you type Ctrl-W in a text field on a website, like when editing a bug report or a "Contact" form, it closes the current tab, losing everything you've just typed. It's always worked in Firefox in the past; this is a new problem with Quantum, and after losing a page of typing for about the 20th time, I was ready to give up and find another browser.

A web search found plenty of people online asking about key bindings like Ctrl-W, but apparently since the deprecation of XUL and XBL extensions, Quantum no longer offers any way to change or even just to disable its built-in key bindings.

I wasted a few days chasing a solution inspired by this clever way of remapping keys only for certain windows using xdotool getactivewindow; I even went so far as to write a Python script that intercepts keystrokes, determines the application for the window where the key was typed, and remaps it if the application and keystroke match a list of keys to be remapped. So if Ctrl-W is typed in a Firefox window, Firefox will instead receive Alt-Backspace. (Why not just type Alt-Backspace, you ask? Because it's much harder to type, can't be typed from the home position, and isn't in the same place on every keyboard the way W is.)

But sadly, that approach didn't work because it turned out my window manager, Openbox, acts on programmatically-generated key bindings as well as ones that are actually typed. If I type a Ctrl-W and it's in Firefox, that's fine: my Python program sees it, generates an Alt-Backspace and everything is groovy. But if I type a Ctrl-W in any other application, the program doesn't need to change it, so it generates a Ctrl-W, which Openbox sees and calls the program again, and you have an infinite loop. I couldn't find any way around this. And admittedly, it's a horrible hack having a program intercept every keystroke. So I needed to fix Firefox somehow.

But after spending days searching for a way to customize Firefox's keys, to no avail, I came to the conclusion that the only way was to modify the source code and rebuild Firefox from source.

Ironically, one of the snags I hit in building it was that I'd named my key remapper "pykey.py", and it was still in my PYTHONPATH; it turns out the Firefox build also has a module called pykey.py and mine was interfering. But eventually I got the build working.

Firefox Key Bindings

I was lucky: building was the only hard part, because a very helpful person on Mozilla's #introduction IRC channel pointed me toward the solution, saving me hours of debugging. Edit browser/base/content/browser-sets.inc around line 240 and remove reserved="true" from key_closeWindow. It turned out I needed to remove reserved="true" from the adjacent key_close line as well.

Another file that's related, but more general, is nsXBLWindowKeyHandler.cpp around line 832; but I didn't need that since the simpler fix worked.

Transferring omni.ja -- or Not

In theory, since browser-sets.inc isn't compiled C++, it seems like you should be able to make this fix without building the whole source tree. In an actual Firefox release, browser-sets.inc is part of omni.ja, and indeed if you unpack omni.ja you'll see the key_closeWindow and key_close lines. So it seems like you ought to be able to regenerate omni.ja without rebuilding all the C++ code.

Unfortunately, in practice omni.ja is more complicated than that. Although you can unzip it and edit the files, if you zip it back up, Firefox doesn't see it as valid. I guess that's why they renamed it .ja: long ago it used to be omni.jar and, like other .jar files, was a standard zip archive that you could edit. But the new .ja file isn't documented anywhere I could find, and all the web discussions I found on how to re-create it amounted to "it's complicated, you probably don't want to try".

And you'd think that I could take the omni.ja file from my desktop machine, where I built Firefox, and copy it to my laptop, replacing the omni.ja file from a released copy of Firefox. But no -- somehow, it isn't seen, and the old key bindings are still active. They must be duplicated somewhere else, and I haven't figured out where.

It sure would be nice to have a way to transfer an omni.ja. Building Firefox on my laptop takes nearly a full day (though hopefully rebuilding after pulling minor security updates won't be quite so bad). If anyone knows of a way, please let me know!

Tags: , ,
[ 16:45 Jun 14, 2018    More tech/web | permalink to this entry | comments ]