Shallow Thoughts

Akkana's Musings on Open Source Computing and Technology, Science, and Nature.

Mon, 17 Feb 2020

A Couple of Nice Hikes

[The lunch spot on Lion Cave Mesa] We've had some wild weather recently. Two weeks ago, our weekly hiking group was sscheduled to go on a hike in sunny White Rock that Dave and I had proposed, a few miles from home. Then the night before the hike, we got our heaviest snowstorm of the year so far.

Sounded like a great opportunity to test those new ice spikes (for shoes) I'd ordered on eBay. We went down Lion Cave Canyon, around the mesa and up Water Canyon, then climbed up to the top of the mesa and went out to the end to a lunch spot with a panoramic view of Water Canyon and the Sangre de Cristos.

[Walking the Neck] Then back across the narrow neck of the mesa. The temperature was just about perfect for hiking with the sun and the snow. The ice spikes worked perfectly -- the snow wasn't deep enough to need snowshoes, but there were plenty of places where it would have been slippery without the spikes.

[Snow Bumps] We also had fun speculating on the cause of the "snow bumps" that formed around the grama grass stems.

[Dramatic light in Pueblo Canyoo] Now, two weeks later, most of the snow is gone and it's a beautiful day with a high of 60. We headed out for a short exploration in Pueblo Canyon, looking for the old airport that some folks in the R/C flying club thought might make a good flying site.

Some clouds moved in while we were walking, making for dramatic views of the cliffs. I just never get tired of the way the changing light plays on the mesas and canyons.

[Dramatic light in Pueblo Canyoo] We didn't find the old airport -- more exploration needed! -- but we did find the new connector to the Tent Rocks Trail, where the Youth Conservation Corps has been busy with trailwork in Pueblo Canyon. And we explored the remains of an old road -- below Anderson Overlook: possibly the original horse/mule road that they used in the Ranch School days before the Manhattan Project.

Another beautiful day in Los Alamos.

Tags: ,
[ 19:58 Feb 17, 2020    More nature | permalink to this entry | comments ]

Wed, 12 Feb 2020

A Raspberry Pi Kiosk

After writing a simple kiosk of rotating quotes and images, I wanted to set up a Raspberry Pi to run the kiosk automatically, without needing a keyboard or any on-site configuration.

The Raspbian Desktop: Too Hard to Configure

Unlike my usual Raspberry Pi hacks, the kiosk would need a monitor and a window system. So instead of my usual Raspbian Lite install, I opted for a full Raspbian desktop image.

Mistake. First, the Raspbian desktop is very slow. I intended to use a Pi Zero W for the kiosk, but even on a Pi 3 the desktop was sluggish.

More important, the desktop is difficult to configure. For instance, a kiosk needs to keep the screen on, so I needed to disable the automatic screen blanking. There are threads all over the web asking how to disable screen blanking, with lots of solutions that no longer apply because Raspbian keeps changing where desktop configuration files are stored.

Incredibly, the official Raspbian answer for how to disable screen blanking in the desktop — I can hardly type, I'm laughing so hard — is: install xscreensaver, which will then add a configuration option to turn off the screensaver. (I actually tried that just to see if it would work, but changed my mind when I saw the long list of dependencies xscreensaver was going to pull in.)

I never did find a way to disable screen blanking, and after a few hours of fighting with it, I decided it wasn't worth it. Setting up Raspbian Lite is so much easier and I already knew how to do it. If I didn't, Die Antwort has a nice guide, Setup a Raspberry Pi to run a Web Browser in Kiosk Mode, that uses my preferred window manager, Openbox. Here are my steps, starting with a freshly burned Raspbian Lite SD card.

Set Up Raspbian Lite with Network and SSH

I wanted to use ssh on my home network while debugging, even though the final kiosk won't need a network. The easiest way to do that is to mount the first partition:

sudo mount /dev/sdX1 /mnt
(sdX is wherever the card shows up on your machine, e.g. sdB) and create two files. First, an empty file named ssh
touch /mnt/ssh

Second, create a file named wpa_supplicant.conf with the settings for your local network:

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1

network={
	ssid="MY-WIFI-SSID"
	psk="MY-WIFI-PASSWORD"
	priority=10
}

Then unmount that partition:

sudo umount /mnt

Copy the Kiosk Files into /home/pi

The second partition on a Raspbian card is the root filesystem, including /home/pi, the pi user's home dictory. Mount /dev/sdX2, copy your kiosk code into /home/pi, and chown the code to the pi user. If you don't know what that means or how to do that, you can skip this step and load the code onto the Pi later once it's up and running, over the network or via a USB stick.

Unmount the SD card and move it to the Raspberry Pi.

Raspbian First-boot Configuration

Boot the Pi with a monitor attached, log in as the pi user, run sudo raspi-config, and:

So the installation won't become too bloated, I like to create the file /etc/apt/apt.conf containing:

APT::Install-Recommends "false";
APT::Install-Suggests "false";
(That's the equivalent of the --no-install-recommends in the Die Antwort guide.)

Update the OS, and install the packages needed to run X, the Openbox window manager, a terminal (I used xterm), and a text editor (I used vim; if you're not familiar with Linux text editors, pico is more beginner-friendly). If you're in a hurry, you can skip the update and dist-upgrade steps.

$ sudo apt update
$ sudo apt dist-upgrade
$ sudo apt install xserver-xorg x11-xserver-utils xinit openbox xterm vim

I was surprised how little time this took: even with all of the X dependencies, the whole thing took less than twenty minutes, compared to the several hours it had taken to dist-upgrade all the packages on the full Raspbian desktop.

Install any Kiosk-specific Packages

Install any packages you need to run your kiosk. My kiosk was based on Python 3 and GTK 3:

sudo apt install python3-cairo python3-gi python3-gi-cairo \
         libgirepository-1.0-1 gir1.2-glib-2.0 python3-html2text
(This also pulled in gir1.2-atk-1.0, gir1.2-freedesktop, gir1.2-gdkpixbuf-2.0, gir1.2-pango-1.0, and gir1.2-gtk-3.0, but I don't think I had to specify any of them explicitly.)

Configure Openbox

Create the Openbox configuration directory:

mkdir -p .config/openbox
Create .config/openbox/autostart containing:
# Disable screen saver/screen blanking/power management
xset s off
xset s noblank
xset -dpms

# Start a terminal
xterm &

Save the file, and test to make sure you can run X:

$ startx

You should see a black screen, a mouse pointer, and after a few seconds, a small xterm window in the center of the screen. You can use the xterm to fiddle with things you want to change, or you can right-click anywhere outside the xterm window to get a menu that will let you exit X and go back to the bare console.

Test Your Kiosk

With X running, you can run your kiosk command. Don't change directories first; the pi user will be /home/pi ($HOME) after automatically logging in, so make sure you can run from there. For instance, I can run my kiosk with:

$HOME/src/scripts/quotekiosk.py $HOME/100-min-kiosk/slideshow/* $HOME/100-min-kiosk/quotes/*.html

Once the command works, edit .config/openbox/autostart and add your command at the end, after the xterm line, with an ampersand (&) after it. Keep the xterm line in place so you'll have a way to recover if things go wrong.

Configure X to Start When the Pi User Logs In

You've already set up the Pi user to be logged in automatically when the machine boots, but pi needs to start X upon login. Create the file .bash_profile containing:

[[ -z $DISPLAY && $XDG_VTNR -eq 1 ]] && startx

You should be ready to go. Reboot, and the Pi should boot up in kiosk mode.

Run in a Loop

Everything working? For extra security, you might want to tweak the autostart file to run your kiosk in a loop. That way, even if the kiosk code crashes for some reason, it will be restarted.

while :
do
    $HOME/src/scripts/quotekiosk.py $HOME/100-min-kiosk/slideshow/* $HOME/100-min-kiosk/quotes/*.html
done

Don't do this until after you've tested everything else; it's hard to debug with the kiosk constantly popping up on top of other windows.

Get Rid of that Pesky Cursor

You might also want to remove that annoying mouse pointer arrow in the middle of the screen. Editing that startx line you just added to .bash_profile:

[[ -z $DISPLAY && $XDG_VTNR -eq 1 ]] && startx -- -nocursor

This step comes last — because once you've disabled the cursor, it will be difficult to use the machine interactively since you won't be able to see where your mouse is. (If you need to make changes later, you can ssh in from another machine, mount the Raspbian SD card on another machine, or use Ctrl-Alt-F2 to switch to a console window where you can edit files.)

... But It's Still Not Quite Hands-Off

The Pi is now set up to work automatically: just plug it in. The problem was the monitor. Someone contributed a TV, but it turned out to be a "smart TV", and it had its own ideas about what it would connect to. Sometimes the HDMI ports worked, sometimes it refused to display anything, and even when it worked, it randomly brightened and dimmed so that the screen was often too dim to see.

So I contributed my old 20" monitor. Everything worked fine at the demo the night before, and I handed it off to the people who were going to be there early for setup. When I arrived at the Roundhouse the next day, there was my monitor, displaying "No Signal". Apparently, while setting it up, someone had bumped the monitor's "Input Source" button; and of course no one there was up to the task of diagnosing that difficult problem. And no one bothered to call me and ask.

Once I arrived, I pressed the Source button a couple of times and the kiosk display was up and running for the rest of the day. Sigh. I can write kiosk software and set up Raspberry Pis; but predicting potential issues non-technical users might encounter is still beyond me.

Tags:
[ 11:08 Feb 12, 2020    More tech | permalink to this entry | comments ]

Sat, 08 Feb 2020

Displaying Quotes on a Kiosk -- and Javascript Memory Leaks

The LWV had a 100th anniversary celebration earlier this week. In New Mexico, that included a big celebration at the Roundhouse. One of our members has collected a series of fun facts that she calls "100-Year Minutes". You can see them at lwvnm.org. She asked me if it would be possible to have them displayed somehow during our display at the Roundhouse.

Of course! I said. "Easy, no problem!" I said.

Famous last words.

There are two parts: first, display randomly (or sequentially) chosen quotes with large text in a fullscreen window. Second, set up a computer (the obvious choice is a Raspberry Pi) run the kiosk automatically. This article only covers the first part; I'll write about the Raspberry Pi setup separately.

A Simple Plaintext Kiosk Python Script

When I said "easy" and "no problem", I was imagining writing a little Python program: get text, scale it to the screen, loop. I figured the only hard part would be the scaling. the quotes aren't all the same length, but I want them to be easy to read, so I wanted each quote displayed in the largest font that would let the quote fill the screen.

Indeed, for plaintext it was easy. Using GTK3 in Python, first you set up a PangoCairo layout (Cairo is the way you draw in GTK3, Pango is the font/text rendering library, and a layout is Pango's term for a bunch of text to be rendered). Start with a really big font size, ask PangoCairo how large the layout would render, and if it's so big that it doesn't fit in the available space, reduce the font size and try again. It's not super elegant, but it's easy and it's fast enough. It only took an hour or two for a working script, which you can see at quotekiosk.py.

But some of the quotes had minor HTML formatting. GtkWebkit was orphaned several years ago and was never available for Python 3; the only Python 3 option I know of for displaying HTML is Qt5's QtWebEngine, which is essentially a fully functioning browser window.

Which meant that it seeming made more sense to write the whole kiosk as a web page, with the resizing code in JavaScript. I say "seemingly"; it didn't turn out that way.

JavaScript: Resizing Text to Fit Available Space

The hard part about using JavaScript was the text resizing, since I couldn't use my PangoCairo resizing code.

Much web searching found lots of solutions that resize a single line to fit the width of the screen, plus a lot of hand-waving suggestions that didn't work. I finally found a working solution in a StackOverflow thread: Fit text perfectly inside a div (height and width) without affecting the size of the div. The only one of the three solutions there that actually worked was the jQuery one. It basically does the same thing my original Python script did: check element.scrollHeight and if it overflows, reduce the font size and try again.

I used the jquery version for a little while, but eventually rewrote it to pure javascript so I wouldn't have to keep copying jquery-min.js around.

JS Timers on Slow Machines

There are two types of timers in Javascript: setTimeout, which schedules something to run once N seconds from now, and setInterval, which schedules something to run repeatedly every N seconds. At first I thought I wanted setInterval, since I want the kiosk to keep running, changing its quote every so often.

I coded that, and it worked okay on my laptop, but failed miserably on the Raspberry Pi Zero W. The Pi, even with a lightweight browser like gpreso (let alone chromium), takes so long to load a page and go through the resize-and-check-height loop that by the time it has finally displayed, it's about ready for the timer to fire again. And because it takes longer to scale a big quote than a small one, the longest quotes give you the shortest time to read them.

So I switched to setTimeout instead. Choose a quote (since JavaScript makes it hard to read local files, I used Python to read all the quotes in, turn them into a JSON list and write them out to a file that I included in my JavaScript code), set the text color to the background color so you can't see all the hacky resizing, run the resize loop, set the color back to the foreground color, and only then call setTimeout again:

function newquote() {
    // ... resizing and other slow stuff here

    setTimeout(newquote, 30000);
}

// Display the first page:
newquote();

That worked much better on the Raspberry Pi Zero W, so I added code to resize images in a similar fashion, and added some fancy CSS fade effects that it turned out the Pi was too slow to run, but it looks nice on a modern x86 machine. The full working kiosk code is quotekioska>).

Memory Leaks in JavaScript's innerHTML

I ran it for several hours on my development machine and it looked great. But when I copied it to the Pi, even after I turned off the fades (which looked jerky and terrible on the slow processor), it only ran for ten or fifteen minutes, then crashed. Every time. I tried it in several browsers, but they all crashed after running a while.

The obvious culprit, since it ran fine for a while then crashed, was a memory leak. The next step was to make a minimal test case.

I'm using innerHTML to change the kiosk content, because it's the only way I know of to parse and insert a snippet of HTML that may or may not contain paragraphs and other nodes. This little test page was enough to show the effect:

<h1>innerHTML Leak</h1>

<p id="thecontent">
</p>

<script type="text/javascript">
var i = 0;
function changeContent() {
    var s = "Now we're at number " + i;
    document.getElementById("thecontent").innerHTML = s;
    i += 1;

    setTimeout(changeContent, 2000);
}

changeContent();
</script>

Chromium has a nice performance recording tool that can show you memory leaks. (Firefox doesn't seem to have an equivalent, alas.)

[Chrome performance graph showing innerHTML node leak] To test a leak, go to More Tools > Developer Tools and choose the Performance tab. Load your test page, then click the Record button. Run it for a while, like a couple of minutes, then stop it and you'll see a graph like this (click on the image for a full-size version).

Both the green line, Nodes, and the blue line, JS Heap, are going up. But if you run it for longer, say, ten minutes, the garbage collector eventually runs and the JS Heap line drops back down. The Nodes line never does: the node count just continues going up and up and up no matter how long you run it.

So it looks like that's the culprit: setting innerHTML adds a new node (or several) each time you call it, and those nodes are never garbage collected. No wonder it couldn't run for long on the poor Raspberry Pi Zero with 512Gb RAM (the Pi 3 with 1Gb didn't fare much better).

It's weird that all browsers would have the same memory leak; maybe something about the definition of innerHTML causes it. I'm not enough of a Javascript expert to know, and the experts I was able to find didn't seem to know anything about either why it happened, or how to work around it.

Python html2text

So I gave up on JavaScript and went back to my original Python text kiosk program. After reading in an HTML snippet, I used the Python html2text module to convert the snippet to text, then displayed it. I added image resizing using GdkPixbuf and I was good to go.

quotekiosk.py ran just fine throughout the centennial party, and no one complained about the formatting not being fancy enough. A happy ending, complete with cake and lemonade. But I'm still curious about that JavaScript leak, and whether there's a way to work around it. Anybody know?

Tags: , , ,
[ 18:48 Feb 08, 2020    More tech/web | permalink to this entry | comments ]

Sat, 01 Feb 2020

Migrate a sqlite3 Flask App to Postgresql

The New Mexico legislature is in session again, which means the New Mexico Bill Tracker I wrote last year is back in season. But I guess the word has gotten out, because this year, I started seeing a few database errors. Specifically, "sqlite3.OperationalError: database is locked".

It turns out that even read queries on an sqlite3 database inside flask and sqlalchemy can sometimes keep the database open indefinitely. Consider something like:

    userbills = user.get_bills()    # this does a read query

    # Do some slow operations that don't involve the database at all
    for bill in userbills:
        slow_update_involving_web_scraping(bill)

    # Now bills are all updated; add and commit them.
    # Here's where the write operations start.
    for bill in userbills:
        db.session.add(bill)
    db.session.commit()

I knew better than to open a write query that might keep the database open during all those long running operations. But apparently, when using sqlite3, even the initial query of the database to get the user's bill list opens the database and keeps it open ... until when? Can you close it manually, then reopen it when you're ready? Does it help to call db.session.commit() after the read query? No one seems to know, and it's not obvious how to test to find out.

I've suspected for a long time that sqlite was only a temporary solution. While developing the billtracker, I hit quite a few difficulties where the answer turned out to be "well, this would be easy in a real database, but sqlite doesn't support that". I figured I'd eventually migrate to postgresql. But I'm such a database newbie that I'd been putting it off.

And rightly so. It turns out that migrating an existing database from sqlite3 to postgresql isn't something that gets written about much; I really couldn't find any guides on it. Apparently everybody but me just chooses the right database to begin with? Anyway, here are the steps on Debian. Obviously, install postgresql first.

Create a User and a Database

Postgresql has its own notion of users, which you need to create. At least on Debian, the default is that if you create a postgres user named martha, then the Linux user martha on the same machine can access databases that the postgres user martha has access to. This is controlled by the "peer" auth method, which you can read about in the postgresql documentation on pg_hba.conf.

First su to the postgres Linux user and run psql:

$ sudo su - postgres
$ psql

Inside psql, create a postgresql user with the same name as your flask user, and create a database for that user:

CREATE USER myflaskuser WITH PASSWORD 'password';
ALTER ROLE myflaskuser SET client_encoding TO 'utf8';
ALTER ROLE myflaskuser SET default_transaction_isolation TO 'read committed';
ALTER ROLE myflaskuser SET timezone TO 'UTC';

CREATE DATABASE dbname;
GRANT ALL PRIVILEGES ON DATABASE dbname TO myflaskuser;

If you like, you can also create a user yourusername and give it access to the same database, to make debugging easier.

With the database created, the next step is to migrate the old data from the sqlite database.

pgloader (if you have a very recent pgloader)

Using sqlalchemy in my flask app meant that I could use flask db upgrade to create the database schema in any database I chose. It does a lovely job of creating an empty database. Unfortunately, that's no help if you already have an existing database full of user accounts.

Some people suggested exporting data in either SQL or CSV format, then importing it into postgresql. Bad idea. There are many incompatibilities between the two databases: identifiers that work in sqlite but not in postgresql (like "user", which is a reserved word in postgres but a common table name in flask-based apps), capitalization of column names, incompatible date formats, and probably many more.

A program called pgloader takes care of many (but not all) of the incompatibilities. Create a file -- I'll call it migrate.pgloader -- like this:

load database
    from 'latest-sqlite-file.db'
    into postgresql:///new_db_name

with include drop, quote identifiers, create tables, create indexes, reset sequences

set work_mem to '16MB', maintenance_work_mem to '512 MB';

Then, from a Linux user who has access to the database (e.g. the myflaskuser you created earlier), run pgloader migrate.pgloader.

That worked nicely on my Ubuntu 19.10 desktop, which has pgloader 3.6.1. It failed utterly on the server, which is running Debian stable and pgloader 3.3.2. Building the latest pgloader from source didn't work on Debian either; it's based on Common Lisp, and the older CL on Debian dumped me into some kind of assembly debugger when I tried to build pgloader. Rather than build CL from source too, I looked for another option.

On an Older OS: Use pgloader Remotely

Postgresql can take commands from remote machines. So you can configure postgresql to accept remote connections, then run the migration from a machine with a new enough pgloader version.

There are two files to edit. The location of postgresql's configuration directory varies with version, so do a locate pg_hba.conf to find it. In that directory, first edit pg_hba.conf and add these lines to the end to allow net socket connections from IP4 and IP6:

host  all     all   0.0.0.0/0     md5
host  all     all   ::/0          md5

In the same directory, edit postgresql.conf and search for listen_addr. Comment out the localhost line if it's uncommented, and add this to allow connections from anywhere, not just localhost:

listen_addresses = '*'

Then restart the database with

service postgresql restart

Modify the migrate.pgloader file from the previous section so the "into" line looks like

    into postgresql://username:password@host/dbname
The username there is the postgres username, if you made that different from the Unix username. You need to use a password because postgres is no longer using peer auth (see that postgres documentation file I linked earlier).

Assuming this You're done with the remote connection part. If you don't need remote database connections for your app, you can now edit postgresql.conf, comment out that listen_addresses = '*' line, and restart the database again with service postgresql restart. Don't remove the two lines you added in pg_hba.conf; flask apparently needs them.

You're ready for the migration. Make sure you have the latest copy of the server's sqlite database, then, from your desktop, run:

pgloader migrate.pgloader

Migrate Autoincrements to Sequences

But that's not enough. If you're using any integer primary keys that autoincrement -- a pretty common database model -- postgresql doesn't understand that. Instead, it has sequence objects. You need to define a sequence, tie it to a table, and tell postgresql that when it adds a new object to the table, the default value of id is the maximum number in the corresponding sequence. Here's how to do that for the table named "user":

CREATE SEQUENCE user_id_seq OWNED by "user".id;
ALTER TABLE "user" ALTER COLUMN id SET default nextval('user_id_seq');
SELECT setval(pg_get_serial_sequence('user', 'id'), coalesce(max(id)+1,1), false) FROM "user";

Note the quotes around "user" because otherwise user is a postgresql reserved word. Repeat these three lines for every table in your database, except that you don't need the quotes around any table name except user.

Incidentally, I've been told that using autoincrement/sequence primary keys isn't best practice, because it can be a bottleneck if lots of different objects are being created at once. I used it because all the models I was following when I started with flask worked that way, but eventually I plan try to switch to using some other unique primary key.

Update: Turns out there was another problem with the sequences, and it was pretty annoying. I ended up with a bunch of indices with names like "idx_15517_ix_user_email" when they should have been "ix_user_email". The database superficially worked fine, but it havoc ensues if you ever need to do a flask/sqlalchemy/alembic migration, since sqlalchemy doesn't know anything about those indices with the funny numeric names. It's apparently possible to rename indices in postgresql, but it's a tricky operation that has to be done by hand for each index.

Now the database should be ready to test.

Test

Your flask app probably has something like this in config.py:

    SQLALCHEMY_DATABASE_URI = os.environ.get('DATABASE_URL') or \
        'sqlite:///' + os.path.join(basedir, 'dbname.db')

If so, you can export DATABSE_URL=postgresql:///dbname and then test it as you usually would. If you normally test on a local machine and not on the server, remember you can tell flask's test server to accept connections from remote machines with flask run --host=0.0.0.0

Database Backups

You're backing up your database, right? That's easier in sqlite where you can just copy the db file.

From the command line, you can back up a postgresql database with: pg_dump dbname > dbname-backup.pg You can do that from Python in a subprocess:

    with open(backup_file, 'w') as fp:
        subprocess.call(["pg_dump", dbname], stdout=fp)

Verify You're Using The New Database

I had some problems with that DATABASE_URL setting; I'd never used it so I didn't realize that it wasn't in the right place and didn't actually work. So I ran through my migration steps, changed DATABASE_URL, thought I was done, and realized later that the app was still running off sqlite3.

It's better to know for sure what your app is running. For instance, you can add a route to routes.py that prints details like that.

You can print app.config["SQLALCHEMY_DATABASE_URI"]. That's enough in theory, but I wanted to know for sure. Turns out str(db.session.get_bind()) will print the connection the flask app's database is actually using. So I added a route that prints both, plus some other information about the running app.

Whew! I was a bit surprised that migrating was as tricky as it was, and that there wasn't more documentation for it. Happy migrations, everyone.

Tags: , , , , ,
[ 12:34 Feb 01, 2020    More tech/web | permalink to this entry | comments ]

Tue, 14 Jan 2020

Plotting War

A recent article on Pharyngula blog, You ain’t no fortunate one, discussed US wars, specifically the qeustion: depending on when you were born, for how much of your life has the US been at war?

It was an interesting bunch of plots, constantly increasing until for people born after 2001, the percentage hit 100%.

Really? That didn't seem right. Wasn't the US in a lot of wars in the past? When I was growing up, it seemed like we were always getting into wars, poking our nose into other countries' business. Can it really be true that we're so much more warlike now than we used to be?

It made me want to see a plot of when the wars were, beyond Pharyngula's percentage-of-life pie charts. So I went looking for data.

The best source of war dates I could find was American Involvement in Wars from Colonial Times to the Present. I pasted that data into a table and reformatted it to turn it into Python data, and used matplotlib to plot it as a Gantt chart. (Script here: us-wars.py.)

[US Wars Since 1900]

Sure enough. If that Thoughtco page with the war dates is even close to accurate -- it could be biased toward listing recent conflicts, but I didn't find a more authoritative source for war dates -- the prevalence of war took a major jump in 2001. We used to have big gaps between wars, and except for Vietnam, the wars we were involved with were short, mostly less than a year each. But starting in 2001, we've been involved in a never-ending series of overlapping wars unprecedented in US history.

The Thoughtco page had wars going back to 1675, so I also made a plot showing all of them (click for the full-sized version). It's no different: short wars, not overlapping, all the way back to before the revolution. We've seen nothing in the past like the current warmongering. [US Wars Since 1675]

Depressing. Climate change isn't the only phenomenon showing a modern "hockey stick" curve, it seems.

Tags: , , ,
[ 12:25 Jan 14, 2020    More politics | permalink to this entry | comments ]

Thu, 09 Jan 2020

Updating a Persistent Window from Javascript Part 2: A Clever Hack

I wrote about various ways of managing a persistent popup window from Javascript, eventually settling on a postMessage() solution that turned out not to work in QtWebEngine. So I needed another solution.

Data URI

First I tried using a data: URI. In that scheme, you encode a page's full content into the URL. For instance: try this in your browser: data:text/html,Hello%2C%20World!

So for a longer page, you can do something like:

    var htmlhead = '<html>\n'
        + '<head>\n'
        + '<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">\n'
        + '<link rel="stylesheet" type="text/css" href="stylesheet.css">\n'
        + '</head>\n'
        + '\n'
        + '<body>\n'
        + '<div id="mydiv">\n';
     var htmltail = '</div>\n'
        + '</body>\n'
        + '</html>\n';

    var encodedDataURI = encodeURI(htmlhead + noteText + htmltail);

    var notewin = window.open('data:text/html,' + encodedDataURI, "notewindow",
                              "width=800,height=500");

Nice and easy -- and it even works from file: URIs!

Well, sort of works. It turns out it has a problem related to the same-origin problems I saw with postMessage. A data: URI is always opened with an origin of about:blank; and two about:blank origin pages can't talk to each other.

But I don't need them to talk to each other if I'm not using postMessage, do I? Yes, I do. The problem is that stylesheet I included in htmlhead above:

<link rel="stylesheet" type="text/css" href="stylesheet.css">\n'
All browsers I tested refuse to open the stylesheet in the about:blank popup. This seems strange: don't people use stylesheets from other domains fairly often? Maybe it's a behavior special to null (about:blank) origin pages. But in any case, I couldn't find a way to get my data: URI popup to load a stylesheet. So unless I hard-code all the styles I want for the notes page into the Javascript that opens the popup window (and I'd really rather not do that), I can't use data: as a solution.

Clever hack: Use the Same Page, Loaded in a Different Way

That's when I finally came across Remy Sharp's page, Creating popups without HTML files. Remy first explores the data: URI solution, and rejects it because of the cross-origin problem, just as I did. But then he comes up with a clever hack. It's ugly, as he acknowledges ... but it works.

The trick is to create the popup with the URL of the parent page that created it, but with a named anchor appended: parentPage.html#popup. Then, in the Javascript, check whether #popup is in the URL. If not, we're in the parent page and still need to call window.open to create the popup. If it is there, then the JS code is being executed in the popup. In that case, rewrite the page as needed. In my case, since I want the popup to show only whatever is in the div named #notes, and the slide content is all inside a div called #page, I can do this:

function updateNoteWindow() {
    if (window.location.hash.indexOf('#notes') === -1) {
        window.open(window.location + '#notes', 'noteWin',
                    'width=300,height=300');
        return;
    }

    // If here, it's the popup notes window.
    // Remove the #page div
    var pageDiv = document.getElementById("page");
    pageDiv.remove();

    // and rename the #notes div so it will be displayed in a different place
    var notesDiv = document.getElementById("notes");
    notesDiv.id = "fullnotes";
}

It works great, even in file: URIs, and even in QtWebEngine. That's the solution I ended up using.

Tags: , ,
[ 19:44 Jan 09, 2020    More tech/web | permalink to this entry | comments ]

Sun, 05 Jan 2020

Updating a Persistent Window from Javascript Part 1: postMessage

I'm trying to update my htmlpreso HTML presentation slide system to allow for a separate notes window.

Up to now, I've just used display mirroring. I connect to the projector at 1024x768, and whatever is on the first (topmost/leftmost) 1024x768 pixels of my laptop screen shows on the projector. Since my laptop screen is wider than 1024 pixels, I can put notes to myself to the right of the slide, and I'll see them but the audience won't.

That works fine, but I'd like to be able to make the screens completely separate, so I can fiddle around with other things while still displaying a slide on the projector. But since my slides are in HTML, and I still want my presenter notes, that requires putting the notes in a separate window, instead of just to the right of each slide.

The notes for each slide are in a <div id="notes"> on each page. So all I have to do is pop up another browser window and mirror whatever is in that div to the new window, right? Sure ... except this is JavaScript, so nothing is simple. Every little thing is going to be multiple days of hair-tearing frustration, and this was no exception.

I should warn you up front that I eventually found a much simpler way of doing this. I'm documenting this method anyway because it seems useful to be able to communicate between two windows, but if you just want a simple solution for the "pop up notes in another window" problem, stay tuned for Part 2.

Step 0: Give Up On file:

Normally I use file: URLs for presentations. There's no need to run a web server, and in fact, on my lightweight netbook I usually don't start apache2 by default, only if I'm actually working on web development.

But most of the methods of communicating between windows don't work in file URLs, because of the "same-origin policy". That policy is a good security measure: it ensures that a page from innocent-url.com can't start popping up windows with content from evilp0wnU.com without you knowing about it. I'm good with that. The problem is that file: URLs have location.origin of null, and every null-origin window is considered to be a different origin -- even if they're both from the same directory. That makes no sense to me, but there seems to be no way around it. So if I want notes in a separate window, I have to run a web server and use http://localhost.

Step 1: A Separate Window

The first step is to pop up the separate notes window, or get a handle to it if it's already up.

JavaScript offers window.open(), but there's a trick: if you just call notewin = window.open("notewin.html", "notewindow") you'll actually get a new tab, not a new window. If you actually want a window, the secret code for that is to give it a size:

  notewin = window.open("notewin.html", "notewindow",
                        "width=800,height=500");

There's apparently no way to just get a handle to an existing window. The only way is to call window.open(), pop up a new window if it wasn't there before, or reloads it if it's already there.

I saw some articles implying that passing an empty string "" as the first argument would return a handle to an existing window without changing it, but it's not true: in Firefox and Chromium, at least, that makes the existing window load about:blank instead of whatever page it already has. So just give it the same page every time.

Step 2: Figure Out When the Window Has Loaded

There are several ways to change the content in the popup window from the parent, but they all have one problem: if you update the content right away after calling window.open, whatever content you put there will be overwritten immediately when the popup reloads its notewin.html page (or even about:blank). So you need to wait until the popup is finished loading.

That sounds suspiciously easy. Assuming you have a function called updateNoteWinContent(), just do this:

// XXX This Doesn't work:
notewin.addEventListener('load', updateNoteWinContent, false);

Except it turns out the "load" event listener isn't called on reloads, at least not in popups. So this will work the first time, when the note window first pops up, but never after that.

I tried other listeners, like "DOMContentLoaded" and "readystatechange", but none of them are called on reload. Why not? Who knows? It's possible this is because the listener gets set too early, and then is wiped out when the page reloads, but that's just idle speculation.

For a while, I thought I was going to have to resort to an ugly hack: sleep for several seconds in the parent window to give the popup time to load: await new Promise(r => setTimeout(r, 3000)); (requires declaring the calling function as async). This works, but ... ick. Fortunately, there's a better way.

Step 2.5: Simulate onLoad with postMessage

What finally worked was a tricky way to use postMessage() in reverse. I'd already experimented with using postMessage() from the parent window to the popup, but it didn't work because the popup was still loading and wasn't ready for the content.

What works is to go the other way. In the code loaded by the popup (notewin.html in this example), put some code at the end of the page that calls

window.opener.postMessage("Loaded");

Then in the parent, handle that message, and don't try to update the popup's content until you've gotten the message:

function receiveMessageFromPopup(event) {
    console.log("Parent received a message from the notewin:", event.data);
    // Optionally, check whether event.data == "Loaded"
    // if you want to support more than one possible message.

    // Update the "notes" div in the popup notewin:
    var noteDiv = notewin.document.getElementById("notes");
    noteDiv.innerHTML = "

Here is some content.

"; } window.addEventListener("message", receiveMessageFromPopup, false);

Here's a complete working test: Test of Persistent Popup Window.

In the end, though, this didn't solve my presentation problem. I got it all debugged and working, only to discover that postMessage doesn't work in QtWebEngine, so I couldn't use it in my slide presentation app. Fortunately, I found a couple of other ways: stay tuned for Part 2.

(Update: Part 2: A Clever Hack.)

Debugging Multiple Windows: Separate Consoles

A note on debugging: One thing that slowed me down was that JS I put in the popup didn't seem to be running: I never saw its console.log() messages. It took me a while to realize that each window has its own web console, both in Firefox and Chromium. So you have to wait until the popup has opened before you can see any debugging messages for it. Even then, the popup window doesn't have a menu, and its context menu doesn't offer a console window option. But it does offer Inspect element, which brings up a Developer Tools window where you can click on the Console tab to see errors and debugging messages.

Tags: , ,
[ 20:29 Jan 05, 2020    More tech/web | permalink to this entry | comments ]

Thu, 02 Jan 2020

Horseshoe Crabs in New Mexico

[Horseshoe crab tracks in snow] On a recent hike to Escobas Mesa, I happened upon these tracks.

"Look! Trilobite tracks!" I exclaimed. But upon examining them more closely, I saw I was wrong. They look a little like trilobites, but they're clearly the tracks of a horseshoe crab.

Either way, quite a rare find in the snowy mountains of New Mexico.

Tags: ,
[ 19:23 Jan 02, 2020    More nature | permalink to this entry | comments ]