A Tricky Website Bug After Upgrading (Shallow Thoughts)

A Tricky Website Bug After Upgrading

Someone contacted me because my Galilean Moons of Jupiter page stopped working.

We've been upgrading the web server to the latest Debian, Bookworm (we were actually two revs back, on Buster, rather than on Bullseye, due to excessive laziness) and there have been several glitches that I had to fix, particularly with the apache2 configuration. But Galilean? That's just a bunch of JavaScript, no server-side involvement like Flask or PHP or CGI. If apache can serve HTML pages (and PHP and Flask), then it should certainly be able to serve HTML pages with JavaScript.

(TLDR: skip to Aha, below, for the fix.)

Of course there weren't any errors in the Apache error log that might have given me a hint what was wrong.

At first I thought it was a symbolic link issue: some of the JS is accessed through symlinks. I spent a while googling various combinations of keywords, and all I could find was people saying FollowSymLinks needed to be enabled, which it already was. I carefully checked all the directory permissions, though if those were a problem it should have shown up long ago on the old apache2 version.

I tried some experiments: when I put a foo.txt in a particular place on the web server, symlinking to /tmp/foo.txt, it failed; but if I made it a symlink to ~/foo.txt, that worked. Hmm, that's inconclusive; symlinks work, but not to /tmp. I'm still not clear why that didn't work, but it looked like symlinks weren't the problem.

The Firefox console had the errors the Apache logs didn't. It said the problem was with loading one specific file: /javascript/datetimepicker/astrotimewidget.js. Sure enough, I couldn't access that file — or anything above it, including /javascript. Instead Firefox gave me a page saying just "denied", again with nothing helpful in the Apache log.

More experimenting. I renamed /javascript to /jjj: now I could access /jjj, /jjj/datetimepicker/astrotimewidget.js and everything in between. It was something special about the name /javascript (which had always worked fine before).

Okay, let's take another look at the apache2 configuration and see if it says anything special about /javascript.

# grep -r javascript /etc/apache2
./conf-available/javascript-common.conf:Alias /javascript /usr/share/javascript/
./conf-available/javascript-common.conf:<Directory "/usr/share/javascript/">
    (plus a few lines from mods-available that didn't look related)

# ls /usr/share/javascript
jquery/  sphinxdoc/  underscore/

Aha

Aha! That made everything clear.

So here's what happened: in Debian Bookworm (or in Bullseye?), they decided to make certain JS libraries, like jquery, Debian packages. They install those libraries in /usr/share/javascript. So far, it sounds like a great idea: having those libraries as Debian packages means they'll automatically get security updates, which I very much want on a web server.

Then, in the Apache2 configuration, they make an alias to make /usr/share/javascript available as /javascript on the website. That's ... not so great. Making the assumption that no one already has their own /javascript directory, containing their own JS libraries or ones they've downloaded, seems ... let's say, flawed.

And then they activate the config file that makes this happen, javascript-common.conf, by default, linking it into /etc/apache2/conf-enabled. So anyone who had a /javascript directory before will find it no longer works after the upgrade.

Once I figured this out, I removed javascript-common.conf from conf-enabled, restarted Apache and everything was fine. I may well use Debian's version of jquery at some point — but I'll do it in a way that doesn't invalidate all the JavaScript I already have.

Incidentally, there may have been a warning about this in the hundreds of pages that scroll past during an upgrade. But I see nothing about apache2 in the Bullseye or Bookworm Release Notes. And since I couldn't find anything in a web search, I decided to write it up in the hope that it would help someone else.

Also, don't take any of this as a "Debian sucks" post. Let me clarify that the upgrades, first to Bullseye and then to Bookworm, were remarkably smooth and uneventful; the Debian developers do an amazing job making upgrades work well despite the thousands of differences people's systems might exhibit, and my disagreeing with one configuration decision doesn't change that.