Shallow Thoughts : : Dec

A motorcycle bike rack

My local mechanic (Ministry of Transport Foreign Auto Repair and Art Gallery in San Jose) often has a motorcycle parked out front. But I'd never noticed the attachment points behind the seat. Until one day, when they were being used.

A bike rack! On a motorcycle! How cool is that?

Customizing syntax highlighting colors in Emacs

Emacs has wonderful syntax highlighting. Words will be displayed in different colors depending on their syntax and the mode of the current file -- for instance, in C code, keywords of the language are highlighted in one color, comments in another, strings in a third.

The problem comes when the colors aren't right. Like that awful gold color that the flyspell spell checker uses for some words. Against a light background it makes the words almost impossible to read. I've struggled for years trying to set up custom color schemes to get around that problem, but I finally learned a simpler way to handle it when you see something in a color you want to change.

The trick is to find out what face you need to change. "Face" to emacs means more than a font face like Sans or Lucida; it means a collection of information about how characters are displayed, including font face, weight, slant, color and other attributes.

When you see something displayed in a color you don't want, place the cursor somewhere in the word. type C-u C-x = will get you the face used, along with all sorts of information about it and a handy Customize what to show link. Or you can go straight to the Customize screen with M-x customize-face -- hit return to customize the face at point (the cursor location).

In the customize-face screen, there's no GUI to choose colors, but you can edit color names. Emacs lists "red" as the foreground color; if you change it to "blue" you'll see a preview of how it will look. Color names come from /etc/X11/rgb.txt, and there are various programs like xcolorsel that will show them -- or better yet, see Wikipedia's X11 color names chart.

Once you've chosen a color, the Save for future sessions button will add a section to your .emacs file with the appropriate elisp code. Of course, you can move this code elsewhere as well. I have a somewhat complex .emacs setup, so I've moved the code into another file. Strangely, I found that my .Xdefaults background color setting no longer worked once I started using custom-set-faces, so I added a line for that as well.

(set-background-color "grey90")
(custom-set-faces
 '(flyspell-duplicate ((((class color)) (:foreground "red" :underline t :weight bold))))
 '(font-lock-comment-face ((((class color) (min-colors 88) (background light)) (:foreground "blue"))))
 )

Bank Website Security

Conversation today with a bank person over the phone:

Me: Can I get you to start sending me statements in the mail again?

Bank rep: We've gone all online now! It's so easy and convenient!

Me: I prefer to limit how much banking I do online, for security reasons.

Bank rep: Oh, but we have two factor security! It's secure! You can change your account name so it doesn't have to be your social security number -- AND you can set a security question so only you can reset your password!

Me: Right.

(The conversation progresses. She promises to send me a statement, but meanwhile it develops that there are some questions I need answered that can't be done easily over mail and require an online account. We proceed to set that up ...

Bank rep: ... and now you're at the password screen, right?

Me (reviewing the list of security questions): Um, you know that every one of your security questions is something that anyone could look up, right? Last 4 digits of driver's license? Last 4 digits of phone number? Last 4 digits of credit card?

Bank rep (astonished): What? Aren't there any that couldn't be looked up?

Me (scanning through list again): Well, the one on "last 4 digits of your best friend's phone number" at least requires guessing who your best friend is before they look up the number.

Seriously, every single one of their security questions was "last 4 digits of" something that's either a matter of public record, or something that's probably trivially available for $5 on shady websites.

Of course, you're thinking, you don't have to use the real 4-digit numbers for any of these. No, of course you don't! You can make up a number and use it as the answer for any of these.

In which case a better, more honest, security question would be: "Please enter a 4-digit PIN."

Miss American Stuck-in-Green-Cross

This is one of the creepiest statues I've seen in a park. A bronze lady has her feet embedded in a green cross, with cut tree stumps below her.

On the pedestal below her, it says:

HELP SAVE OUR TREES

THE FOREST IS THE MOTHER OF THE RIVERS

A small plaque below that says:

DEDICATED TO
THE AMERICAN GREEN CROSS
BY
GLENDALE CHAPTER N^o 1
MCMXXVIII

On the wide of the pedestal, it says:

CONSERVE THE FORESTS
PREVENT EROSION —
RENEW SOIL FERTILITY
PERPETUATE THE LUMBER SUPPLY

The title of the work, as given on an even smaller plaque on the gruond in front of the statue, is "Miss American Green Cross". Apparently it was created in 1928 by sculptor Frederick Willard Proctor, for an environmental group (although I don't usually think of "the lumber supply" being a prime concern of environmental groups).

The statue was first erected at Glendale High School in 1928. But she suffered some damage and abuse over the next few years, including being hit by a car. And then at some point in the early 1930s she disappeared. No one knew what had happened to her.

She wasn't officially rediscovered until 1954, when some hikers reported seeing it near the old Brand family cemetery, now part of Brand Park. She stood there for another three and a half decades, where she continued to be vandalized, acquiring scratches as well as grafiti, and eventually losing both arms.

Eventually, in 1990, after some debate over materials and methods, the city of Glendale restored the statue and moved down the trail to itsmis current location near Brand Library at the foot of the Brand Park hiking trails.

I've chuckled at this statue for years, whenever I visit Glendale and hike Brand Park. I still find her trapped legs, crucifixion motif, and pile of razed stumps creepy. But I must say that her history is a lot more interesting than I had imagined.

Decoding RFC 2047 email headers (like spam Subjects in other charsets)

Having not had much luck with spam filtering solutions like SpamAssassin, I'm forever having to add new spam filters by hand. For instance, after about the sixth time I get "President Waives Refi Requirement" or "Melt your fat! MUST WATCH this video now!" within a couple of hours, I'm pretty tired of it and don't want to see any more of them.

With mail filtering programs like procmail or maildrop, it's easy enough to match a pattern like "Subject:.*Refi Requirement" or "Subject:.*Melt your fat" and filter that message to a spam folder (or /dev/null).

But increasingly, I add patterns I'm seeing in spam messages, and yet the messages with those patterns keep coming in. Why? Because the spammers are using RFC 2047 to encode the subject into some other character set.

Here's how it works. A spammer sends a subject line that looks something like this:

Subject: =?utf-8?B?U3RvcCBPdmVycGF5aW5nIGZvciBQcmludGVyIEluaw==?=

Mail programs are smart enough to decode this into:

Subject: Stop Overpaying for Printer Ink

but spam filtering programs often aren't, so your "printer ink" filter won't catch it. And if you look through your spam folder with tools like grep to see why it didn't get caught, or to find particularly spammy subjects that might call for a filter (grep Subject spamfolder | sort is pretty handy), these encoded subjects will be incognito.

I briefly tried setting up a filter that spam-filed anything with =? in the Subject line. But that's way too broad a brush -- not all people there are legitimate reasons for using other charsets even in English language email. It's relatively rare, but it happens. And some bots, notably the Adafruit forum notification bot and the bot that sends out announcements from my alma mater, unaccountably encode the charset even when they're sending mail entirely in US ASCII.

So what's really needed is not to filter out all messages that specify a charset, but to decode the Subject so the spam filter can see it and filter it accordingly.

How? I couldn't find any ready-made tool available for Linux that could decode RFC 2047 headers; but the Python email package makes decoding a one-line task. In the Python interpreter:

$ python
Python 2.7.3 (default, Aug  1 2012, 05:16:07) 
Type "help", "copyright", "credits" or "license" for more information.
>>> import email
>>> email.Header.decode_header("Subject: =?utf-8?B?U3RvcCBPdmVycGF5aW5nIGZvciBQcmludGVyIEluaw==?=")
[('Subject:', None), ('Stop Overpaying for Printer Ink', 'utf-8')]
>>>

So it's easy to write a script that can pull headers out of email messages (files) and decode them. Just look for the line starting with the header you want to match -- e.g. "Subject:" -- and pass that line to email.Header.decode_header().

Only one snag. If the subject is longer than about 20 characters, spammers will often opt to split it up into multiple groups, sometimes even in different character sets. So for example, you might see something like this, spread over multiple lines:

Subject: =?windows-1252?Q?Earn_your_degree_=97_on_your_time?=
        =?windows-1252?Q?_and_terms?=

The script has to handle that too. If it's reading a header, it has to check the next line, and if that line begins with whitespace, treat it as more of the header.

The resulting script, decodemail.py (on github), seems pretty handy and should be able to be plugged in to a mail filtering program.

A Brief Tutorial on rsyslog.conf

Every now and then. I find myself puzzled by which Linux system messages are going to which files in /var/log. I vaguely knew it was all configurable via the syslog service, specifically the file /etc/rsyslog.conf (on some systems it's still/etc/syslog.conf) and the directory /etc/rsyslog.d.

But every time I started wading into that 822-line rsyslog.conf(5) man page I end up deciding "perhaps another time".

But when you have a problem at work and messages are being logged to the wrong place and filling up the disk, "another time" arrives.

If you're like most people, you don't need to know esoterica like how to use a plug-in to log in a special custom format to a named pipe; you just want to know how to change the file so you see the messages you want to see, or so you don't have the same messages being logged in three different places. The man page isn't good about separating the practical information everyone needs from the esoterica.

Strangely, there doesn't seem to be much in the way of simple rsyslog web tutorials, either. So now, I present to you:

rsyslog.conf(5): the "Good Parts" Version

First, remember that you don't need to create an /etc/rsyslog.conf from scratch. You just need to be able to read the existing one and modify it a little. So start with the file you already have.

MODULES section

rsyslog has tons of modules available. They're listed in the man page with no clue offered as to what they all do. Just leave that section alone and don't worry about it.

GLOBAL DIRECTIVES

You can probably leave that section alone too.

But if you need to configure something globally -- for instance, the user who will own the files in /var/log -- the rsyslog.conf man page actually isn't too bad in describing the various options.

rsyslog.d

On some systems, notably Ubuntu, most of the configuration happens in smaller files in rsyslog.d rather than in the main rsyslog.conf. What follows applies to those files as well as the main one.

Rules section

The rest of the file(s) comprise rules for what gets logged where.

Each rule includes a selector (what gets logged) and an action (where it will get logged). Each selector includes a facility (what type of message we're talking about) and a priority (how important it is). But a selector can have several facilities/priorities, which is what makes the file look so complicated.

Enough theory. Let's look at some practical examples. These examples are taken from a plug computer running Debian.

auth,authpriv.*                 /var/log/auth.log

Messages of type auth or authpriv (the facility), with any priority, get logged to the file /var/log/auth.log.

*.*;auth,authpriv.none          -/var/log/syslog

Any message type, with any priority, gets logged to /var/log/syslog; except for auth and authpriv messages (which is good since they're already being logged to auth.log). The special priority none prevents those messages from being logged even though they would have been included in the *.*.

What's that dash in front of the filename? It's not documented in the man page, but it turns out to mean "Don't sync after every write to the file". Except that rsyslogd won't sync anyway, unless you add a special directive in the Global Directives section. So for most people, a dash makes no difference one way or the other -- it will be ignored.

So why is it there in the file, especially since the man page doesn't even document it? I have no idea; probably no one from the various distros has audited these files for years.

daemon.*                        -/var/log/daemon.log

There are quite a few facilities n addition to auth and daemon. Here's the list: auth, authpriv, cron, daemon, kern, lpr, mail, news, syslog, user, uucp and local0 through local7. There are also a couple of deprecated ones: security (considered to be the same as auth) and mark (for internal use only).

Selector priorities

Priorities can be: debug, info, notice, warning, err, crit, alert, emerg; plus the deprecated warn, error and panic (treated as warning, err and emerg).

*.=debug;\
        auth,authpriv.none;\
        news.none;mail.none     -/var/log/debug

Normally, specifying a priority like debug means to log anything of that priority or higher. So specifying *.debug would log everything. Adding an equals sign, =debug, means log only debug messages but nothing higher. This rule also excludes auth, authpriv, news and mail messages even if they're debug.

*.=info;*.=notice;*.=warn;\
        auth,authpriv.none;\
        cron,daemon.none;\
        mail,news.none          -/var/log/messages

Yow! The familiar /var/log/messages sure has a complicated rule. It gets anything of priority info, notice, or warn, unless they're facility aurh, authpriv, cron, daemon, mail or news.

Note that this overlaps with some of the other rules. So you'll see a lot of the same messages showing up in messages and syslog (which you'll recall got *.*). That was what got me started down this road: all that duplicated logging on our space-limited plug computers.

daemon.*;mail.*;\
        news.err;\
        *.=debug;*.=info;\
        *.=notice;*.=warn       |/dev/xconsole

Notice that until now, the rules have logged only to filenames. Remember all the rest of that complicated man page, explaining all the other actions besides filenames? Here's the only one that typically comes up: there's a device called /dev/xconsole where you can direct errors, and then you can run the xconsole program (you might have to specify the file, xconsole -file /dev/xconsole) to see the output.

Applying this to specific messages

How do you figure out which messages have what facilities and priorities?

That's the tricky part. Mostly, you can't. You can guess, or look at the source code, or just change rules in rsyslog.conf and see what happens.

But with a little experimentation, this guide should help you configure your own syslog configuration file and get rid of all those redundant messages filling up your disk.

Good luck!