Shallow Thoughts : : Apr

Article: /etc/fstab basics

On Linux Planet, an article about the /etc/fstab file and how to customize it: Understanding fstab.

Tags: writing, linux
[ 11:37 Apr 22, 2010    More writing | permalink to this entry | ]

Tee in Python

I needed a way to send the output of a Python program to two places simultaneously: print it on-screen, and save it to a file.

Normally I'd use the Linux command tee for that: prog | tee prog.out saves a copy of the output to the file prog.out as well as printing it. That worked fine until I added something that needed to prompt the user for an answer. That doesn't work when you're piping through tee: the output gets buffered and doesn't show up when you need it to, even if you try to flush() it explicitly.

I investigated shell-based solutions: the output I need is on sterr, while Python's raw_input() user prompt uses stdout, so if I could get the shell to send stderr through tee without stdout, that would have worked. My preferred shell, tcsh, can't do this at all, but bash supposedly can. But the best examples I could find on the web, like the arcane prog 2>&1 >&3 3>&- | tee prog.out 3>&- didn't work.

I considered using /dev/tty or opening a pty, but those calls only work on Linux and Unix and the program is otherwise cross-platform.

What I really wanted was a class that acts like a standard Python file object, but when you write to it it writes to two places: the log file and stderr.

I found an example of someone trying to write a Python tee class, but it didn't work: it worked for write() but not for print >>

I am greatly indebted to KirkMcDonald of #python for finding the problem. In the Python source implementing >>, PyFile_WriteObject (line 2447) checks the object's type, and if it's subclassed from the built-in file object, it writes directly to the object's fd instead of calling write().

The solution is to use composition rather than inheritance. Don't make your file-like class inherit from file, but instead include a file object inside it. Like this:

import sys

class tee :
    def __init__(self, _fd1, _fd2) :
        self.fd1 = _fd1
        self.fd2 = _fd2

    def __del__(self) :
        if self.fd1 != sys.stdout and self.fd1 != sys.stderr :
            self.fd1.close()
        if self.fd2 != sys.stdout and self.fd2 != sys.stderr :
            self.fd2.close()

    def write(self, text) :
        self.fd1.write(text)
        self.fd2.write(text)

    def flush(self) :
        self.fd1.flush()
        self.fd2.flush()

stderrsav = sys.stderr
outputlog = open(logfilename, "w")
sys.stderr = tee(stderrsav, outputlog)

And it works! print >>sys.stderr, "Hello, world" now goes to the file as well as stderr, and raw_input still works to prompt the user for input.

In general, I'm told, it's not safe to inherit from Python's built-in objects like file, because they tend to make assumptions instead of making virtual calls to your overloaded methods. What happened here will happen for other objects too. So use composition instead when extending Python's built-in types.

Tags: programming, python
[ 09:48 Apr 16, 2010    More programming | permalink to this entry | ]

Function keys on an Apple keyboard, on Linux

Quick tip from Dave, passed along to someone else trying to use an Apple keyboard on Linux:

On Linux, for some reason Apple keyboards' function keys don't work by default. Most of them try to run special functions instead, like volume up/down or play/pause.

But you can get normal function keys by talking to the kernel module that drives the keyboard:

echo 2 >  /sys/module/hid_apple/parameters/fnmode

This will only last until shutdown, so put that line in /etc/rc.local or a similar place so it runs every time you boot.

Here's an Ubuntu help page on Apple Keyboards with more information and other tricks.

Tags: linux, apple, kernel, keyboard
[ 16:03 Apr 15, 2010    More linux/kernel | permalink to this entry | ]

"Joe-job" spam (forged From addresses)

I'm in a Yahoo group where a spammer just posted a message that looked like it was coming from someone in the group, so Yahoo allowed it.

The list owner posted a message about using good passwords so your account isn't hacked since that causes problems for everyone.

Of course, that's good advice and using good passwords is always a good idea. But I though this sounded more like a Joe-job spam, in which the spammer forges the From address to look like it's coming from someone else.

Normal users encounter this in two ways:

1. You start getting tons of bounce messages that look as though you sent spam to hundreds of people and they're refusing it.
2. You see spam that looks like it came from a friend of yours, or spam on a mailing list that looks like it came from a legitimate member of that list.

Since this sort of attack is so common, I felt the victim didn't deserve being harangued about not having set up a good password. So I posted a short note to the list explaining about Joe-jobs. But to make the point, I forged the From address of the list owner. Indeed, it got through Yahoo and out to the list just fine:

[ ... ] the spam probably wasn't from a bad password. It was probably just a spammer forging the header to look like it's from a legitimate user. It's called a "joe-job": http://en.wikipedia.org/wiki/Joe-job

To illustrate, I've changed the From address on this message to look like it's coming from Adam. I have not hacked [listowner]'s account or guessed his password or anything else. If this works, and looks like it came from [listowner], then the spam could have been done the same way -- and there's no need to blame the owner of the account, or accuse them of having a bad password.

Why does this work? Why doesn't Yahoo just block messages from user@isp.com if the mail doesn't come from isp.com?

They can't! Many, many people don't send mail from the domains in their email addresses. In effect, people forge their From header all the time. Here are some examples:

• You're using you@gmail.com, but you're using Thunderbird or Eudora or Evolution or something to read and send mail from home.
• You're on your computer at home, but you're sending work-related email from your work account.
• You're on your laptop, using Thunderbird or whatever, mailing from you@isp.com, but you're at a friend's house or a hotel or conference or somewhere.
• You're sending mail from a public terminal somewhere (eek, do people really type their mail info in to these things?)
• You're reading and sending mail from a mobile phone.
• You're sending mail from your own domain, me@mydomain.com, but you're at home or somewhere else other than wherever mydomain.com is hosted.

If mailing lists rejected posts in all these cases, people would be pretty annoyed. So they don't. But that means that now and then, some Joe-job spam gets through to mailing lists. Unfortunately.

(Update: The message that inspired this may very well have been a hacked password after all case, based on the mail headers. But I found that a lot of people didn't know about Joe-jobbing, so I thought this was worth writing up anyway.)

Tags: security, email, joe-job, spam
[ 22:28 Apr 13, 2010    More tech/email | permalink to this entry | ]

Article: Linux Boot Camp, part 2: Upstart

On Linux Planet, an article about how Upstart manages the Linux boot process, how it's used in various distros, and how to explore and control it:

The Upstart Startup Manager (Linux Boot Camp part 2)

Tags: writing, linux, boot
[ 10:49 Apr 08, 2010    More writing | permalink to this entry | ]

Where 2.0 2010

Last week I had the opportunity to go to the Where 2.0 conference (thanks, Linux Pro Magazine!) Then, on the weekend, the free WhereCamp followed it up.

I'd been to WhereCamp last year. It was wonderful, geeky, highly technical and greatly inspiring. I thought I was the only person interested in mapping, especially in Python, and after the first couple of sessions I was blown away with how little I knew and what a thriving and expert community there was. I was looking forward to the full experience this year -- I figured Where 2.0 must be similar but even better.

Actually they're completely different events. Where 2.0 was dominated by location-aware startups: people with iPhone games (Foursquare and others in a similar mold), shopping apps (find the closest pizza place to your location!) and so on. The talks were mostly 15 minutes long, so while there were lots of people there with fascinating apps or great stories to tell, there was no time to get detail on anything. I think the real point of Where 2.0 is to get a sketch of who's doing what so you can go collar them in the "hallway track" later and make business deals.

Here are some highlights from Where 2.0. I'll write up WhereCamp separately.

Ignite Where

The Ignite session Tuesday night was great fun, as Ignite sessions almost always are.

The Ignite session was broken in the middle by a half-hour interlude where a bunch of startups gave one-minute presentations on their products, then the audience voted on the best, then an award was given which had already been decided and had nothing to do with the audience vote (we didn't even get to find out which company the audience chose). Big yawner: one minute isn't long enough for anyone to show off a product meaningfully, and I wasn't the only one there who brought reading material to keep them occupied until the second round of Ignite talks started up again.

Best Ignite talks (Ignite Where 2.0 videos here):

• App Stores Suck, Jonathan Stark
• Why Your Data Sucks, Paul Ramsey
• Crowdsourcing the Impossible: Ushahidi-Haiti, Patrick Meier
• Have Chickens, Need Lasers!, Martin Isenburg -- which didn't actually involve lasers as far as I could tell, but it was certainly lively.

Wednesday talks

Patrick Meier gave a longer version of his Ignite talk on Mobilizing Ushahidi-Haiti, full of interesting stories of how OpenStreetMap and other technologies like Twitter came together to help in the Haiti rescue effort.

Clouds, Crowds, and Shrouds: How One Government Agency Seeks to Change the Way It Spatially Enables Its Information, by Terrance Busch of the US Defense Intelligence Agency, was an interesting look into the challenges of setting up a serious mapping effort, then integrating later with commercial and crowdsourced efforts.

In Complexities in Bringing Home Environmental Awareness, Kim Balassiano of the US EPA showed the EPA MyEnvironment page, where you can find information about local environmental issues like toxic waste cleanups. They want users to enter good news too, like composting workshops or community gardens, but so far the data on the map is mostly bad. Still a useful site.

Thursday talks

There were a couple of interesting keynotes on Thursday morning, but work kept me at home. I thought I could catch them on the live video stream, but unfortunately the stream that had worked fine on Wednesday wasn't working on Thursday, so I missed the Mark L. DeMulder's talk on the USGS's National Map efforts. Fortunately, they were at WhereCamp where they gave much more detail. Likewise, I missed the big ESRI announcement that everyone was talking about all afternoon -- they released some web thing, but as far as I can tell they're still totally Windows-centric and thus irrelevant to a Linux and open source user. But I want to go back and view the video anyway.

There was another talk on Thursday which I won't name, but it had a few lessons for speakers:

• Be aware of when you're speaking, so somebody doesn't have to come find you in the audience and say "Hey, you're next. Are you coming?"
• If you're not bringing your own laptop, try to get access to the presentation machine beforehand and test out your presentation.
• Especially if you're planning on showing a video that may require downloading nonstandard software.

Base Map 2.0 was a panel-slash-debate between Steve Coast (OpenStreetMap), Timothy Trainor (U.S. Census Bureau), Peter ter Haar (Ordnance Survey), Di-Ann Eisnor (Platial), and moderated by Ian White of Urban Mapping. It was fabulous. I've never seen such a lively panel: White kept things moving, told jokes, asked provocative and sometimes inflammatory questions and was by far the best panel moderator I've seen. The panelists kept up with him and gave cogent, interesting and illuminating answers. Two big issues were the just-announced release of Ordnance Survey data, and licensing issues causing mismatches between OSM, OS and Census datasets.

Community-based Grassroots Mapping with Balloons and Kites in Lima, Peru by Jeffrey Warren was another fabulous talk. He builds balloons out of garbage bags, soda bottles and a digital camera, goes to poor communities in places like Lima and teaches the community (including the kids) how to map their own communities. This is more than an academic exercise for them, since maps can help them prove title to their land. Check it out at GrassrootsMapping.org and build your own aerial mapping balloon! (He was at WhereCamp, too, where we got to see the equipment up close.)

Visualizing Spatio-temporal War Casualty Data in Google Earth by Sean Askay of Google was just as good. He's built a KML file called Map the Fallen showing US and allied casualties from Iraq: the soldiers' hometowns, place of death, age, gender, and lots of other details about them with links to tribute pages, plus temporal information showing how casualties changed as the war progressed. It's an amazing piece of work, and sobering ... and I was most annoyed to find out that it needs a version of Google Earth that doesn't run on Linux, so I can't run it for myself. Boo!

Overall, a very fun conference, though it left me hungry for detail. Happily, after a day off there was WhereCamp to fill that void.

Tags: mapping, conferences
[ 22:34 Apr 05, 2010    More conferences | permalink to this entry | ]

Article: Making Bash Error Messages Friendlier

A while back I worked on an error handler for bash that made the shell a lot friendlier for newbies (or anyone else, really).

Linux Planet gave me the chance to write it up in more detail, explaining a bit more about how it works: Making Bash Error Messages Friendlier.

Tags: writing, linux, shell
[ 17:17 Apr 02, 2010    More writing | permalink to this entry | ]

Quick tip: User-mounted filesystems with "exec"

Second time I've run into this -- time to write it down.

Trying to run debootstrap to install the new Ubuntu beta, I kept hitting a snag right at the beginning: debootstrap kept saying the filesystem was mounted with the wrong permissions, and it couldn't create an executable file or a device.

I have lines for each of my spare filesystems in /etc/fstab, like this:

/dev/sda4  /lucid  ext3  relatime,user,noauto  0   0

That way, if I'm booted into one OS but I want to check a file from another, I can mount it without needing sudo, just by typing mount /lucid.

Not being able to create executable files means it's mounted with the noexec flag. I checked another machine and saw that it was using lines like

/dev/sda4  /lucid  ext3  exec,user,noauto  0   0

I added the "exec," to fstab, unmounted and remounted ... and it was still mounted with noexec.

Turns out on some Linux versions, making a filesystem user mountable turns off exec even if you've specified it explicitly. You have to add the exec after the user.

But that still didn't make debootstrap happy, because it couldn't create a device. That's a separate fstab option, dev, and user implies nodev.

So here's the fstab entry that finally worked:

/dev/sda4  /lucid  ext3  relatime,user,exec,dev,noauto  0   0

Tags: linux, filesystem, fstab, tips
[ 23:07 Apr 01, 2010    More linux | permalink to this entry | ]