[ 11:37 Apr 22, 2010 More writing | permalink to this entry | comments ]
Normally I'd use the Linux command
tee for that:
prog | tee prog.out saves a copy of the output to the
file prog.out as well as printing it. That worked fine until
I added something that needed to prompt the user for an answer.
That doesn't work when you're piping through tee: the output gets
buffered and doesn't show up when you need it to, even if you try
to flush() it explicitly.
I investigated shell-based solutions: the output I need is on
sterr, while Python's raw_input() user prompt uses stdout, so
if I could get the shell to send stderr through tee without stdout,
that would have worked. My preferred shell, tcsh, can't do this at all,
but bash supposedly can. But the best examples I could find on the
web, like the arcane
prog 2>&1 >&3 3>&- | tee prog.out 3>&-
I considered using /dev/tty or opening a pty, but those calls only work on Linux and Unix and the program is otherwise cross-platform.
What I really wanted was a class that acts like a standard
but when you write to it it writes to two places: the log file and stderr.
I found an example of someone
to write a Python tee class, but it didn't work: it worked for
write() but not for print
I am greatly indebted to KirkMcDonald of #python for finding the problem.
In the Python source implementing
PyFile_WriteObject (line 2447) checks the object's type, and if it's
subclassed from the built-in
file object, it writes
directly to the object's
fd instead of calling
The solution is to use composition rather than inheritance. Don't make your
file-like class inherit from
file, but instead include a
file object inside it. Like this:
import sys class tee : def __init__(self, _fd1, _fd2) : self.fd1 = _fd1 self.fd2 = _fd2 def __del__(self) : if self.fd1 != sys.stdout and self.fd1 != sys.stderr : self.fd1.close() if self.fd2 != sys.stdout and self.fd2 != sys.stderr : self.fd2.close() def write(self, text) : self.fd1.write(text) self.fd2.write(text) def flush(self) : self.fd1.flush() self.fd2.flush() stderrsav = sys.stderr outputlog = open(logfilename, "w") sys.stderr = tee(stderrsav, outputlog)
And it works!
print >>sys.stderr, "Hello, world" now
goes to the file as well as stderr, and
works to prompt the user for input.
In general, I'm told, it's not safe to inherit from
Python's built-in objects like
file, because they tend
to make assumptions instead of making virtual calls to your
overloaded methods. What happened here will happen for other objects too.
So use composition instead when extending Python's built-in types.
On Linux, for some reason Apple keyboards' function keys don't work by default. Most of them try to run special functions instead, like volume up/down or play/pause.
But you can get normal function keys by talking to the kernel module that drives the keyboard:
echo 2 > /sys/module/hid_apple/parameters/fnmode
This will only last until shutdown, so put that line in /etc/rc.local or a similar place so it runs every time you boot.
Here's an Ubuntu help page on Apple Keyboards with more information and other tricks.
The list owner posted a message about using good passwords so your account isn't hacked since that causes problems for everyone.
Of course, that's good advice and using good passwords is always a good idea. But I though this sounded more like a Joe-job spam, in which the spammer forges the From address to look like it's coming from someone else.
Normal users encounter this in two ways:
Since this sort of attack is so common, I felt the victim didn't deserve being harangued about not having set up a good password. So I posted a short note to the list explaining about Joe-jobs. But to make the point, I forged the From address of the list owner. Indeed, it got through Yahoo and out to the list just fine:
[ ... ] the spam probably wasn't from a bad password. It was probably just a spammer forging the header to look like it's from a legitimate user. It's called a "joe-job": http://en.wikipedia.org/wiki/Joe-job
To illustrate, I've changed the From address on this message to look like it's coming from Adam. I have not hacked [listowner]'s account or guessed his password or anything else. If this works, and looks like it came from [listowner], then the spam could have been done the same way -- and there's no need to blame the owner of the account, or accuse them of having a bad password.
Why does this work? Why doesn't Yahoo just block messages from firstname.lastname@example.org if the mail doesn't come from isp.com?
They can't! Many, many people don't send mail from the domains in their email addresses. In effect, people forge their From header all the time. Here are some examples:
If mailing lists rejected posts in all these cases, people would be pretty annoyed. So they don't. But that means that now and then, some Joe-job spam gets through to mailing lists. Unfortunately.
(Update: The message that inspired this may very well have been a hacked password after all case, based on the mail headers. But I found that a lot of people didn't know about Joe-jobbing, so I thought this was worth writing up anyway.)
Where 2.0 conference (thanks, Linux Pro Magazine!) Then, on the weekend, the free WhereCamp followed it up.
I'd been to WhereCamp last year. It was wonderful, geeky, highly technical and greatly inspiring. I thought I was the only person interested in mapping, especially in Python, and after the first couple of sessions I was blown away with how little I knew and what a thriving and expert community there was. I was looking forward to the full experience this year -- I figured Where 2.0 must be similar but even better.
Actually they're completely different events. Where 2.0 was dominated by location-aware startups: people with iPhone games (Foursquare and others in a similar mold), shopping apps (find the closest pizza place to your location!) and so on. The talks were mostly 15 minutes long, so while there were lots of people there with fascinating apps or great stories to tell, there was no time to get detail on anything. I think the real point of Where 2.0 is to get a sketch of who's doing what so you can go collar them in the "hallway track" later and make business deals.
Here are some highlights from Where 2.0. I'll write up WhereCamp separately.
The Ignite session Tuesday night was great fun, as Ignite sessions almost always are.
The Ignite session was broken in the middle by a half-hour interlude where a bunch of startups gave one-minute presentations on their products, then the audience voted on the best, then an award was given which had already been decided and had nothing to do with the audience vote (we didn't even get to find out which company the audience chose). Big yawner: one minute isn't long enough for anyone to show off a product meaningfully, and I wasn't the only one there who brought reading material to keep them occupied until the second round of Ignite talks started up again.
Best Ignite talks (Ignite Where 2.0 videos here):
Patrick Meier gave a longer version of his Ignite talk on Mobilizing Ushahidi-Haiti, full of interesting stories of how OpenStreetMap and other technologies like Twitter came together to help in the Haiti rescue effort.
Clouds, Crowds, and Shrouds: How One Government Agency Seeks to Change the Way It Spatially Enables Its Information, by Terrance Busch of the US Defense Intelligence Agency, was an interesting look into the challenges of setting up a serious mapping effort, then integrating later with commercial and crowdsourced efforts.
In Complexities in Bringing Home Environmental Awareness, Kim Balassiano of the US EPA showed the EPA MyEnvironment page, where you can find information about local environmental issues like toxic waste cleanups. They want users to enter good news too, like composting workshops or community gardens, but so far the data on the map is mostly bad. Still a useful site.
There were a couple of interesting keynotes on Thursday morning, but work kept me at home. I thought I could catch them on the live video stream, but unfortunately the stream that had worked fine on Wednesday wasn't working on Thursday, so I missed the Mark L. DeMulder's talk on the USGS's National Map efforts. Fortunately, they were at WhereCamp where they gave much more detail. Likewise, I missed the big ESRI announcement that everyone was talking about all afternoon -- they released some web thing, but as far as I can tell they're still totally Windows-centric and thus irrelevant to a Linux and open source user. But I want to go back and view the video anyway.
There was another talk on Thursday which I won't name, but it had a few lessons for speakers:
Base Map 2.0 was a panel-slash-debate between Steve Coast (OpenStreetMap), Timothy Trainor (U.S. Census Bureau), Peter ter Haar (Ordnance Survey), Di-Ann Eisnor (Platial), and moderated by Ian White of Urban Mapping. It was fabulous. I've never seen such a lively panel: White kept things moving, told jokes, asked provocative and sometimes inflammatory questions and was by far the best panel moderator I've seen. The panelists kept up with him and gave cogent, interesting and illuminating answers. Two big issues were the just-announced release of Ordnance Survey data, and licensing issues causing mismatches between OSM, OS and Census datasets.
Community-based Grassroots Mapping with Balloons and Kites in Lima, Peru by Jeffrey Warren was another fabulous talk. He builds balloons out of garbage bags, soda bottles and a digital camera, goes to poor communities in places like Lima and teaches the community (including the kids) how to map their own communities. This is more than an academic exercise for them, since maps can help them prove title to their land. Check it out at GrassrootsMapping.org and build your own aerial mapping balloon! (He was at WhereCamp, too, where we got to see the equipment up close.)
Visualizing Spatio-temporal War Casualty Data in Google Earth by Sean Askay of Google was just as good. He's built a KML file called Map the Fallen showing US and allied casualties from Iraq: the soldiers' hometowns, place of death, age, gender, and lots of other details about them with links to tribute pages, plus temporal information showing how casualties changed as the war progressed. It's an amazing piece of work, and sobering ... and I was most annoyed to find out that it needs a version of Google Earth that doesn't run on Linux, so I can't run it for myself. Boo!
Overall, a very fun conference, though it left me hungry for detail. Happily, after a day off there was WhereCamp to fill that void.
error handler for bash that made the shell a lot friendlier for newbies (or anyone else, really).
Linux Planet gave me the chance to write it up in more detail, explaining a bit more about how it works: Making Bash Error Messages Friendlier.
Trying to run debootstrap to install the new Ubuntu beta, I kept hitting a snag right at the beginning: debootstrap kept saying the filesystem was mounted with the wrong permissions, and it couldn't create an executable file or a device.
I have lines for each of my spare filesystems in /etc/fstab, like this:
/dev/sda4 /lucid ext3 relatime,user,noauto 0 0That way, if I'm booted into one OS but I want to check a file from another, I can mount it without needing sudo, just by typing
Not being able to create executable files means it's mounted with
noexec flag. I checked another machine and saw
that it was using lines like
/dev/sda4 /lucid ext3 exec,user,noauto 0 0
I added the "exec," to fstab, unmounted and remounted ... and it was
still mounted with
Turns out on some Linux versions, making a filesystem
mountable turns off
exec even if you've specified it
explicitly. You have to add the
exec after the
But that still didn't make debootstrap happy, because it couldn't
create a device. That's a separate fstab option,
So here's the fstab entry that finally worked:
/dev/sda4 /lucid ext3 relatime,user,exec,dev,noauto 0 0