Shallow Thoughts : : Feb

langgrep: search only in scripts of a specified language

I was making a minor tweak to my garmin script that uses gpsbabel to read in tracklogs and waypoints from my GPS unit, and I needed to look up the syntax of how to do some little thing in sh script. (One of the hazards of switching languages a lot: you forget syntax details and have to look things up a lot, or at least I do.)

I have quite a collection of scripts in various languages in my ~/bin (plus, of course, all the scripts normally installed in /usr/bin on any Linux machine) so I knew I'd have lots of examples. But there are scripts of all languages sharing space in those directories; it's hard to find just sh examples. For about the two-hundredth time, I wished, "Wouldn't it be nice to have a command that can search for patterns only in files that are really sh scripts?"

And then, the inevitable followup ... "You know, that would be really easy to write."

So I did -- a little python hack called langgrep that takes a language, grep arguments and a file list, looks for a shebang line and only greps the files that have a shebang matching the specified language.

Of course, while writing langgrep I needed langgrep, to look up details of python syntax for things like string.find (I can never remember whether it's string.find(s, pat) or s.find(pat); the python libraries are usually nicely object-oriented but strings are an exception and it's the former, string.find). I experimented with various shell options -- this is Unix, so of course there are plenty of ways of doing this in the shell, without writing a script. For instance:

grep find `egrep -l '#\\!.*python' *`
grep find `file * | grep python | sed 's/:.*//'`
i in foo; file $i|grep python && grep find $i; done    # in sh/bash

These are all pretty straightforward, but when I try to make them into tcsh aliases things get a lot trickier. tcsh lets you make aliases that take arguments, so you can use !:1 to mean the first argument, !2-$ to mean all the arguments starting with the second one. That's all very well, but when you put them into a shell alias in a file like .cshrc that has to be parsed, characters like ! and $ can mean other things as well, so you have to escape them with \. So the second of those three lines above turns into something like

alias greplang "grep \!:2-$ `file * | grep \!:1 | sed 's/:.*//'`"

except that doesn't work either, so it probably needs more escaping somewhere. Anyway, I decided after a little alias hacking that figuring out the right collection of backslash escapes would probably take just as long as writing a python script to do the job, and writing the python script sounded more fun.

So here it is: my langgrep script. (Awful name, I know; better ideas welcome!) Use it like this (if python is the language you're looking for, find is the search pattern, and you want -w to find only "find" as a whole word):

langgrep python -w find ~/bin/*

Tags: programming, python, shell
[ 10:57 Feb 28, 2009    More programming | permalink to this entry | ]

Updated Network Schemes page

I've used my simple network schemes setup for many years. No worries about how distros come up with a new network configuration GUI every release; no worries about all the bloat that NetworkManager insists on before it will run; no extra daemons running all the time polling my net just in case I might want to switch networks suddenly. It's all command-line based; if I'm at home, I type

netscheme home

and my network will be configured for that setup until I tell it otherwise. If I go to a cafe with an open wi-fi link, I type netscheme wifi; I have other schemes for places I go where I need a wireless essid or WEP key. It's all very easy and fast.

Last week for SCALE I decided it was silly to have to su and create a new scheme file for conferences where all I really needed was the name of the network (the essid), so I added a quick hack to my netscheme script so that typing netscheme foo, where there's no existing scheme by that name, will switch to a scheme using foo as the essid. Worked nicely, and that inspired me to update the "documentation".

I wrote an elaborate page on my network schemes back around 2003, but of course it's all changed since then and I haven't done much toward updating the page. So I've rewritten it completely, taking out most of the old cruft that doesn't seem to apply any more. It's here: Howto Set Up Multiple Network Schemes on a Linux Laptop.

Tags: linux, laptop, networking
[ 10:51 Feb 27, 2009    More linux/laptop | permalink to this entry | ]

Linux Planet article: DotDot and Permissions

Probably the last in the commandline series (at least for a while, today's article covers the meaning of . and .. in Linux pathnames, and how to tell what permissions a file has.

Sharing Files in Linux and Understanding Pathnames

Tags: writing, shell, CLI
[ 22:26 Feb 26, 2009    More writing | permalink to this entry | ]

Back up: sorry for the downtime

File under "Is it really worth running your own server?":

About a week before we'd planned to leave for SCALE, our router/firewall started acting flaky. Every could of days it would just stop routing, for no apparent reason. A few days before a trip is no time to debug a problem like that, so we re-purposed another router we happened to have sitting around. It seemed to work fine; we tested it from both inside and outside over several days, and everything was working fine.

We drove down to LA as planned, spent a few hours hanging out and having dinner with family, then decided on a quick email check before bed. Um ... check where? There was no shallowsky.com on the net ... nor any of the other domains we host from that server. Lovely. Down less than 7 hours after we'd left, and no way of fixing it until we got home a week later.

Luckily for me, a friend was generously willing to host my mail for the week I was gone (including the associated bucketloads of spam). That didn't solve the web downtime, but at least in theory I wouldn't miss any important messages that came in. At least, I wouldn't miss mail that happened to come from servers that checked the new MX record; turns out a lot of servers don't, and just keep re-checking their cached address for days or weeks, bouncing messages accordingly. Not much I could do about that.

Anyway, now I'm back from SCALE (the conference went well) ... and it wasn't the router at all. The repurposed router is chugging along just fine; it was the DSL modem that coincidentally chose our departure day to stop talking to the net. Never underestimate the power of coincidence.

Sorry for the downtime! Maybe it really is time to move this domain to an ISP.

Tags: blogging, downtime
[ 21:03 Feb 23, 2009    More misc | permalink to this entry | ]

Navigating the Linux filesystem

My latest Linux Planet article covers how to find your way around the Linux filesystem in the command-line, for anyone who wants to graduate from file managers and start using the shell.

Navigating the Linux Filesystem (and the Digg link for those so inclined).

Tags: writing, shell, CLI
[ 18:42 Feb 12, 2009    More writing | permalink to this entry | ]

Bruny Island night life: penguins and shearwaters

After a week in Tasmania, supposedly the most wildlife-packed state in Australia, without seeing anything besides ducks (mostly mallards) and songbirds (mostly sparrows and starlings), I was getting desperate.

I had one last hope: Bruny Island, touted as the wild and unspoiled place to see wildlife ... though the wildlife touted in the tourist brochures mostly seems to involve paying for a boat ride to see sea birds and fur seals. Nobody ever talks about marsupials wandering around -- are there any? Since it's an island, how would they get there? Nobody ever mentions the intriguing spot marked "penguin rookery" on "The Neck" between North and South Bruny. After last year's tremendous experience at the Philip Island Penguin Parade, I thought it might be worth booking a room on Bruny in the hope of seeing (a) penguins and (b) other nocturnal wildlife.

We booked into the "Bruny Island Hotel", a tiny pub with two lodging units billing itself as "Australia's Southernmost Hotel" (a claim dubious claim -- we saw plenty of lodging farther south, though their actual names didn't include the word "hotel"). We were a little taken aback when we saw the place but it turned out to be clean and comfortable, and right on the bay. And the pub had some wonderful aromas from the daily curry special (which, we found that night, tasted as good as it smelled).

Since we'd caught an early ferry, we spent the day exploring Bruny, including a bushwalk up to Mt. Mangana. The narrow and overgrown trail climbs steadily through thick forest, but the adventurous part of the hike came in one of the few sunny, rocky clearings, where a quite large black snake (something between a meter and a meter and a half long and as thick around as Dave's wrist) slithered off the trail right in front of me. Then right after that, Dave spotted a much smaller snake, the size of a large garter snake, a bit off the trail. Should I mention that all Tasmanian snakes are venomous? (Checking the books later, the large one was a black tiger snake -- quite dangerous -- while the smaller one was probably a white-lipped snake, considered only moderately dangerous.)

After that our appreciation of the scenery declined a bit as we kept our eyes glued to the trail ahead of us, but we saw no more snakes and eventually emerged into a clearing that gave us great views of a radio tower but no views of much of anything else. On Mt Mangana, the journey is the point, not the destination.

On the way back down, when we got to the rocky clearing, both of our colubrid friends were there to meet us. Dave, in the lead, stamped a bit and the larger snake slithered off ahead of us on the trail -- not quite the reaction we'd been hoping for -- while the smaller snake coiled into a ball but remained off the trail. Eventually the large snake left the trail and Dave quickly passed it while I snapped a shot of its disappearing tail. Now it was my turn to pass -- but the snake was no longer visible. Where was it now? I was searching the trailside where it had disappeared when I heard a rustling in the bush beside and behind me and saw the snake's head appearing -- it had circled around behind me! (I'm sure this wasn't a strategic move, merely some sort of coincidence: I used to keep snakes and though they're fascinating and beautiful, intelligence isn't really their strong point.) I high-tailed it down the trail and we finished the walk safely.

That evening, we headed over to the penguin rookery, where it turned out that we had happened to choose the one night when there was a ranger talk and program there. I wasn't sure whether that was a good or a bad thing, since it meant a crowd, but it turned out all to the good, partly because it meant a lot more high-powered red-masked flashlights to point out the penguins, but mostly because the real show there isn't penguins at all.

The Bruny Island penguin rookery is also a rookery for short-tailed shearwaters -- known as "muttonbirds" because they're "harvested" for their meat, said to taste like mutton. Their life cycle is fascinating. They spend the nothern hemisphere summer up in the Bering Sea near Alaska, but around September they migrate down to southern Australia, a trip that takes about a week and a half including stopping to feed. They breed and lay a single egg, which both parents incubate until it hatches in mid-January. Then the parents feed the chick until it grows to twice the size of its parents (some 10 kg! while still unable to fly). Then the parents leave the chicks and fly back north. This is the stage at which the overgrown chicks are "harvested" for meat. The chicks who don't get picked off (they're protected in Tasmania) live off their fat deposits until their flight feathers come in, at which point they fly north to join the adults.

We were there about a week after hatching, while the parents were feeding the chicks. The adult shearwaters spend all day fishing while the chick sleeps in a burrow in the sand. At sunset, the adults come flying back, where they use both voice and vision to locate the right burrow. The catch: a bird that migrates from Alaska to Tasmania, and takes casual flights to Antarctica for food, is designed to fly fast. Shearwaters aren't especially good at landing in confined spaces, especially when loaded with fish. The other catch is that there are many thousands of them (the ranger said there were 14,000 nesting at that rookery alone).

So, come dusk, the air is filled with thousands of fast-flying shearwaters circling and looking for their burrows and working up the nerve to land, which they eventually do with a resounding thump. They crash into bushes, the boardwalk, or, uncommonly, people who are there to watch the show. It's kind of like watching the bats fly out of Carlsbad caverns ... if the bats weighed five kilos each and flew at 20-30mph. The night fills with the eerie cries of shearwaters calling to each other, the growling of shearwaters fighting over burrows, and the thumps of shearwaters making bad landings.

Penguins? We saw a few, mostly chicks coming out of their burrows to await a food-carrying parent, and late in the evening a handful came out of the water and climbed the beach. Penguins normally find each other by sound, and at Philip Island they were quite noisy, but at Bruny most of the penguins we saw were silent (we did hear a few penguin calls mixed in with the cacophony of shearwaters). But we didn't really miss the penguins with the amazing shearwater show.

When we finally drove back to the hotel, we drove slowly, hoping to see nocturnal wildlife. We knew by then that Bruny does have mammals (however they might have gotten there) because of the universal sign: roadkill. And we did see wildlife: three penguins, two small red wallabies, three smaller red animals with fuzzy tails (ringtailed and brushtailed possums?) and one barely-glimpsed small sand-colored animal the size and shape of a weasel (I wonder if it could have been a brown bandicoot? It didn't look mouselike and didn't have spots like a quoll).

Success! A spectacular evening.

Tags: travel, australia, tasmania, nature, birds
[ 12:24 Feb 11, 2009    More travel/tasmania | permalink to this entry | ]

Widescreen laptop for presentations

I've written before about how I'd like to get a netbook like an Asus Eee, except that the screen resolution puts me off: no one makes a netbook with vertical resolution of more than 600. Since most projectors prefer 1024x768, I'm wary of buying a laptop that can't display that resolution.

(What was wrong with my beloved old Vaio? Nothing, really, except that the continued march of software bloat means that a machine that can't use more than 256M RAM is hurting when trying to run programs (*cough* Firefox *cough) that start life by grabbing about 90M and goes steadily up from there. I can find lightweight alternatives for nearly everything else, but not for the browser -- Dillo just doesn't cut it.)

Ebay turned out to be the answer: there are lots of subnotebooks there, nice used machines with full displays at netbook prices. And so a month before LCA I landed a nice Vaio TX650 with 1.5G RAM, Pentium M, Intel 915GM graphics and Centrino networking. All nice Linux-supported hardware.

But that raised another issue: how do widescreen laptops (the TX650 is 1366x768) talk to a projector? I knew it was possible -- I see people presenting from widescreen machines all the time -- but nobody ever writes about how it works.

The first step was to get it talking to an external monitor at all. I ran a VGA cable to my monitor, plugged the other end into the Vaio (it's so nice not to need a video dongle!) and booted. Nothing. Hmm.

But after some poking and googling, I learned that with Intel graphics, xrandr is the answer:

xrandr --output VGA --mode 1024x768

switches the external VGA signal on, and

xrandr --auto

switches it back off.

Update, April 2010: With Ubuntu Lucid, this has changed and now it's
xrandr --output VGA1 --mode 1024x768
-- in other words, VGA changed to VGA1. You can run xrandr with no arguments to get a list of possible output devices and find out whether X sees the external projector or screen correctly.

Well, mostly. Sometimes it doesn't work -- like, unfortunately, at the lightning talk session, so I had to give my talk without visuals. I haven't figured that out yet. Does the projector have to be connected before I run xrandr? Should it not be connected until after I've already run xrandr? Once it's failed, it doesn't help to run xrandr again ... but a lot of fiddling and re-plugging the cable and power cycling the projector can sometimes fix the problem, which obviously isn't helpful in a lightning talk situation.

Eventually I'll figure that out and blog it (ideas, anyone?) but the real point of today's article is resolution. What I wanted to know was: what happened to that wide 1366-pixel screen when I was projecting 1024 pixels? Would it show me some horrible elongated interpolated screen? Would it display on the left part of the laptop screen, or the middle part?

The answer, I was happy to learn, is that it does the best thing possible: it sends the leftmost 1024 pixels to the projector, while still showing me all 1366 pixels on the laptop screen.

Why ... that means ... I can write notes for myself, to display in the rightmost 342 screen pixels! All it took was a little bit of CSS hacking in my HTML slide presentation package, and it worked fine. Now I have notes just like my Mac friends with their Powerpoint and their dual-head video cards, only I get to use Linux and HTML. How marvellous! I could get used to this widescreen stuff.

Tags: laptop, X11, linux, speaking, projector, lca2009, linux.conf.au
[ 22:12 Feb 06, 2009    More linux/laptop | permalink to this entry | ]

Use GIMP Tool Presets for setting multiple crop aspect ratios

Making desktop backgrounds in GIMP is a bit tedious if you have several machines with screens of different sizes. The workflow goes something like this:

First, choose Crop tool and turn on Fixed: Aspect Ratio. Then loop over images:

1. Load an image
2. Go to Tool Options
3. Type in the aspect ratio: 4:3, 8:5, 5:4, 1366:768 etc.
4. Go to the image and crop.
5. Image->Scale (I have this on Shift-S, can't remember whether that was a default binding or not).
6. Ctrl-K to delete the current width (Ctrl-U also works, but beeps; I'm not sure why)
7. Type in the desired width (1024 or 1680 or 1366 or whatever) (I always hit Tab here, though it's probably not necessary)
8. Click Scale or type Alt-S (unfortunately, Return doesn't work in this dialog).
9. Save As to the appropriate name and path for the current resolution
10. Undo (the Scale), Undo (the Crop)
11. Load a new image (continue loop)

But you can use Save Options (Tool Presets) to avoid step 3, typing in the aspect ratio. Here's how:

• Set up the aspect ratio you want in the Crop tool, with Fixed checked
• Click on Save Options (the floppy disk icon in the lower left of Tool Options)
• Choose a name (choose New Entry first if you've already saved options).

Repeat, for each aspect ratio you might want to use.

Now clicking on Restore Options gives you a menu of all your commonly used aspect ratios -- much faster than typing them in every time. Too bad there's no way to use this shortcut for the Scale step, or to do Crop and Scale in one operation.

Nice shortcut! But having done that, I realized I could shorten it even more: I could make a selection (not a crop) with one of my preset aspect ratios, then run a script that would figure out from the aspect ratio which absolute size I wanted, crop and scale, and either save it to the right place, or make a new image so I could save without needing to Redo or Save a Copy. That was an easy script to write, so here it is: wallpaper.py.

Tags: gimp, tips, programming, hack
[ 23:45 Feb 05, 2009    More gimp | permalink to this entry | ]

Tasmania Photos

I still haven't finished writing up a couple of blog entries from bumming around Tasmania after LCA2009, but I did get some photos uploaded: Tasmania photos. Way too many photos of cute Tassie devils and other animals at the Bonorong wildlife park, as well as the usual collection of scenics and silly travel photos.

Tags: travel, tasmania, lca2009, nature, photo
[ 15:49 Feb 04, 2009    More travel/tasmania | permalink to this entry | ]