Shallow Thoughts : tags : programming

Finding versions of installed packages in Debian/Ubuntu

Checking versions in Debian-based systems is a bit of a pain.

This happens to me a couple of times a month: for some reason I need to know what version of something I'm currently running -- often a library, like libgtk. aptitude show will tell you all about a package -- but only if you know its exact name. You can't do aptitude show libgtk or even aptitude show '*libgtk*' -- you have to know that the package name is libgtk2.0-0. Why is it libgtk2.0-0? I have no idea, and it makes no sense to me.

So I always have to do something like aptitude search libgtk | egrep '^i' to find out what packages I have installed that matches the name libgtk, find the package I want, then copy and paste that name after typing aptitude show.

But it turns out it's super easy in Python to query Debian packages using the Python apt package. In fact, this is all the code you need:

import sys
import apt

cache = apt.cache.Cache()

pat = sys.argv[1]

for pkgname in cache.keys():
    if pat in pkgname:
        pkg = cache[pkgname]
        instver = pkg.installed
        if instver:
            print pkg.name, instver.version

Then run aptver libgtk and you're all set.

In practice, I wanted nicer formatting, with columns that lined up, so the actual script is a little longer. I also added a -u flag to show uninstalled packages as well as installed ones. Amusingly, the code to format the columns took about twice as many lines as the code that does the actual work. There doesn't seem to be a standard way of formatting columns in Python, though there are lots of different implementations on the web. Now there's one more -- in my aptver on github.

Tags: linux, debian, ubuntu, python, programming
[ 15:07 May 15, 2013    More linux | permalink to this entry | comments ]

Parsing NOAA historical weather data

We've been considering the possibility of moving out of the Bay Area to somewhere less crowded, somewhere in the desert southwest we so love to visit. But that also means moving to somewhere with much harsher weather.

How harsh? It's pretty easy to search for a specific location and get average temperatures. But what if I want to make a table to compare several different locations? I couldn't find any site that made that easy.

No problem, I say. Surely there's a Python library, I say. Well, no, as it turns out. There are Python APIs to get the current weather anywhere; but if you want historical weather data, or weather data averaged over many years, you're out of luck.

NOAA purports to have historical climate data, but the only dataset I found was spotty and hard to use. There's an FTP site containing directories by year; inside are gzipped files with names like 723710-03162-2012.op.gz. The first two numbers are station numbers, and there's a file at the top level called ish-history.txt with a list of the station codes and corresponding numbers. Not obvious!

Once you figure out the station codes, the files themselves are easy to parse, with lines like

STN--- WBAN   YEARMODA    TEMP       DEWP      SLP        STP       VISIB      WDSP     MXSPD   GUST    MAX     MIN   PRCP   SNDP   FRSHTT
724945 23293  20120101    49.5 24    38.8 24  1021.1 24  1019.5 24    9.9 24    1.5 24    4.1  999.9    68.0    37.0   0.00G 999.9  000000

Each line represents one day (20120101 is January 1st, 2012), and the codes are explained in another file called GSOD_DESC.txt. For instance, MAX is the daily high temperature, and SNDP is snow depth.

So all I needed was to decode the station names, download the right files and parse them. That took about a day to write (including a lot of time wasted futzing with mysterious incantations for matplotlib).

Little accessibility refresher: I showed it to Dave -- "Neat, look at this, San Jose is the blue pair, Flagstaff is green and Page is red." His reaction: "This makes no sense. They all look the same to me. I have no idea which is which." Oops -- right. Don't use color as your only visual indicator. I knew that, supposedly! So I added markers in different shapes for each site. (I wish somebody would teach that lesson to Google Maps, which uses color as its only indicator on the traffic layer, so it's useless for red-green colorblind people.)

Back to the data -- it turns out NOAA doesn't actually have that much historical data available for download. If you search on most of these locations, you'll find sites that claim to have historical temperatures dating back 50 years or more, sometimes back to the 1800s. But NOAA typically only has files starting at about 2005 or 2006. I don't know where sites are getting this older data, or how reliable it is.

Still, averages since 2006 are still interesting to compare. Here's a run of noaatemps.py KSJC KFLG KSAF KLAM KCEZ KPGA KCNY. It's striking how moderate California weather is compared to any of these inland sites. No surprise there. Another surprise was that Los Alamos, despite its high elevation, has more moderate weather than most of the others -- lower highs, higher lows. I was a bit disappointed at how sparse the site list was -- no site in Moab? Really? So I used Canyonlands Field instead.

Anyway, it's fun for a data junkie to play around with, and it prints data on other weather factors, like precipitation and snowpack, although it doesn't plot them yet. The code is on my GitHub scripts page, under Weather.

Anyone found a better source for historical weather information? I'd love to have something that went back far enough to do some climate research, see what sites are getting warmer, colder, or seeing greater or lesser spreads between their extreme temperatures. The NOAA dataset obviously can't do that, so there must be something else that weather researchers use. Data on other countries would be interesting, too. Is there anything that's available to the public?

Tags: python, programming, weather, data
[ 21:57 Apr 13, 2013    More programming | permalink to this entry | comments ]

Letters not used in Python keywords

One of the closing lightning talks at PyCon this year concerned the answers to a list of Python programming puzzles given at some other point during the conference. I hadn't seen the questions (I'm still not sure where they are), but some of the problems looked fun.

One of them was: "What are the letters not used in Python keywords?" I hadn't known about Python's keyword module, which could come in handy some day:

>>> import keyword
>>> keyword.kwlist
['and', 'as', 'assert', 'break', 'class', 'continue', 'def', 'del', 'elif', 'else', 'except', 'exec', 'finally', 'for', 'from', 'global', 'if', 'import', 'in', 'is', 'lambda', 'not', 'or', 'pass', 'print', 'raise', 'return', 'try', 'while', 'with', 'yield']

So, given the list of keywords, what's the best way to find the list of unique letters?

Any time you want a list of unique anything, you want a set. For instance,

>>> set([1, 2, 3, 2, 2, 4, 5, 1, 5])
set([1, 2, 3, 4, 5])

But first you need a list of letters so can make a set out of it.

Split the list of words into a list of letters

My first idea was to use list comprehensions. You can split a single word into letters like this:

>>> [ x for x in 'hello' ]
['h', 'e', 'l', 'l', 'o']

It took a bit of fiddling to get the right syntax to apply that to every word in the list:

>>> [[c for c in w] for w in keyword.kwlist]
[['a', 'n', 'd'], ['a', 's'], ['a', 's', 's', 'e', 'r', 't'], ... ]

Update: Dave Foster points out that [list(w) for w in keyword.kwlist] is another way, simpler and cleaner way than the double list comprehension.

That's a list of lists, so it needs to be flattened into a single list of letters before we can turn it into a set.

Flatten the list of lists

There are lots of ways to flatten a list of lists. Here are four of them:

[item for sublist in [[c for c in w] for w in keyword.kwlist] for item in sublist]

reduce(lambda x,y: x+y, [[c for c in w] for w in keyword.kwlist])

import itertools
list(itertools.chain.from_iterable([[c for c in w] for w in keyword.kwlist]))

sum([[c for c in w] for w in keyword.kwlist], [])

That last one, using sum(), makes use of the fact that Python uses + for list concatenation -- in other words, that [1, 2, 3] + [4, 5, 6] is [1, 2, 3, 4, 5, 6]. But the first method (item for sublist in) is faster: see Making a flat list out of list of lists in Python on StackOverflow. And another StackOverflow thread has a nice script for plotting speed vs. list size of various flatteners.

A simpler way of making the set

But it turns out none of this list comprehension stuff is needed anyway. set('word') splits words into letters already:

>>> set('bubble')
set(['e', 'b', 'u', 'l'])

Ignore the order -- elements of a set often end up displaying in some strange order. The important thing is that it has all the letters and no repeats.

Now we have an easy way of making a set containing the letters in one word. But how do we apply that to a list of words?

Again I initially tried using list comprehensions, then realized there's an easier way. Given a list of strings, it's trivial to join them into a single string using ''.join(). And that gives us our set of letters within keywords:

>>> set(''.join(keyword.kwlist))
set(['a', 'c', 'b', 'e', 'd', 'g', 'f', 'i', 'h', 'k', 'm', 'l', 'o', 'n', 'p', 's', 'r', 'u', 't', 'w', 'y', 'x'])

What letters are not in the set?

Almost done! But the original problem was to find the letters not in keywords. We can do that by subtracting this set from the set of all letters from a to z. How do we get that? The string module will give us a list:

>>> string.lowercase
'abcdefghijklmnopqrstuvwxyz'

You could also use a list comprehension and ord and chr (alas, range won't give you a range of letters directly):

>>> [chr(i) for i in range(ord('a'), ord('z')+1)]
['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']

It's a bit longer, but doesn't require an import.

Now that you have your a-z set, just subtract the two sets:

>>> set(string.lowercase[:]) - set(''.join(keyword.kwlist))
set(['q', 'j', 'z', 'v'])

So the only letters not used in Python keywords are q, j, z and v.

Just a useless little ditty, really ... but I thought it was a fun exercise, so maybe you will too.

Tags: programming, python
[ 12:36 Mar 19, 2013    More programming | permalink to this entry | comments ]

SimpleCV on Raspberry Pi

I'm at PyCon, and I spent a lot of the afternoon in the Raspberry Pi lab.

Raspberry Pis are big at PyCon this year -- because everybody at the conference got a free RPi! To encourage everyone to play, they have a lab set up, well equipped with monitors, keyboards, power and ethernet cables, plus a collection of breadboards, wires, LEDs, switches and sensors.

I'm primarily interested in the RPi as a robotics controller, one powerful enough to run a camera and do some minimal image processing (which an Arduino can't do). And on Thursday, I attended a PyCon tutorial on the Python image processing library SimpleCV. It's a wrapper for OpenCV that makes it easy to access parts of images, do basic transforms like greyscale, monochrome, blur, flip and rotate, do edge and line detection, and even detect faces and other objects. Sounded like just the ticket, if I could get it to work on a Raspberry Pi.

SimpleCV can be a bit tricky to install on Mac and Windows, apparently. But the README on the SimpleCV git repository gives an easy 2-line install for Ubuntu. It doesn't run on Debian Squeeze (though it installs), because apparently it depends on a recent version of pygame and Squeeze's is too old; but Ubuntu Pangolin handled it just fine.

The question was, would it work on Raspbian Wheezy? Seemed like a perfect project to try out in the PyCon RPi lab. Once my RPi was set up and I'd run an apt-get update, I used used netsurf (the most modern of the lightweight browsers available on the RPi) to browse to the SimpleCV installation instructions. The first line,

sudo apt-get install ipython python-opencv python-scipy python-numpy python-pygame python-setuptools python-pip

was no problem. All those packages are available in the Raspbian repositories.

But the second line,

sudo pip install https://github.com/ingenuitas/SimpleCV/zipball/master

failed miserably. Seems that pip likes to put its large downloaded files in /tmp; and on Raspbian, running off an SD card, /tmp quite reasonably is a tmpfs, running in RAM. But that means it's quite small, and programs that expect to be able to use it to store large files are doomed to failure.

I tried a couple of simple Linux patches, with no success. You can't rename /tmp to replace it with a symlink to a directory on the SD card, because /tmp is always in use. And pip makes a new temp directory name each time it's run, so you can't just symlink the pip location to a place on the SD card.

I thought about rebooting after editing the tmpfs out of /etc/fstab, but it turns out it's not set up there, and it wasn't obvious how to disable the tmpfs. Searching later from home, the size is set in /etc/default/tmpfs. As for disabling the tmpfs and using the SD card instead, it's not clear. There's a block of code in /etc/init.d/mountkernfs.sh that makes that decision; it looks like symlinking /tmp to somewhere else might do it, or else commenting out the code that sets RAMTMP="yes". But I haven't tested that.

Instead of rebooting, I downloaded the file to the SD card:

wget https://github.com/ingenuitas/SimpleCV/master

But it turned out it's not so easy to pip install from a local file. After much fussing around I came up with this, which worked:

pip install http:///home/pi/master --download-cache /home/pi/tmp

That worked, and the resulting SimpleCV install worked nicely! I typed some simple tests into the simplecv shell, playing around with their built-in test image "lenna":

img = Image('lenna')
img.show()
img.binarize().show()
img.toGray().show()
img.edges().show()
img.invert().show()

And, for something a little harder, some face feature detection: let's find her eyes and outline them in yellow.

img.listHaarFeatures()
img.findHaarFeatures('eye.xml').draw(color=Color.YELLOW)

SimpleCV is lots of fun! And the edge detection was quite fast on the RPi -- this may well be usable by a robot, once I get the motors going.

Tags: raspberry pi, python, programming, hardware, linux
[ 20:43 Mar 16, 2013    More linux/install | permalink to this entry | comments ]

Whac-a-Moon: Watch Europa appear and disappear this Sunday

This is an edited and updated version of my "Shallow Sky" column this month in the SJAA Ephemeris newsletter.

A few months ago, I got email from a Jupiter observer calling my attention to an interesting phenomenon of Jupiter's moons that I hadn't seen before. The person who mailed me described himself as a novice, and wasn't quite sure what he had seen, but he knew it was odd. After some further discussion we pinned it down.

He was observing Jupiter at 11/11/12 at 00.25 UT (which would have been mid-afternoon here in San Jose). Three of the moons were visible, with only Ganymede missing. Then Ganymede appeared: near Jupiter's limb, but not right on it. As he watched over the next few minutes, Ganymede seemed to be moving backward -- in toward Jupiter rather than away from it. Eventually it disappeared behind the planet.

It turned out that what he was seeing was the end of an eclipse. Jupiter was still a few months away from opposition, so the shadow thrown by the planet streamed off to one side as seen from our inner-planet vantage point on Earth. At 0:26 UT on that evening, long before he started observing, Ganymede, still far away from Jupiter's limb, had entered Jupiter's shadow and disappeared into eclipse. It took over two hours for Ganymede to cross Jupiter's shadow; but at 2:36, when it left the shadow, it hadn't yet disappeared behind the planet. So it became visible again. It wasn't until 2:50 that Ganymede finally disappeared behind Jupiter.

So it was an interesting effect -- bright Ganymede appearing out of nowhere, moving in toward Jupiter then disappearing again fourteen minutes later. It was something I'd never seen, or thought to look for. It's sort of like playing Whac-a-mole -- the moon appears only briefly, so you've got to hit it with your telescope at just the right time if you want to catch it before it disappears again.

A lot of programs don't show this eclipse effect -- including, I'm sad to say, my own Javascript Jupiter's moons web page. (I have since remedied that.) The open source program Stellarium shows the effect; on the web, Sky and Telescope's Jupiter's Moons page shows it, and even prints out a table of times of various moon events, including eclipses.

These eclipse events aren't all that uncommon -- but only when the sun angle is just right. Searching in late February and early March this year, I found several events for Ganymede and Europa (though, sadly, many of them were during our daytime). By mid-March, the angles have changed so that Europa doesn't leave Jupiter's shadow until after it's disappeared behind the planet's limb; but Ganymede is farther out, so we can see Ganymede appearances all the way through March and for months after.

The most interesting view, it seems to me, is right on the boundary when the moon only appears for a short time before disappearing again. Like the Europa eclipse that's happening this Sunday night, March 10.

Reporting on that one got a little tricky -- because that's the day we switch to Daylight Savings time. I have to confess that I got a little twisted up trying to compare results between programs that use UTC and programs that use local time -- especially when the time zone converter I was using to check my math told me "That time doesn't exist!" Darnit, if we'd all just use UTC all the time, astronomy calculations would be a lot easier! (Not to mention dropping the silly Daylight Savings Time fiasco, but that's another rant.)

Before I go into the details, I want to point out that Jupiter's moons are visible even in binoculars. So even if you don't have a telescope, grab binoculars and set them up in as steady a way as you can -- if you don't have a tripod adaptor, try bracing them on the top of a gate or box.

On Sunday night, March 10, at some time around 7:40 pm PDT, Europa peeks out from behind Jupiter's northeast limb. (All times are given in PDT; add 7 hours for GMT.) The sky will still be bright here in California -- the sun sets at 7:12 that night -- but Jupiter will be 66 degrees up and well away from the sun, so it shouldn't give you too much trouble. Once Europa pops out, keep a close eye on it -- because if Sky & Tel's calculations are right, it will disappear again just four minutes later, at 7:44, into eclipse in Jupiter's shadow. It will remain invisible for almost three hours, finally reappearing out of nowhere, well off Jupiter's limb, at around 10:24 pm.

Here's a link to my Javascript Jupiter just before Europa reappears.

I want to stress that those times are approximate. In fact, I tried simulating the event in several different programs, and got wildly varying times:

	Io disappears	Europa appears	Europa disappears	Europa reappears	Io appears
XEphem	7:15	7:43	7:59	10:06	10:43
S&T Jupiter's Moons	7:16	7:40	7:44	10:24	10:48
Javascript Jupiter	7:17	7:45	7:52	10:15	10:41
Stellarium	6:21	6:49	7:05	9:32	10:01

You'll note Stellarium seems to have a time zone problem ... maybe because I ran the prediction while we were still in standard time, not daylight savings time.

I'm looking forward to timing the events to see which program is most accurate. I'm betting on XEphem. Once I know the real times, maybe I can adjust my Javascript Jupiter's code to be more accurate. If anyone else times the event, please send me your times, in case something goes wrong here!

Anyway, the spread of times makes it clear that when observing this sort of phenomenon, you should always set up the telescope ten or fifteen minutes early, just in case. And ten extra minutes spent observing Jupiter -- even without moons -- is certainly never time wasted! Just keep an eye out for Europa to appear -- and be ready to whack that moon before it disappears again.

Tags: astronomy, programming, javascript
[ 10:30 Mar 09, 2013    More science/astro | permalink to this entry | comments ]

New project: Metapho image tagger

I'm excited about my new project: MetaPho, an image tagger.

It arose out of a discussion on the LinuxChix Techtalk list: photo collection management software. John Sturdy was looking for an efficient way of viewing and tagging large collections of photos. Like me, he likes fast, lightweight, keyboard-driven programs. And like me, he didn't want a database-driven system that ties you forever to one image cataloging program. I put my image tags in plaintext files, named Keywords, so that I can easily write scripts to search or modify them, or user grep, and I can even make quick changes with a text editor.

I shared some tips on how I use my Pho image viewer for tagging images, and it sounded close to what he was looking for. But as we discussed ideas about image tagging, we realized that there were things he wanted to do that pho doesn't do well, things not offered by any other image tagger we've been able to find. While discussing how we might add new tagging functionality to pho, I increasingly had the feeling that I was trying to fit off-road tires onto a Miata -- or insert your own favorite metaphor for "making something do something it wasn't designed to do."

Pho is a great image viewer, but the more I patched it to handle tagging, the uglier and more complicated the code got, and it also got more complex to use.

And really, everything we needed for tagging could be easily done in a Python-GTK application. (Pho is written in C because it does a lot of complicated focus management to deal with how window managers handle window moving and resizing. A tagger wouldn't need any of that.)

I whipped up a demo image viewer in a few hours and showed it to John. We continued the discussion, I made a GitHub repo, and over the next week or so the code grew into an efficient and already surprisingly usable image tagger.

We have big plans for it, like tags organized into categories so we can have lots of tags without cluttering the interface too much. But really, even as it is, it's better than anything I've used before. I've been scanning in lots of photos from old family albums (like this one of my mother and grandmother, and me at 9 months) and it's been great to be able to add and review tags easily.

If you want to check out MetaPho, or contribute to it (either code or user interface design), it lives in my MetaPho repository on GitHub. And I wrote up a quick man page in markdown format: metapho.1.md.

Feedback and contributors welcome!

Tags: programming, pho, image viewer, python, tagging, metapho
[ 18:31 Feb 21, 2013    More programming | permalink to this entry | comments ]

Converting C to Python with a vi regexp

I'm fiddling with a serial motor controller board, trying to get it working with a Raspberry Pi. (It works nicely with an Arduino, but one thing I'm learning is that everything hardware-related is far easier with Arduino than with RPi.)

The excellent Arduino library helpfully provided by Pololu has a list of all the commands the board understands. Since it's Arduino, they're in C++, and look something like this:

#define QIK_GET_FIRMWARE_VERSION         0x81
#define QIK_GET_ERROR_BYTE               0x82
#define QIK_GET_CONFIGURATION_PARAMETER  0x83
[ ... ]
#define QIK_CONFIG_DEVICE_ID                        0
#define QIK_CONFIG_PWM_PARAMETER                    1

and so on.

On the Arduino side, I'd prefer to use Python, so I need to get them to look more like:

    QIK_GET_FIRMWARE_VERSION = 0x81
    QIK_GET_ERROR_BYTE = 0x82
    QIK_GET_CONFIGURATION_PARAMETER = 0x83
[ ... ]
    QIK_CONFIG_DEVICE_ID = 0
    QIK_CONFIG_PWM_PARAMETER = 1

... and so on ... with an indent at the beginning of each line since I want this to be part of a class.

There are 32 #defines, so of course, I didn't want to make all those changes by hand. So I used vim. It took a little fiddling -- mostly because I'd forgotten that vim doesn't offer + to mean "one or more repetitions", so I had to use * instead. Here's the expression I ended up with:

.,$s/\#define *\([A-Z0-9_]*\) *\(.*\)/ \1 = \2/

In English, you can read this as:

From the current line to the end of the file (,.$/), look for a pattern consisting of only capital letters, digits and underscores ([A-Z0-9_]). Save that as expression #1 (\( \)). Skip over any spaces, then take the rest of the line (.*), and call it expression #2 (\( \)).

Then replace all that with a new line consisting of 4 spaces, expression 1, a spaced-out equals sign, and expression 2 ( \1 = \2).

Who knew that all you needed was a one-line regular expression to translate C into Python?

(Okay, so maybe it's not quite that simple. Too bad a regexp won't handle the logic inside the library as well, and the pin assignments.)

Tags: regexp, linux, programming, python, editors, vim
[ 20:38 Jan 19, 2013    More linux/editors | permalink to this entry | comments ]

Comparing sunset times with PyEphem

We were marveling at how early it's getting dark now -- seems like a big difference even compared to a few weeks ago. Things change fast this time of year.

Since we're bouncing back and forth a lot between southern and northern California, Dave wondered how Los Angeles days differed from San Jose days. Of course, San Jose being nearly 4 degrees farther north, it should have shorter days -- but by the weirdness of orbital mechanics that doesn't necessarily mean that the sun sets earlier in San Jose. His gut feel was that LA was actually getting an earlier sunset.

"I can calculate that," I said, and fired up a Python interpreter to check with PyEphem. Since PyEphem doesn't know San Jose (hmph! San Jose is bigger than San Francisco) I used San Francisco.

Since PyEphem's Observer class only has next_rising() and next_setting(), I had to set a start date of midnight so I could subtract the two dates properly to get the length of the day.

>>> sun = ephem.Sun()
>>> la = ephem.city('Los Angeles')
>>> sf = ephem.city('San Francisco')
>>> 
>>> mid = ephem.Date('2012/10/31 8:00')
>>> 
>>> la.next_rising(sun, start=mid)
2012/10/31 14:11:57
>>> la.next_setting(sun, start=mid)
2012/11/1 01:00:45
>>> la.next_setting(sun, start=mid) - la.next_rising(sun, start=mid)
0.45055988136300584
>>> 
>>> sf.next_rising(sun, start=mid)
2012/10/31 14:34:19
>>> sf.next_setting(sun, start=mid)
2012/11/1 01:11:44
>>> sf.next_setting(sun, start=mid) - sf.next_rising(sun, start=mid)
0.4426457611261867

So Dave's intuition was right: northern California really does have a later sunset than southern California at this time of year, even though the total day length is shorter -- the difference in sunrise time makes up for the later sunset.

How much shorter?

>>> (la.next_setting(sun, start=mid) - la.next_rising(sun, start=mid)) - (sf.next_setting(sun, start=mid) - sf.next_rising(sun, start=mid))
0.007914120236819144
>>> ((la.next_setting(sun, start=mid) - la.next_rising(sun, start=mid)) - (sf.next_setting(sun, start=mid) - sf.next_rising(sun, start=mid))) * 24
0.18993888568365946
>>> ((la.next_setting(sun, start=mid) - la.next_rising(sun, start=mid)) - (sf.next_setting(sun, start=mid) - sf.next_rising(sun, start=mid))) * 24 * 60
11.396333141019568

And we have our answer -- there's about 11 minutes difference in day length between SF and LA.

Tags: astronomy, programming, python
[ 10:46 Oct 31, 2012    More science/astro | permalink to this entry | comments ]

Asynchronous sound playing in Python

A little while back I wrote about my Python xchat script to play sound alerts.

But one thing that's been annoying me about it -- it was a problem with the old perl alert script too -- is the repeated sounds. If lots of twitter updates come in on the Bitlbee channel, or if someone pastes numerous lines into a channel, I hear POPPOPPOPPOPPOPPOP or repetitions of whatever the alert sound is for that type of message. It's annoying to me, but even more so to anyone else in the same room.

It would be so much nicer if I could have it play just one repetition of any given alert, even if there are eight lines all coming in at the same time. So I decided to write a Python class to handle that.

My existing code used subprocesses to call the basic ALSA sound player, /usr/bin/aplay -q. Initially I used
if not os.fork() : os.execl(APLAY, APLAY, "-q", alertfile)
but I later switched to the cleaner
subprocess.call([APLAY, '-q', alertfile])
But of course, it would be better to do it all from Python without requiring an external process like aplay. So I looked into that first.

Sadly, it turns out Python audio support is a mess. The built-in libraries are fairly limited in functionality and formats, and the external libraries that handle sound are mostly unmaintained, unless you want to pull in a larger library like pygame. After a little web searching I decided that maybe an aplay subprocess wasn't so bad after all.

Okay, so how should I handle the subprocesses? I decided the best way was to keep track of what sound was currently playing. If another alert fires for the same sound while that sound is already playing, just ignore it. If an alert comes in for a different sound, then wait() for the current sound to finish, then start the new sound.

That's all quite easy with Python's subprocess module. subprocess.Popen() returns a Popen object that tracks a process ID and can check whether that process has finished or not. If self.curpath is the path to the sound currently playing and self.current is the Popen object for whatever aplay process is currently running, then:

    if self.current :
        if self.current.poll() == None :
            # Current process hasn't finished yet. Is this the same sound?
            if path == self.curpath :
                # A repeat of the currently playing sound.
                # Don't play it more than once.
                return
            else :
                # Trying to play a different sound.
                # Wait on the current sound then play the new one.
                self.wait()

    self.curpath = path
    self.current = subprocess.Popen([ "/usr/bin/aplay", '-q', path ] )

Finally, it's a good idea when exiting the program to check whether any aplay process is running, and wait() for it. Otherwise, you might end up with a zombie aplay process.

    def __del__(self) :
        self.wait()

I don't know if xchat actually closes down Python objects gracefully, so I don't know whether the __del__ destructor will actually be called. But at least I tried. It's possible that a context manager might be more reliable.

The full scripts are on github at pyplay.py for the basic SoundPlayer class, and chatsounds.py for the xchat script that includes SoundPlayer.

Tags: programming, python, audio
[ 12:07 Oct 17, 2012    More programming | permalink to this entry | comments ]

Writing xchat scripts in Python (to play sound alerts)

I use xchat as my IRC client. Mostly I like it, but its sound alerts aren't quite as configurable as I'd like. I have a few channels, like my Bitlbee Twitter feed, where I want a much more subtle alert, or no alert at all. And I want an easy way of turning sounds on and off, in case I get busy with something and need to minimize distractions.

Years ago I grabbed a perl xchat plug-in called "Smet's NickSound" that did something close to what I wanted. I've hacked a few things into it. But every time I try to customize it any further, I'm hit with the pain of write-only Perl. I've written Perl scripts, honest. But I always have a really hard time reading anyone else's Perl code and figuring out what it's doing. When I dove in again recently to try to figure out why I was getting so many alerts when first starting up xchat, I finally decided: learning how to write a Python xchat script couldn't be any harder than reverse engineering a Perl one.

First, of course, I looked for an existing nick sound Python script ... and totally struck out. In fact, mostly I struck out on finding any xchat Python scripts at all. I know there are Python bindings for xchat, because there's documentation for them. But sample plug-ins? Nope. For some reason, nobody's writing xchat plug-ins in Python.

I eventually found two minimal examples: this very simple example and the more elaborate utf8decoder. I was able to put them together and cobble up a working nick sound plug-in. It's easy once you have an example to work from to help you figure out the event hook arguments.

So here's my own little example, which may help the next person trying to learn xchat Python scripting: chatsounds.py on github.

Tags: programming, python, xchat, irc
[ 21:13 Sep 26, 2012    More programming | permalink to this entry | comments ]

Python: Do two dates span a particular day of the week or month?

When I'm using my RSS reader FeedMe, I normally check every feed every day. But that can be wasteful: some feeds, like World Wide Words, only update once a week. A few feeds update even less often, like serialized books that come out once a month or whenever the author has time to add something new.

So I decided it would be nice to add some "when" logic to FeedMe, so I could add when = Sat in the config section for World Wide Words and have it only update once a week.

That sounded trivial -- a little python parsing logic to tell days from numbers, a few calls to time.localtime() and I was done.

Except of course I wasn't. Because sometimes, like when I'm on vacation, I don't always update every day. If I missed a Saturday, then I'd never see that week's edition of World Wide Words. And that would be terrible!

So what I really needed was a way to ask, "Has a Saturday occurred (including today) since the last time I ran feedme?"

The last time I ran feedme is easy to determine: it's in the last modified date of the cache file. Or, in more Pythonic terms, it's statbuf = os.stat(cachefile).st_mtime. And of course I can get the current time with time.localtime(). But how do I figure out whether a given week or month day falls between those two dates?

I'm sure this particular wheel has been invented many times. There's probably even a nifty Python library somewhere to do it. But how do you google for that? I tried to think of keywords and found nothing. So I went for a nice walk in the redwoods and thought about it for a bit, and came up with a solution.

Turns out for the week day case, you can just use modular arithmetic: if (weekday_2 - target_weekday) % 7 < (weekday_2 - weekday_1) then the day does indeed fall between the two dates.

Things are a little more complicated for the day of the month, though, because you don't know whether you need mod 30 or 31 or 29 or 28, so you either have to make your own table, or import the calendar module just so you can call calendar.monthrange().

I decided it was easier to use logic: if the difference between the two dates is greater than 31, then it definitely includes any month day. Otherwise, check whether they're in the same month or not, and do greater than/less than comparisons on the three dates.

Throw in a bunch of conversion to make it easy to call, and a bunch of unit tests to make sure everything works and my later tweaks don't break anything, and I had a nice function I could call from Feedme.

falls_between.py on github

Tags: programming, python
[ 21:07 Sep 19, 2012    More programming | permalink to this entry | comments ]

Pho 0.9.8 released

I decided to give myself a birthday present and release version 0.9.8 of Pho, my image viewer, at long last.

I've been using it essentially unchanged for many months now, occasionally tweaking things or fixing minor bugs ... but I haven't run into any bugs in quite a while, and think I've fixed all the pending ones. Been meaning to make a release for a long time, but somehow I keep getting sidetracked and forgetting about it.

This should rationalize the version number again ... the official releases have been 0.9.7-preN forever, but there was an unofficial 0.9.7 and even a 0.9.8 that snuck in along with some patches I got from David Gardner. It's been confusing. So now it's officially 0.9.8, and any figure versions will start with 0.9.9, and we might even see a 1.0 one of these days. (I suppose it's time -- Pho is ten years old!)

So here it is: Pho 0.9.8. I think it's working well. If you're already a Pho user, or if you want a lightweight image viewer that's also good at triaging and annotating large batches of images, you might want to take a look.

Tags: programming, pho, image viewer
[ 12:36 Sep 05, 2012    More programming | permalink to this entry | comments ]

GIMP plug-in to export scaled-down versions of images

In a discussion on Google+arising from my Save/Export clean plug-in, someone said to the world in general

PLEASE provide an option to select the size of the export. Having to scale the XCF then export then throw out the result without saving is really awkward.

I thought, What a good idea! Suppose you're editing a large image, with layers and text and other jazz, saving in GIMP's native XCF format, but you want to export a smaller version for the web. Every time you make a significant change, you have to: Scale (remembering the scale size or percentage you're targeting); Save a Copy (or Export in GIMP 2.8); then Undo the Scale. If you forget the Undo, you're in deep trouble and might end up overwriting the XCF original with a scaled-down version.

If I had a plug-in that would export to another file type (such as JPG) with a scale factor, remembering that scale factor so I didn't have to, it would save me both effort and risk.

And that sounded pretty easy to write, using some of the tricks I'd learned from my Save/Export Clean and wallpaper scripts. So I wrote export-scaled.py

It's still brand new, so if anyone tries it, I'd appreciate knowing if it's useful or if you have any problems with it.

Geeky programming commentary

(Folks not interested in the programming details can stop reading now.)

Linked input fields

One fun project was writing a set of linked text entries for the dialog:

Scale to:

Percentage 100 %

Width: 640

Height: 480

Change any one of the three, and the other two change automatically. There's no chain link between width and height: It's assumed that if you're exporting a scaled copy, you won't want to change the image's aspect ratio, so any one of the three is enough.

That turned out to be surprisingly hard to do with GTK SpinBoxes: I had to read their values as strings and parse them, because the numeric values kept snapping back to their original values as soon as focus went to another field.

Image parasites

Another fun challenge was how to save the scale ratio, so the second time you call up the plug-in on the same image it uses whatever values you used the first time. If you're going to scale to 50%, you don't want to have to type that in every time. And of course, you want it to remember the exported file path, so you don't have to navigate there every time.

For that, I used GIMP parasites: little arbitrary pieces of data you can attach to any image. I've known about parasites for a long time, but I'd never had occasion to use them in a Python plug-in before. I was pleased to find that they were documented in the official GIMP Python documentation, and they worked just as documented. It was easy to test them, too: in the Python console (Filters->Python->Console...), type something like

img = gimp_image_list()[0]
img.parasite_list()
img.parasite_find(img.parasite_list()[0])

and so forth. Nice!

Not prompting for JPG settings

My plug-in was almost done. But when I ran it and told it to save to filenamecopy.jpg, it prompted me with that annoying JPEG settings dialog. Okay, being prompted once isn't so bad. But then when I exported a second time, it prompted me again, and didn't remember the values from before. So the question was, what controls whether the settings dialog is shown, and how could I prevent it?

Of course, I could prompt the user for JPEG quality, then call jpeg-save-file directly -- but what if you want to export to PNG or GIF or some other format? I needed something more general

Turns out, nobody really remembers how this works, and it's not documented anywhere. Some people thought that passing run_mode=RUN_WITH_LAST_VALS when I called pdb.gimp_file_save() would do the trick, but it didn't help.

So I guessed that there might be a parasite that was storing those settings: if the JPEG save plug-in sees the parasite, it uses those values and doesn't prompt. Using the Python console technique I just mentioned, I tried checking the parasites on a newly created image and on an image read in from an existing JPG file, then saving each one as JPG and checking the parasite list afterward.

Bingo! When you read in a JPG file, it has a parasite called 'jpeg-settings'. (The new image doesn't have this, naturally). But after you write a file to JPG from within GIMP, it has not only 'jpeg-settings' but also a second parasite, 'jpeg-save-options'.

So I made the plug-in check the scaled image after saving it, looking for any parasites with names ending in either -settings or -save-options; any such parasites are copied to the original image. Then, the next time you invoke Export Scaled, it does the same search, and copies those parasites to the scaled image before calling gimp-file-save.

That darned invisible JPG settings dialog

One niggling annoyance remained. The first time you get the JPG settings dialog, it pops up invisibly, under the Export dialog you're using. So if you didn't know to look for it by moving the dialog, you'd think the plug-in had frozen. GIMP 2.6 had a bug where that happened every time I saved, so I assumed there was nothing I can do about it.

GIMP 2.8 has fixed that bug -- yet it still happened when my plug-in called gimp_file_save: the JPG dialog popped up under the currently active dialog, at least under Openbox.

There isn't any way to pass window IDs through gimp_file_save so the JPG dialog pops up as transient to a particular window. But a few days after I wrote the export-scaled, I realized there was still something I could do: hide the dialog when the user clicks Save. Then make sure that I show it again if any errors occur during saving.

Of course, it wasn't quite that simple. Calling chooser.hide() by itself does nothing, because X is asynchronous and things don't happen in any predictable order. But it's possible to force X to sync the display: chooser.get_display().sync().

I'm not sure how robust this is going to be -- but it seems to work well in the testing I've done so far, and it's really nice to get that huge GTK file chooser dialog out of the way as soon as possible.

Tags: gimp, programming, python
[ 17:34 Sep 02, 2012    More gimp | permalink to this entry | comments ]

GIMP: Re-uniting Save and Export

In GIMP 2.8, the developers changed the way you save files. "Save" is now used only for GIMP's native format, XCF (and compressed variants like .xcf.gz and .xcf.bz2). Other formats that may lose information on layers, fonts and other aspects of the edited image must be "Exported" rather than saved.

This has caused much consternation and flameage on the gimp-user mailing list, especially from people who use GIMP primarily for simple edits to JPEG or PNG files.

I don't particularly like the new model myself. Sometimes I use GIMP in the way the developers are encouraging, adding dozens of layers, fonts, layer masks and other effects. Much more often, I use GIMP to crop and rescale a handful of JPG photos I took with my camera on a hike. While I found it easy enough to adapt to using Ctrl-E (Export) instead of Ctrl-S (Save), it was annoying that when I exited the app, I'd always get am "Unsaved images" warning, and it was impossible to tell from the warning dialog which images were safely exported and which might not have been saved or exported at all.

But flaming on the mailing lists, much as some people seem to enjoy it (500 messages on the subject and still counting!) wasn't the answer. The developers have stated very clearly that they're not going to change the model back. So is there another solution?

Yes -- a very simple solution, in fact. Write a plug-in that saves or exports the current image back to its current file name, then marks it as clean so GIMP won't warn about it when you quit.

It turned out to be extremely easy to write, and you can get it here: GIMP: Save/export clean plug-in. If it suits your GIMP workflow, you can even bind it to Ctrl-S ... or any other key you like.

Warning: I deliberately did not add any "Are you sure you want to overwrite?" confirmation dialogs. This plug-in will overwrite your current file, without asking for permission. After all, that's its job. So be aware of that.

How it's written

Here are some details about how it works. Non software geeks can skip the rest of this article.

When I first looked into writing this, I was amazed at how simple it was: really just two lines of Python (plus the usual plug-in registration boilerplate).

    pdb.gimp_file_save(img, drawable, img.filename, img.filename)
    pdb.gimp_image_clean_all(img)

The first line saves the image back to its current filename. (The gimp-file-save PDB call still handles all types, not just XCF.) The second line marks the image as clean.

Both of those are PDB calls, which means that people who don't have GIMP Python could write script-fu to do this part.

So why didn't I use script-fu? Because I quickly found that if I bound the plug-in to Ctrl-S, I'd want to use it for new images -- images that don't have a filename yet. And for that, you need to pop up some sort of "Save as" dialog -- something Python can do easily, and Script-fu can't do at all.

A Save-as dialog with smart directory default

I couldn't use the standard GIMP save-as dialog: as far as I can tell, there's no way to call that dialog from a plug-in. But it turned out the GTK save-as dialog has no default directory to start in: you have to set the starting directory every time. So I needed a reasonable initial directory.

I didn't want to come up with some desktop twaddle like ~/My Pictures or whatever -- is there really anyone that model fits? Certainly not me. I debated maintaining a preference you could set, or saving the last used directory as a preference, but that complicates things and I wasn't sure it's really that helpful for most people anyway.

So I thought about where I usually want to save images in a GIMP session. Usually, I want to save them to the same directory where I've been saving other images in the same session, right?

I can figure that out by looping through all currently open images with for img in gimp.image_list() : and checking os.path.dirname(img.filename) for each one. Keep track of how many times each directory is being used; whichever is used the most times is probably where the user wants to store the next image.

Keeping count in Python

Looping through is easy, but what's the cleanest, most Pythonic way of maintaining the count for each directory and finding the most popular one? Naturally, Python has a class for that, collections.Counter.

Once I've counted everything, I can ask for the most common path. The code looks a bit complicated because most_common(1) returns a one-item list of a tuple of the single most common path and the number of times it's been used -- for instance, [ ('/home/akkana/public_html/images/birds', 5) ]. So the path is the first element of the first element, or most_common(1)[0][0]. Put it together:

    counts = collections.Counter()
    for img in gimp.image_list() :
        if img.filename :
            counts[os.path.dirname(img.filename)] += 1
    try :
        return counts.most_common(1)[0][0]
    except :
        return None

So that's the only tricky part of this plug-in. The rest is straightforward, and you can read the code on GitHub: save-export-clean.py.

Tags: gimp, programming, python
[ 11:26 Aug 21, 2012    More gimp | permalink to this entry | comments ]

Driving motors, CHEAPLY, with an Arduino

Some time ago, I wrote about my explorations into the options for driving motors from an Arduino.

Motor shields worked well -- but they cost around $50 each, more than the Arduino itself. That's fine for a single one, but I'm teaching an Arduino workshop (this Thursday!) for high school girls, and I needed something I could buy more cheaply so I could get more of them.

(Incidentally, if any women in the Bay Area want to help with the workshop this Thursday, June 28 2012, I could definitely use a few more helpers! Please drop me an email.)

What I found on the web and on the Arduino IRC channel was immensely confusing to someone who isn't an electronics expert -- most people recommended things like building custom H-bridge circuits out of zener diodes.

But it's not that complicated after all. I got the help I needed from ITP Physical Computing's DC Motor Control Using an H-Bridge. It turns out you can buy a chip called an SN754410 that implements an H-bridge circuit -- including routing a power source to the motors while keeping the Arduino's power supply isolated -- for under $2. I ordered my SN754410 chips from Jameco and they arrived the next day.

(Technically, the SN754410 is a "quad half-bridge" rather than an "dual h-bridge". In practice I'm not sure of the difference. There's another chip, the L298, which is a full h-bridge and is also cheap to buy -- but it's a bit harder to use because the pins are wonky and it doesn't plug in directly to a breadboard unless you bend the pins around. I decided to stick with the SN754410; but the L298 might be better for high-powered motors.)

Now, the SN754410 isn't as simple to use as a motor shield. It has a lot of wires -- for two motors, you'll need six Arduino output pins, plus a 5v reference and ground, the four wires to the two motors, and the two wires to the motor power supply. Here's the picture of the wiring, made with Fritzing.

With all those wires, I didn't want to make the girls wire them up themselves -- it's way too easy to make a mistake and connect the wrong pin (as I found when doing my own testing). So I've wired up several of them on mini-breadboards so they'll be more or less ready to use. They look like little white mutant spiders with all the wires going everywhere.

A simple library for half-bridge motor control

The programming for the SN754410 is a bit less simple than motor shields as well. For each motor, you need an enable pin on the Arduino -- the analog signal that controls the motor's speed, so it needs to be one of the Arduino's PWM output pins, 9, 10 or 11 -- plus two logic pins, which can be any of the digital output pins.

To spin the motor in one direction, set the first logic pin high and the second low; to spin in the other direction, reverse the pins, with the first one low and the second one high. That's simple enough to program -- but I didn't look forward to trying to explain it to a group of high school students with basically no programming experience.

To make it simpler for them, I wrote a drop-in library that simplifies the code quite a bit. It defines a Motor object that you can initialize with the pins you'll be using -- the enable pin first, then the two logic pins. Initialize them in setup() like this:

#include 

Motor motors[2] = { Motor(9, 2, 3), Motor(10, 4, 5) };

void setup()
{
    motors[0].init();
    motors[1].init();
}

Then from your loop() function, you can make calls like this:

    motors[0].setSpeed(128);
    motors[1].setSpeed(-85);

Setting a negative speed will tell the library to reverse the logic pins so the motor spins the opposite direction.

I hope this will make motors easier to deal with for the girls who choose to try them. (We'll be giving them a choice of projects, so some of them may prefer to make light shows with LEDs, or music with piezo buzzers.)

You can get the code for the HalfBridge library, and a sample sketch that uses it, at my Arduino github repository

Cheap and easy motor control -- and I have a fleet of toy cars to connect to them. I hope this ends up being a fun workshop!

Tags: arduino, hardware, education, programming
[ 21:32 Jun 25, 2012    More hardware | permalink to this entry | comments ]

Viewing and modifying epub ebook tags

My epub Books folder is starting to look like my physical bookshelf at home -- huge and overflowing with books I hope to read some day. Mostly free books from the wonderful Project Gutenberg and DRM-free books from publishers and authors who support that model.

With the Nook's standard library viewer that's impossible to manage. All you can do is sort all those books alphabetically by title or author and laboriously page through, some five books to a page, hoping the one you want will catch your eye. Worse, sometimes books show up in the author view but don't show up in the title view, or vice versa. I guess Barnes & Noble think nobody keeps more than ten or so books on their shelves.

Fortunately on my rooted Nook I have the option of using better readers, like FBreader and Aldiko, that let me sort by tags. If I want to read something about the Civil War, or Astronomy, or just relax with some Science Fiction, I can browse by keyword.

Well, in theory. In practice, tagging of ebooks is inconsistent and not very useful.

For instance, the Gutenberg tags for Othello are:

• Tragedies
• Othello (Fictitious character) -- Drama
• Jealousy -- Drama
• Interracial marriage -- Drama
• Venice (Italy) -- Drama
• Muslims -- Drama

while the tags for Vanity Fair are

• Satire
• England -- Fiction
• Married women -- Fiction
• Female friendship -- Fiction
• Social classes -- Fiction
• British -- Europe -- Fiction
• Waterloo, Battle of, Waterloo, Belgium, 1815 -- Fiction

The Prince and the Pauper's tag list looks like:

• Edward VI, King of England, 1537-1553 -- Fiction
• Impostors and imposture -- Fiction
• Social classes -- Fiction
• Poor children -- Fiction
• Lookalikes -- Fiction
• Princes -- Fiction
• Boys -- Fiction
• London (England) -- Fiction
• Historical fiction

while Captains Courageous looks like

• Sea stories
• Saltwater fishing -- Fiction
• Children of the rich -- Fiction
• Bildungsromans
• Fishing boats -- Fiction
• Teenage boys -- Fiction
• Rescues -- Fiction
• Fishers -- Fiction
• Grand Banks of Newfoundland -- Fiction

I can understand wanting to tag details like this, but few of those tags are helpful when I'm browsing books on my little handheld device. I can't imagine sitting down to read and thinking, "Let's see, what books do I have on Interracial marriage? Or Saltwater fishing? No, on second thought I'd rather read some fiction set in the time of Edward VI, King of England, 1537-1553."

And of course, with over 90 books loaded on my ebook readers, it means I have hundreds of entries in my tags list, with few of them including more than one book.

Clearly what I needed to do was to change the tags on my ebooks.

Viewing and modifying epub tags

That ought to be simple, right? But ebooks are still a very young technology, and there's surprisingly little software devoted to them. Calibre can probably do it if you don't mind maintaining your whole book collection under calibre; but I like to be able to work on files one at a time or in small groups. And I couldn't find a program that would let me do that.

What to do? Well, epub is a fairly simple XML format, right? So modifying it with Python shouldn't that hard.

Managing epub in Python

An epub file is a collection of XML files packaged in a zip archive. So I unzipped one of my epub books and poked around. I found the tags in a file called content.opf, inside a <metadata> tag. They look like this:

<dc:subject>Science fiction</dc:subject>

So I could use Python's zipfile module to access the content.opf file inside the zip archive, then use the xml.dom.minidom parser to get to the tags. Writing a script to display existing tags was very easy.

What about replacing the old, unweildy tag list with new, simple tags?

It's easy enough to add nodes in Python's minidom. So the trick is writing it back to the epub file. The zipfile module doesn't have a way to modify a zip file in place, so I created a new zip archive and copied files from the old archive to the new one, replacing content.opf with a new version.

Python's difficulty with character sets in XML

But I hit a snag in writing the new content.opf. Python's XML classes have a toprettyxml() method to write the contents of a DOM tree. Seemed simple, and that worked for several ebooks ... until I hit one that contained a non-ASCII character. Then Python threw a UnicodeEncodeError: 'ascii' codec can't encode character u'\u2014' in position 606: ordinal not in range(128).

Of course, there are ways (lots of them) to encode that output string -- I could do

ozf.writestr(info, dom.toprettyxml().encode(encoding, 'xmlcharrefreplace'))

, or

writestr(info, dom.toprettyxml(encoding=encoding)

Except ... what should I pass as the encoding? The content.opf file started with its encoding:
<?xml version='1.0' encoding='UTF-8'?>
but Python's minidom offers no way to get that information. In fact, none of Python's XML parsers seem to offer this.

Since you need a charset to avoid the UnicodeEncodeError, the only options are (1) always use a fixed charset, like utf-8, for content.opf, or (2) open content.opf and parse the charset line by hand after Python has already parsed the rest of the file. Yuck! So I chose the first option ... I can always revisit that if the utf-8 in content.opf ever causes problems.

The final script

Charset difficulties aside, though, I'm quite pleased with my epubtags.py script. It's very handy to be able to print tags on any .epub file, and after cleaning up the tags on my ebooks, it's great to be able to browse by category in FBreader. Here's the program: epubtag.py.

Tags: ebook, programming, python
[ 12:05 Jun 09, 2012    More programming | permalink to this entry | comments ]

Creating packages for a Launchpad PPA

In a previous article I wrote about how to use stdeb to turn a Python script into source and binary Debian/Ubuntu packages.

You can distribute a .deb file that people can download and install; but it's a lot easier for people to install if you set up a repository, so they can get automatic updates from you. If you're targeting Ubuntu, the best way to do that is to set up a Launchpad Personal Package Archive, or PPA.

Create your PPA

First, create your PPA. If you don't have a Launchpad account yet, create one, add a GPG key, and sign the Code of Conduct. Then log in to your account and click on Create a new PPA.

You'll have to pick a name and a display name for your PPA. The default is "ppa", and many people leave personal PPAs as that. You might want to give it a display name of yourname-ppa or something similar if it's for a collection of stuff; or you're only going to use it for software related to one program or package, name it accordingly.

Ubuntu requires nonstandard paths

When you're creating your package with stdeb, if you're ultimately targeting a PPA, you'll only need the souce dsc package, not the binary deb. But as you'll see, you'll need to rebuild it to make Launchpad happy.

If you're intending to go through the Developer.ubuntu.com process, there are specific requirements for version numbering and tarball naming -- see "Packaging" in the App Review Board Guidelines. Your app will also need to install unusual locations -- in particular, any files it installs, including the script itself, need to be in /opt/extras.ubuntu.com/<packagename> instead of a more standard location.

How the user is supposed to run these apps (run a script to add each of /opt/extras.ubuntu.com/* to your path?) is not clear to me; I'm not sure this app review thing has been fully thought out. In any case, you may need to massage your setup.py accordingly, and keep a separate version around for when you're creating the Ubuntu version of your app.

There are also apparently some problems loading translation files for an app in /opt/extras.ubuntu.com which may require some changes to your Python code.

Prepare and sign your package

Okay, now comes the silly part. You know that source .dsc package you just made? Now you have to unpack it and "build" it before you can upload it. That's partly because you have to sign it with your GPG key -- stdeb apparently can't do the signing step.

Normally, you'd sign a package with debsign deb_dist/packagename_version.changes (then type your GPG passphrase when prompted). Unfortunately, that sort of signing doesn't work here. If you used stdeb's bdist_deb to generate both binary and source packages, the .changes file it generates will contain both source and binary and Launchpad will reject it. If you used sdist_dsc to generate only the source package, then you don't have a .changes file to sign and submit to Launchpad. So here's how you can make a signed, source-only .changes file Launchpad will accept.

Since this will extract all your files again, I suggest doing this in a temporary directory to make it easier to clean up afterward:

$ mkdir tmp
$ cd tmp
$ dpkg-source -x ../deb_dist/packagename_version.dsc
$ cd packagename_version

Now is a good time to take a look at the deb_dist/packagename_version/debian/changelog that stdeb created, and make sure it got the right version and OS codename for the Ubuntu release you're targeting -- oneiric, precise, quantal or whatever. stdeb's default is "unstable" (Debian) so you'll probably need to change it. You can cross-check this information in the deb_dist/packagename_version.changes file, which is the file you'll actually be uploading to the PPA.

Finally, build and sign your source package:

$ debuild -S -sa
  [type your GPG passphrase when prompted, twice]
$ dput ppa:yourppa ../packagename_version_source.changes

Upload the package

Finally, it's time to upload the package:

$ dput ppa:your-ppa-name deb_dist/packagename_version.changes

This will give you some output and eventually probably tell you Successfully uploaded packages. It's lying -- it may have failed. Watch your inbox for messages. If Launchpad rejects your changes, you should get an email fairly quickly.

If Launchpad accepts the changes, you'll get an Accepted email. Great! But don't celebrate quite yet. Launchpad still has to build your package before it can be installed. If you try to add your PPA now, you'll get a 404.

Wait for Launchpad to build

You might as well add your repository now so you can install from it once it's ready:

$ sudo add-apt-repository ppa:your-ppa-name

But don't apt-get update yet! if you try that too soon, you'll get a 404, or an Ign meaning that the repository exists but there are no packages in it for your architecture. It might be as long as a few hours before Launchpad builds your package.

To keep track of this, go to your Launchpad PPA page (something like https://launchpad.net/~yourname/+archive/ppa) and look under PPA Statistics for something like "1 package waiting to build". Click on that link, then in the page that comes up, click on the link like i386 build of pkgname version in ubuntu precise RELEASE. That should give you a time estimate.

Wondering why it's being built for i386 when Python should be arch independent? Worry not -- that's just the architecture that's doing the building. Once it's built, your package should install anywhere.

Once the Launchpad build page finally says the package is built, it's finally safe to run the usual apt-get update.

Add your key

But when you apt-get update you may get an error like this:

The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 16126D3A3E5C1192

Obviously you have your own public key, so what's up? You have to import the key from Ubuntu's keyserver, and then export it into apt-key, before apt can use it -- even if it's your own key.

For this, you need the last 8 digits given in the NO PUBKEY message. Take those 8 digits and run these two commands:

gpg --keyserver keyserver.ubuntu.com --recv 3E5C1192
gpg --export --armor 3E5C1192 | sudo apt-key add -

I'm told that apt-add-repository is supposed to add the key automatically, but it didn't for me. Maybe it will if you wait until after your package is built before calling apt-add-repository.

Now if you apt-get update, you should see no errors. Finally, you can apt-get install pkgname. Congratulations! You have a working PPA package.

Tags: ubuntu, linux, programming
[ 12:34 May 30, 2012    More programming | permalink to this entry | comments ]

Use stdeb to make Debian packages for a Python package

I write a lot of little Python scripts. And I use Ubuntu and Debian. So why aren't any of my scripts packaged for those distros?

Because Debian packaging is absurdly hard, and there's very little documentation on how to do it. In particular, there's no help on how to take something small, like a Python script, and turn it into a package someone else could install on a Debian system. It's pretty crazy, since RPM packaging of Python scripts is so easy.

Recently at the Ubuntu Developers' Summit, Asheesh of OpenHatch pointed me toward a Python package called stdeb that simplifies a lot of the steps and makes Python packaging fairly straightforward.

You'll need a setup.py file to describe your Python script, and you'll probably want a .desktop file and an icon. If you haven't done that before, see my article on Packaging Python for MeeGo for some hints.

Then install python-stdeb. The package has some requirements that aren't listed as dependencies, so you'll need to install:

apt-get install python-stdeb fakeroot python-all

(I have no idea why it needs python-all, which installs only a directory /usr/share/doc/python-all with some policy documentation files, but if you don't install it, stdeb will fail later.)

Now create a config file for stdeb to tell it what Debian/Ubuntu version you're going to be targeting, if it's anything other than Debian unstable (stdeb's default). Unfortunately, there seems to be no way to pass this on the command line rather than in a config file. So if you want to make packages for several distros, you'll have to edit the config file for every distro you want to support. Here's what I'm using for Ubuntu 12.04 Precise Pangolin:

[DEFAULT]
Suite: precise

Now you're ready to run stdeb. I know of two ways to run it. You can generate both source and binary packages, like this:

python setup.py --command-packages=stdeb.command bdist_deb

Or you can generate source packages only, like this:

python setup.py --command-packages=stdeb.command sdist_dsc

Either syntax creates a directory called deb_dist. It contains a lot of files including a source .dsc, several tarballs, a copy of your source directory, and (if you used bdist_deb) a binary .deb package.

If you used the bdist_deb form, don't be put off that it concludes with a message:

dpkg-buildpackage: binary only upload (no source included)

It's fibbing: the source .dsc is there as well as the binary .deb. I presume it prints the warning because it creates them as separate steps, and the binary is the last step.

Now you can use dpkg -i to install your binary deb, or you can use the source dsc for various purposes, like creating a repository or a Launchpad PPA. But those involve a lot more steps -- so I'll cover that in a separate article about creating PPAs.

Update: you can find that article here: Creating packages for a Launchpad PPA.

Tags: debian, ubuntu, linux, programming, python
[ 10:44 May 26, 2012    More programming | permalink to this entry | comments ]

Playing an MP3 file from an Android app

I've mostly been enormously happy with my upgrade from my old Archos 5 to the Samsung Galaxy Player 5.0. The Galaxy does everything I always wanted the Archos to do, all those things the Archos should have done but couldn't because of its buggy and unsupported Android 1.6.

That is, I've been happy with everything except one thing: my birdsong app no longer worked.

I have a little HTML app based on my "tweet" python script which lets you choose from a list of birdsong MP3 files. (The actual MP3 files are ripped from the excellent 4-CD Stokes Field Guide to Western Bird Songs set.) The HTML app matches bird names as you type in characters. (If you're curious, an earlier test version is at tweet.html.)

On the Archos, I ran that under my WebClient Android app (I had to modify the HTML to add a keyboard, since in Android 1.6 the soft keyboard doesn't work in WebView text fields). I chose a bird, and WebView passed off the MP3 file to the Archos' built-in audio player. Worked great.

On the Samsung Galaxy, no such luck. Apparently Samsung's built-in media player can only play files it has indexed itself. If you try to use it to play an arbitrary file, say, "Song_Sparrow.mp3", it will say: unknown file type. No matter that the file ends in .mp3 ... and no matter that I've called intent.setDataAndType(Uri.parse(url), "audio/mpeg"); ... and no matter that the file is sitting on the SD cad and has in fact been indexed already by the media player. You didn't navigate to it via the media player's UI, so it refuses to play it.

I haven't been able to come up with an answer to how to make Samsung's media player more flexible, and I was just starting a search for alternate Android MP3 player apps, when I ran across Play mp3 in SD Card, using Android's MediaPlayer and Error creating MediaPlayer with Uri or file in assets which gave me the solution. Instead of using an intent and letting WebView call up a music application, you can use an Android MediaPlayer to play your file directly.

Here's what the code looks like, inside setWebViewClient() which is itself inside onCreate():

            @Override
            public boolean shouldOverrideUrlLoading(WebView view, String url) {
                if (url.endsWith(".mp3")) {
                    MediaPlayer mediaPlayer = new MediaPlayer();
                    try {
                        mediaPlayer.setDataSource(getApplicationContext(), Uri.parse(url));
                        mediaPlayer.prepare();
                        mediaPlayer.start();
                    }
                    catch (IllegalArgumentException e) { showMessage("Illegal argument exception on " + url); }
                    catch (IllegalStateException e) { showMessage("Illegal State exception on " + url); }
                    catch (IOException e) { showMessage("I/O exception on " + url); }
                }
            }

showMessage() is my little wrapper that pops up an error message dialog. Of course, you can handle other types, not just files ending in .mp3.

And now I can take the Galaxy out on a birdwalk and use it to help me identify bird songs.

Tags: android, programming, nature, birds
[ 13:28 May 06, 2012    More programming | permalink to this entry | comments ]

Venus is at its brightest -- why? And how to calculate it

Venus has been a beautiful sight in the evening sky for months, but at the end of April it's reaching a brightness peak, magnitude -4.7.

By then, if you look at it in a telescope or even good binoculars, you'll see it has waned to a crescent. That's a bit non-obvious: when the moon is a crescent, it's a lot fainter than a full moon. So why is Venus brightest in its crescent phase?

It has to do with their orbits. The moon is always about the same distance away, about 385,000 km or 239,000 miles (I've owned cars with more miles than that!), though it varies a little, from 362,600 km at perigee to 405,400 km at apogee.

When we look at the full moon, not only are we seeing the whole Earth-facing surface illuminated, but the central part of that light is reflecting straight up off the moon's surface. When we look at a crescent moon, we're seeing light that's near the moon's sunrise or sunset point -- dimmer and more spread out than the concentrated light of noon -- and in addition we're seeing less of it.

Venus, in contrast, varies its distance from us immensely. We can't see Venus when it's "full", because it's on the other side of the sun from us and lost in the sun's glow. It'll next be there a year from now, in April of 2013. But if we could see it when it's full, Venus would be a distant 1.7 AU from us. An AU is an Astronomical Unit, the average distance of the earth from the sun or about 89 million miles, so Venus when it's full is about 170 million miles away. Its disk is a tiny 9.9 arcseconds (an arcsecond is 1/3600 of a degree) -- about the size of Mars this month.

In contrast, when we look at the crescent Venus around the end of this month, although we're only seeing about 28% of its surface illuminated, and that only with glancing twilight rays, it's much closer to us -- less than half an AU, or about 45 million miles -- and its disk extends a huge 37 arcseconds, bigger than Jupiter this month.

Of course, eventually, as Venus pulls between us and the sun, its crescent gets so slim that even its huge size can't compensate. So its peak brightness happens when those two curves cross, when the disk is somewhere around 27% illuminated, as happens at the end of this month and the beginning of May.

Exactly when? Good question. The RASC Handbook says Venus' "greatest illuminated extent" is on April 30, but PyEphem and XEphem say Venus is actually brighter from May 3-8 ... and when it emerges from the sun's glare and moves into the morning sky in June, it'll be slightly brighter still, peaking at magnitude -4.8 in the first week of July.)

Tracking Venus with PyEphem

When I started my Shallow Sky column this month, I saw the notice of Venus's maximum brightness and greatest illuminated extent in the RASC Handbook. But I wanted more details -- how much did its distance and size really change, when would the brightness peak again as it emerged from the sun's glare, when would it next be "full"?

PyEphem made it easy to calculate all this. Just create an ephem.Venus() object, calculate its values for any date of interest, then print out parameters like phase, mag, earth_distance and size. In just a few minutes of programming, I had a nice table of Venus data.

import ephem

venus = ephem.Venus()

print '%10s   %6s %6s %6s %6s' % ('date', '%', 'mag', 'dist', 'size')
def print_venus(when) :
    venus.compute(when)
    fmt = '%02d-%02d-%02d   %6.2f %6.2f %6.2f %6.2f'
    trip = when.triple()
    print fmt % (trip[0], trip[1], trip[2],
                 venus.phase, venus.mag, venus.earth_distance, venus.size)

# Loop from the beginning of 2012 through the middle of 2013:
d = ephem.date('2012')
end_date = ephem.date('2013/6/1')
while d < end_date :
    print_venus(d)
    # Add a week:
    d = ephem.date(d + ephem.hour * 24)

I've found PyEphem very handy for calculations like this -- and it's great to be able to double-check listings in other publications.

Tags: astronomy, programming, python
[ 13:44 Apr 27, 2012    More science/astro | permalink to this entry | comments ]

Android WebView can't goBack from a page with iframes

I've been fighting a bug in Android's WebView class for ages: on some pages, clicking FeedViewer's back arrow (which calls WebView::goBack()) doesn't go back to the previous page. Instead, it jumps to some random position in the current page. If you repeat it, eventually, after five or so tries (depending on the page), eventually goBack() will finally work and you'll be back at the previous page.

It was especially frustrating in that it didn't happen everywhere -- only in pages from certain sites. I saw it all the time in pages from the Los Angeles Times and from Make Magazine, but only rarely on other sites.

But I finally tracked it down: it's because those pages include the HTML <iframe> tag. Apparently, if a WebView is on a page (at least if it's a local page) that contains N iframes, the first N calls to goBack will jump somewhere in the document -- probably the location of the associated iframe, though I haven't verified that -- and only on the N+1st call will the WebView actually go back to the previous page.

The oddest thing is, this doesn't seem to be reported anywhere. Android's bug tracker finds nothing for webview iframe goback, and web searching hasn't found a hint of it, even though I see this in Android versions from 1.6 through 2.3.5. How is it possible that other people haven't noticed this? I wonder if it only happens on local file:// URLs, and not many people use those.

In any case, I'm very happy to have found the answer at last. It was easy enough to modify FeedMe to omit iframes (and who wants iframes in simplified HTML anyway?), and it's great to have a Back button that works on any page.

Tags: android, programming
[ 19:56 Apr 21, 2012    More programming | permalink to this entry | comments ]

Quick Guide to Android ADB

I've been fiddling with several new Android devices, which means I have to teach myself how to use adb all over again.

adb is the Android Debug Bridge, and it's great for debugging. It lets you type commands on your desktop keyboard rather than tapping them into the soft keyboard in Android's terminal emulator, it gives you a fast way to install apps, and most important, it lets you get Java stack traces from crashing programs.

Alas, the documentation is incomplete and sometimes just plain wrong. Since I don't need adb very often, I always forget how to use it between sessions, and it takes some googling to figure out the tricks. Here are the commands I've found most helpful.

Start the server

First you have to start the adb, and that must be done as root. But adb isn't a system program and probably lives in some path like /home/username/path/to/android-sdk-linux_x86/tools. Even if you put it in your own path, it may not be in root's. You can probably run it with the explicit path:

$ sudo /path/to/android-sdk-linux_x86/tools/sudo adb start-server

or you can add it to root's path:

# export PATH=$PATH:/path/to/android/android-sdk-linux_x86/tools
# adb start-server

If you're also running eclipse, that probably won't work the first time, because eclipse may also have started an adb server (that gets in the way when you try to run adb manually). if you don't see "* daemon started successfully *", try killing the server and restarting it:

# adb kill-server
# adb start-server
* daemon not running. starting it now on port 5037 *
* daemon started successfully *

Keep trying until you see that "* daemon started successfully *" message.

Connecting

$ adb usb

Occasionally, this will give "error: closed". Don't panic -- sometimes this actually means "I noticed something connected on USB and automatically connected to it, so no need to connect again." It's mysterious, and no one seems to have an explanation for what's really happening. Anyway, try running some adb commands and you may find you're actually connected.

Shell and install

The most fun is running an interactive shell on your Android device.

$ adb shell

It's just busybox, not a full shell, so you don't have nice things like tab completion. But it's still awfully useful.

You can also install apps. On some devices (like the Nook, where I haven't found a way to allow install from non-market sources), it's the only way to install an apk file.

$ adb install /path/to/appname.apk

If the app is already installed, you'll get an error. Theoretically you can also do adb uninstall first, but when I tried that it just printed "Failure". But you can use -r for "reinstall":

$ adb install -r /path/to/appname.apk

There is no mention of either uninstall or -r in the online adb documentation, though adb -h mentions it.

Update: To uninstall, you need the full name of the package. To get the names of installed packages (another undocumented command), do this: adb shell pm list packages

Debug crashes with logcat

Finally, for debugging crashes, you can start up a logcat and see system messages, including stack traces from crashing apps:

$ adb logcat

Logcat is great for fixing reproducible crashes. Sadly, it's not so good for random/sporadic crashes that happen during the course of using the device.

You're supposed to be able to do adb logcat -s AppName if you're only interested in debugging messages from one app, but that doesn't work for me -- I get no output even when the app runs and crashes.

Tags: android, programming
[ 10:32 Apr 09, 2012    More tech | permalink to this entry | comments ]

Image manipulation in Python

Someone asked me about determining whether an image was "portrait" or "landscape" mode from a script.

I've long had a script for automatically rescaling and rotating images, using ImageMagick under the hood and adjusting automatically for aspect ratio. But the scripts are kind of a mess -- I've been using them for over a decade, and they started life as a csh script back in the pnmscale days, gradually added ImageMagick and jpegtran support and eventually got translated to (not very good) Python.

I've had it in the back of my head that I should rewrite this stuff in cleaner Python using the ImageMagick bindings, rather than calling its commandline tools. So the question today spurred me to look into that. I found that ImageMagick isn't the way to go, but PIL would be a fine solution for most of what I need.

ImageMagick: undocumented and inconstant

Ubuntu has a python-pythonmagick package, which I installed. Unfortunately, it has no documentation, and there seems to be no web documentation either. If you search for it, you find a few other people asking where the documentation is.

Using things like help(PythonMagick) and help(PythonMagick.Image), you can ferret out a few details, like how to get an image's size:

import PythonMagick
filename = 'img001.jpg'
img = PythonMagick.Image(filename)
size = img.size()
print filename, "is", size.width(), "x", size.height()

Great. Now what if you want to rescale it to some other size? Web searching found examples of that, but it doesn't work, as illustrated here:

>>> img.scale('1024x768')
>>> img.size().height()
640

The built-in help was no help:

>>> help(img.scale)
Help on method scale:

scale(...) method of PythonMagick.Image instance
    scale( (Image)arg1, (Geometry)arg2) -> None :
    
        C++ signature :
            void scale(Magick::Image {lvalue},Magick::Geometry)

So what does it want for (Geometry)? Strings don't seem to work, 2-tuples don't work, and there's no Geometry object in PythonMagick. By this time I was tired of guesswork. Can the Python Imaging Library do better?

PIL -- the Python Imaging Library

PIL, happily, does have documentation. So it was easy to figure out how to get an image's size:

from PIL import Image
im = Image.open(filename)
w = im.size[0]
h = im.size[1]
print filename, "is", w, "x", h

It was equally easy to scale it to half its original size, then write it to a file:

newim = im.resize((w/2, h/2))
newim.save("small-" + filename)

Reading EXIF

Wow, that's great! How about EXIF -- can you read that? Yes, PIL has a module for that too:

import PIL.ExifTags

exif = im._getexif()
for tag, value in exif.items():
    decoded = PIL.ExifTags.TAGS.get(tag, tag)
    print decoded, '->', value

There are other ways to read exif -- pyexiv2 seems highly regarded. It has documentation, a tutorial, and apparently it can even write EXIF tags.

If neither PIL nor pyexiv2 meets your needs, here's a Stack Overflow thread on other Python EXIF solutions, and here's another discussion of Python EXIF. But since you probably already have PIL, it's certainly an easy way to get started.

What about the query that started all this: how to find out whether an image is portrait or landscape? Well, the most important thing is the image dimensions themselves -- whether img.size[0] > img.size[1]. But sometimes you want to know what the camera's orientation sensor thought. For that, you can use this code snippet:

for tag, value in exif.items():
    decoded = PIL.ExifTags.TAGS.get(tag, tag)
    if decoded == 'Orientation':
        print decoded, ":", value

Then compare the number you get to this Exif Orientation table. Normal landscape-mode photos will be 1.

Given all this, have I actually rewritten resizeall and rotateall using PIL? Why, no! I'll put it on my to-do list, honest. But since the scripts are actually working fine (just don't look at the code), I'll leave them be for now.

Tags: programming, python, imaging, imagemagick,
[ 14:33 Mar 16, 2012    More programming | permalink to this entry | comments ]

HTML and Javascript Presentations

When I give talks that need slides, I've been using my Slide Presentations in HTML and JavaScript for many years. I uploaded it in 2007 -- then left it there, without many updates.

But meanwhile, I've been giving lots of presentations, tweaking the code, tweaking the CSS to make it display better. And every now and then I get reminded that a few other people besides me are using this stuff.

For instance, around a year ago, I gave a talk where nearly all the slides were just images. Silly to have to make a separate HTML file to go with each image. Why not just have one file, img.html, that can show different images? So I wrote some code that lets you go to a URL like img.html?pix/whizzyphoto.jpg, and it will display it properly, and the Next and Previous slide links will still work.

Of course, I tweak this software mainly when I have a talk coming up. I've been working lately on my SCALE talk, coming up on January 22: Fun with Linux and Devices (be ready for some fun Arduino demos!) Sometimes when I overload on talk preparation, I procrastinate by hacking the software instead of the content of the actual talk. So I've added some nice changes just in the past few weeks.

For instance, the speaker notes that remind me of where I am in the talk and what's coming next. I didn't have any way to add notes on image slides. But I need them on those slides, too -- so I added that.

Then I decided it was silly not to have some sort of automatic reminder of what the next slide was. Why should I have to put it in the speaker notes by hand? So that went in too.

And now I've done the less fun part -- collecting it all together and documenting the new additions. So if you're using my HTML/JS slide kit -- or if you think you might be interested in something like that as an alternative to Powerpoint or Libre Office Presenter -- check out the presentation I have explaining the package, including the new features.

You can find it here: Slide Presentations in HTML and JavaScript

Tags: speaking, javascript, html, web, programming, tech
[ 20:08 Jan 12, 2012    More speaking | permalink to this entry | comments ]

Parsing HTML in Python

I've been having (mis)adventures learning about Python's various options for parsing HTML.

Up until now, I've avoided doing any HTMl parsing in my RSS reader FeedMe. I use regular expressions to find the places where content starts and ends, and to screen out content like advertising, and to rewrite links. Using regexps on HTML is generally considered to be a no-no, but it didn't seem worth parsing the whole document just for those modest goals.

But I've long wanted to add support for downloading images, so you could view the downloaded pages with their embedded images if you so chose. That means not only identifying img tags and extracting their src attributes, but also rewriting the img tag afterward to point to the locally stored image. It was time to learn how to parse HTML.

Since I'm forever seeing people flamed on the #python IRC channel for using regexps on HTML, I figured real HTML parsing must be straightforward. A quick web search led me to Python's built-in HTMLParser class. It comes with a nice example for how to use it: define a class that inherits from HTMLParser, then define some functions it can call for things like handle_starttag and handle_endtag; then call self.feed(). Something like this:

from HTMLParser import HTMLParser

class MyFancyHTMLParser(HTMLParser):
  def fetch_url(self, url) :
    request = urllib2.Request(url)
    response = urllib2.urlopen(request)
    link = response.geturl()
    html = response.read()
    response.close()
    self.feed(html)   # feed() starts the HTMLParser parsing

  def handle_starttag(self, tag, attrs):
    if tag == 'img' :
      # attrs is a list of tuples, (attribute, value)
      srcindex = self.has_attr('src', attrs)
      if srcindex < 0 :
        return   # img with no src tag? skip it
      src = attrs[srcindex][1]
      # Make relative URLs absolute
      src = self.make_absolute(src)
      attrs[srcindex] = (attrs[srcindex][0], src)

    print '<' + tag
    for attr in attrs :
      print ' ' + attr[0]
      if len(attr) > 1 and type(attr[1]) == 'str' :
        # make sure attr[1] doesn't have any embedded double-quotes
        val = attr[1].replace('"', '\"')
        print '="' + val + '"')
    print '>'

  def handle_endtag(self, tag):
    self.outfile.write('</' + tag.encode(self.encoding) + '>\n')

Easy, right? Of course there are a lot more details, but the basics are simple.

I coded it up and it didn't take long to get it downloading images and changing img tags to point to them. Woohoo! Whee!

The bad news about HTMLParser

Except ... after using it a few days, I was hitting some weird errors. In particular, this one:
HTMLParser.HTMLParseError: bad end tag: ''

It comes from sites that have illegal content. For instance, stories on Slate.com include Javascript lines like this one inside <script></script> tags:
document.write("<script type='text/javascript' src='whatever'></scr" + "ipt>");

This is technically illegal html -- but lots of sites do it, so protesting that it's technically illegal doesn't help if you're trying to read a real-world site.

Some discussions said setting self.CDATA_CONTENT_ELEMENTS = () would help, but it didn't.

HTMLParser's code is in Python, not C. So I took a look at where the errors are generated, thinking maybe I could override them. It was easy enough to redefine parse_endtag() to make it not throw an error (I had to duplicate some internal strings too). But then I hit another error, so I redefined unknown_decl() and _scan_name(). And then I hit another error. I'm sure you see where this was going. Pretty soon I had over 100 lines of duplicated code, and I was still getting errors and needed to redefine even more functions. This clearly wasn't the way to go.

Using lxml.html

I'd been trying to avoid adding dependencies to additional Python packages, but if you want to parse real-world HTML, you have to. There are two main options: Beautiful Soup and lxml.html. Beautiful Soup is popular for large projects, but the consensus seems to be that lxml.html is more error-tolerant and lighter weight.

Indeed, lxml.html is much more forgiving. You can't handle start and end tags as they pass through, like you can with HTMLParser. Instead you parse the HTML into an in-memory tree, like this:

  tree = lxml.html.fromstring(html)

How do you iterate over the tree? lxml.html is a good parser, but it has rather poor documentation, so it took some struggling to figure out what was inside the tree and how to iterate over it.

You can visit every element in the tree with

for e in tree.iter() :
  print e.tag

But that's not terribly useful if you need to know which tags are inside which other tags. Instead, define a function that iterates over the top level elements and calls itself recursively on each child.

The top of the tree itself is an element -- typically the <html></html> -- and each element has .tag and .attrib. If it contains text inside it (like a <p> tag), it also has .text. So to make something that works similarly to HTMLParser:

def crawl_tree(tree) :
  handle_starttag(tree.tag, tree.attrib)
  if tree.text :
    handle_data(tree.text)
  for node in tree :
    crawl_tree(node)
  handle_endtag(tree.tag)

But wait -- we're not quite all there. You need to handle two undocumented cases.

First, comment tags are special: their tag attribute, instead of being a string, is <built-in function Comment> so you have to handle that specially and not assume that tag is text that you can print or test against.

Second, what about cases like <p>Here is some <i>italicised</i> text.</p> ? in this case, you have the p tag, and its text is "Here is some ". Then the p has a child, the i tag, with text of "italicised". But what about the rest of the string, " text."?

That's called a tail -- and it's the tail of the adjacent i tag it follows, not the parent p tag that contains it. Confusing!

So our function becomes:

def crawl_tree(tree) :
  if type(tree.tag) is str :
    handle_starttag(tree.tag, tree.attrib)
    if tree.text :
      handle_data(tree.text)
    for node in tree :
      crawl_tree(node)
    handle_endtag(tree.tag)
  if tree.tail :
    handle_data(tree.tail)

See how it works? If it's a comment (tree.tag isn't a string), we'll skip everything -- except the tail. Even a comment might have a tail:
<p>Here is some <!-- this is a comment --> text we want to show.</p>
so even if we're skipping comment we need its tail.

I'm sure I'll find other gotchas I've missed, so I'm not releasing this version of feedme until it's had a lot more testing. But it looks like lxml.html is a reliable way to parse real-world pages. It even has a lot of convenience functions like link rewriting that you can use without iterating the tree at all. Definitely worth a look!

Tags: python, programming, html
[ 14:04 Jan 08, 2012    More programming | permalink to this entry | comments ]

Plotting the Analemma

My SJAA planet-observing column for January is about the Analemma and the Equation of Time.

The analemma is that funny figure-eight you see on world globes in the middle of the Pacific Ocean. Its shape is the shape traced out by the sun in the sky, if you mark its position at precisely the same time of day over the course of an entire year.

The analemma has two components: the vertical component represents the sun's declination, how far north or south it is in our sky. The horizontal component represents the equation of time.

The equation of time describes how the sun moves relatively faster or slower at different times of year. It, too, has two components: it's the sum of two sine waves, one representing how the earth speeds up and slows down as it moves in its elliptical orbit, the other a function the tilt (or "obliquity") of the earth's axis compared to its orbital plane, the ecliptic.

The Wikipedia page for Equation of time includes a link to a lovely piece of R code by Thomas Steiner showing how the two components relate. It's labeled in German, but since the source is included, I was able to add English labels and use it for my article.

But if you look at photos of real analemmas in the sky, they're always tilted. Shouldn't they be vertical? Why are they tilted, and how does the tilt vary with location? To find out, I wanted a program to calculate the analemma.

Calculating analemmas in PyEphem

The very useful astronomy Python package PyEphem makes it easy to calculate the position of any astronomical object for a specific location. Install it with: easy_install pyephem for Python 2, or easy_install ephem for Python 3.

import ephem
observer = ephem.city('San Francisco')
sun = ephem.Sun()
sun.compute(observer)
print sun.alt, sun.az

The alt and az are the altitude and azimuth of the sun right now. They're printed as strings: 25:23:16.6 203:49:35.6 but they're actually type 'ephem.Angle', so float(sun.alt) will give you a number in radians that you can use for calculations.

Of course, you can specify any location, not just major cities. PyEphem doesn't know San Jose, so here's the approximate location of Houge Park where the San Jose Astronomical Association meets:

observer = ephem.Observer()
observer.name = "San Jose"
observer.lon = '-121:56.8'
observer.lat = '37:15.55'

You can also specify elevation, barometric pressure and other parameters.

So here's a simple analemma, calculating the sun's position at noon on the 15th of each month of 2011:

    for m in range(1, 13) :
        observer.date('2011/%d/15 12:00' % (m))
        sun.compute(observer)

I used a simple PyGTK window to plot sun.az and sun.alt, so once it was initialized, I drew the points like this:

    # Y scale is 45 degrees (PI/2), horizon to halfway to zenith:
    y = int(self.height - float(self.sun.alt) * self.height / math.pi)
    # So make X scale 45 degrees too, centered around due south.
    # Want az = PI to come out at x = width/2.
    x = int(float(self.sun.az) * self.width / math.pi / 2)
    # print self.sun.az, float(self.sun.az), float(self.sun.alt), x, y
    self.drawing_area.window.draw_arc(self.xgc, True, x, y, 4, 4, 0, 23040)

So now you just need to calculate the sun's position at the same time of day but different dates spread throughout the year.

And my 12-noon analemma came out almost vertical! Maybe the tilt I saw in analemma photos was just a function of taking the photo early in the morning or late in the afternoon? To find out, I calculated the analemma for 7:30am and 4:30pm, and sure enough, those were tilted.

But wait -- notice my noon analemma was almost vertical -- but it wasn't exactly vertical. Why was it skewed at all?

Time is always a problem

As always with astronomy programs, time zones turned out to be the hardest part of the project. I tried to add other locations to my program and immediately ran into a problem.

The ephem.Date class always uses UTC, and has no concept of converting to the observer's timezone. You can convert to the timezone of the person running the program with localtime, but that's not useful when you're trying to plot an analemma at local noon.

At first, I was only calculating analemmas for my own location. So I set time to '20:00', that being the UTC for my local noon. And I got the image at right. It's an analemma, all right, and it's almost vertical. Almost ... but not quite. What was up?

Well, I was calculating for 12 noon clock time -- but clock time isn't the same as mean solar time unless you're right in the middle of your time zone.

You can calculate what your real localtime is (regardless of what politicians say your time zone should be) by using your longitude rather than your official time zone:

    date = '2011/%d/12 12:00' % (m)
    adjtime = ephem.date(ephem.date(date) \
                    - float(self.observer.lon) * 12 / math.pi * ephem.hour)
    observer.date = adjtime

Maybe that needs a little explaining. I take the initial time string, like '2011/12/15 12:00', and convert it to an ephem.date. The number of hours I want to adjust is my longitude (in radians) times 12 divided by pi -- that's because if you go pi (180) degrees to the other side of the earth, you'll be 12 hours off. Finally, I have to multiply that by ephem.hour because ... um, because that's the way to add hours in PyEphem and they don't really document the internals of ephem.Date.

Set the observer date to this adjusted time before calculating your analemma, and you get the much more vertical figure you see here. This also explains why the morning and evening analemmas weren't symmetrical in the previous run.

This code is location independent, so now I can run my analemma program on a city name, or specify longitude and latitude.

PyEphem turned out to be a great tool for exploring analemmas. But to really understand analemma shapes, I had more exploring to do. I'll write about that, and post my complete analemma program, in the next article.

Tags: analemma, astronomy, science, programming, python, writing
[ 19:54 Dec 29, 2011    More science/astro | permalink to this entry | comments ]

Calculating the Solstice and shortest day

Today is the winter solstice -- the official beginning of winter.

The solstice is determined by the Earth's tilt on its axis, not anything to do with the shape of its orbit: the solstice is the point when the poles come closest to pointing toward or away from the sun. To us, standing on Earth, that means the winter solstice is the day when the sun's highest point in the sky is lowest.

You can calculate the exact time of the equinox using the handy Python package PyEphem. Install it with: easy_install pyephem for Python 2, or easy_install ephem for Python 3. Then ask it for the date of the next or previous equinox. You have to give it a starting date, so I'll pick a date in late summer that's nowhere near the solstice:

>>> ephem.next_solstice('2011/8/1')
2011/12/22 05:29:52

That agrees with my RASC Observer's Handbook: Dec 22, 5:30 UTC. (Whew!)

PyEphem gives all times in UTC, so, since I'm in California, I subtract 8 hours to find out that the solstice was actually last night at 9:30. If I'm lazy, I can get PyEphem to do the subtraction for me:

ephem.date(ephem.next_solstice('2011/8/1') - 8./24)
2011/12/21 21:29:52

I used 8./24 because PyEphem's dates are in decimal days, so in order to subtract 8 hours I have to convert that into a fraction of a 24-hour day. The decimal point after the 8 is to get Python to do the division in floating point, otherwise it'll do an integer division and subtract int(8/24) = 0.

The shortest day

The winter solstice also pretty much marks the shortest day of the year. But was the shortest day yesterday, or today? To check that, set up an "observer" at a specific place on Earth, since sunrise and sunset times vary depending on where you are. PyEphem doesn't know about San Jose, so I'll use San Francisco:

>>> import ephem
>>> observer = ephem.city("San Francisco")
>>> sun = ephem.Sun()
>>> for i in range(20,25) :
...   d = '2011/12/%i 20:00' % i
...   print d, (observer.next_setting(sun, d) - observer.previous_rising(sun, d)) * 24
2011/12/20 20:00 9.56007901422
2011/12/21 20:00 9.55920379754
2011/12/22 20:00 9.55932991847
2011/12/23 20:00 9.56045709446
2011/12/24 20:00 9.56258416496

I'm multiplying by 24 to get hours rather than decimal days.

So the shortest day, at least here in the bay area, was actually yesterday, 2011/12/21. Not too surprising, since the solstice wasn't that long after sunset yesterday.

If you look at the actual sunrise and sunset times, you'll find that the latest sunrise and earliest sunset don't correspond to the solstice or the shortest day. But that's all tied up with the equation of time and the analemma ... and I'll cover that in a separate article.

Tags: astronomy, science, programming, python, writing
[ 10:28 Dec 22, 2011    More science/astro | permalink to this entry | comments ]

New trails, and new PyTopo 1.1 release

A new trail opened up above Alum Rock park! Actually a whole new open space preserve, called Sierra Vista -- with an extensive set of trails that go all sorts of interesting places.

Dave and I visit Alum Rock frequently -- we were married there -- so having so much new trail mileage is exciting. We tried to explore it on foot, but quickly realized the mileage was more suited to mountain bikes. Even with bikes, we'll be exploring this area for a while (mostly due to not having biked in far too long, so it'll take us a while to work up to that much riding ... a combination of health problems and family issues have conspired to keep us off the bikes).

Of course, part of the fun of discovering a new trail system is poring over maps trying to figure out where the trails will take us, then taking GPS track logs to study later to see where we actually went.

And as usual when uploading GPS track logs and viewing them in pytopo, I found some things that weren't working quite the way I wanted, so the session ended up being less about studying maps and more about hacking Python.

In the end, I fixed quite a few little bugs, improved some features, and got saved sites with saved zoom levels working far better.

Now, PyTopo 1.0 happened quite a while ago -- but there were two of us hacking madly on it at the time, and pinning down the exact time when it should be called 1.0 wasn't easy. In fact, we never actually did it. I know that sounds silly -- of all releases to not get around to, finally reaching 1.0? Nevertheless, that's what happened.

I thought about cheating and calling this one 1.0, but we've had 1.0 beta RPMs floating around for so long (and for a much earlier release) that that didn't seem right.

So I've called the new release PyTopo 1.1. It seems to be working pretty solidly. It's certainly been very helpful to me in exploring the new trails. It's great for cross-checking with Google Earth: the OpenCycleMap database has much better trail data than Google does, and pytopo has easy track log loading and will work offline, while Google has the 3-D projection aerial imagery that shows where trails and roads were historically (which may or may not correspond to where they decide to put the new trails). It's great to have both.

Anyway, here's the new PyTopo.

Tags: mapping, programming, python
[ 19:59 Nov 16, 2011    More mapping | permalink to this entry | comments ]

The Stanford online Machine Learning class

In case you haven't been following it, Stanford's computer science department began a grand experiment in online learning early this month: free, upper division college courses, given online and open to the whole world. There are three classes offered: Artificial Intelligence, Machine Learning and Introduction to Databases.

They've sparked an incredible response: exact numbers don't seem to be available, but rumor is that AI had about 130,000 enrolees, while ML had about 70,000. (Nobody seems to have published numbers for DB.) Update, a day later: @seemsArtless tweets that ML currently has 87,000 registered users.

Why so much interest? Surely there are lots of places to get free information (like wikipedia) and even course lectures (like MIT). And there are plenty of places to take classes for relatively low cost, like local junior colleges or ed2go.

What's different about the Stanford classes is that they cover advanced material, in far more depth than you'd find at a junior college or typical online site. They offer graded homework so you can see how you're doing, and there are other students taking the class at the same time, so if you get stuck, there are all sorts of discussion groups you can turn to. It's one thing to read a textbook or watch a video by yourself; I find a class much more helpful, and judging by the response to the Stanford classes, I'm not alone in that.

I agonized over whether to take AI or Machine Learning. They both sounded so interesting! Since I couldn't decide, I initially signed up for both, figuring I'd drop one if the load was too great. By the end of the second week, I'd settled on Machine Learning. I was starting to dread the AI class flash quizzes -- which didn't always work right, but made it hard to proceed until you'd answered the question right even if you couldn't see the question -- and to feel frustrated about the lectures, which clearly were meant as a jumping off point for students to go do their own outside reading.

On the other hand, I was really enjoying the Machine Learning lectures, and looking forward to each new one. And the real kicker: Machine Learning includes programming assignments, so students can implement the algorithms Professor Ng talks about in the lectures.

What's great about Machine Learning

Andrew Ng's video lectures are wonderfully clear, well paced and full of interesting content.

He uses a lot of graphs to help students visualize what's going on geometrically, rather than just relying on the equations. (Better yet, in the programming exercises he shows us how to create those graphs for ourselves.)

And he's great about flagging certain portions as possibly review (you can skip this lecture if you already know linear algebra) or advanced (this is some extra background for people who know calculus, but you can skip it and still do fine in the course).

The technology is simpler than that used in the AI course. If you have a slow net connection or travel a lot, you can download the lectures as mp4 files and watch them offline. You can download lecture slides as a PDF or PPT. Review questions (graded) are handled with simple HTML forms. All very simple, well-tested technology, and it works great. I've had no problems accessing the servers, submitting homework or anything else -- very impressive!

But the heart of the course is the programming exercises. ML is taught in GNU octave, a framework and language for numerical computing and matrix operations. Students aren't absolutely required to use octave, but it's highly recommended: Professor Ng says he's found that students learn much faster that way. Sounds good to me, and octave looks like a useful skill, well worth acquiring. I'm having fun learning it.

The programming exercises come with a lot of scaffold code plus a few files with "Your code goes here". The actual amount of coding isn't large. But I'm finding that it does the job: it forces me to make sure I understand the matrix operations discussed in the lectures. And at the end, you come out with something that's actually useful! From the first few weeks, I have linear and logistic regression code that I could use to analyze and visualize all sorts of datasets. Now, at the end of week 4, we're halfway through writing a neural network to recognize handwritten numerals from image data. How cool is that?

Suggestions for improvement

The class is a huge success. Who would have thought that you could teach something this advanced on such a huge scale, so effectively?

I have only a couple of small suggestions -- ways the class could be even better next time.

• An errata page. In week 3, there was an error in the lecture and notes, a - instead of a +, that made one part of programming ex. 2 quite a bit trickier than it would otherwise have been. If I hadn't noticed that the slides used + in some places and - in others, I might never have gotten that part of the assignment working. Lots of other people found that too, and there were discussions in the Q&A forum ... but you wouldn't find it without coming up with clever search terms.
• The Q&A forum would be so much more useful if it was organized by topic, and/or by week. There are some great discussions there, but the only way of getting to them is by searching for the right terms. There's no way to browse discussions, see how people are doing on assignment 3, or look for errata and similar warnings. It would help make the class more of a community, more like a real in-person class.

Hope for future expansion

I mentioned my suggestions because I fervently hope there is a "next time". These classes are a great service, and I hope the huge response isn't putting too much burden on the instructors.

"Common wisdom" among providers of online classes seems to be that there's no demand outside of enrolled university students for hard courses, courses with prerequisites, and especially courses that involve (shudder) math. Just look at the offerings from any online courseware or adult ed program -- they're long on art appreciation and "Introduction to MS Word", short on physics and econometrics. Even the for-pay online degree mills concentrate on humanities and business, not technical subjects.

Stanford's experiment has proven that "common wisdom" is wrong -- that tens of thousands of students will jump at the chance to take highly technical, mathematical courses. I'd love to see the model expanded to other subjects, such as statistics, economics, physics, geology and climate science.

And, yes, there is money to be made here. If this many people will take a free class, wouldn't quite a few of them be willing to pay? Most couldn't afford $1000 like UC Extension classes -- but how about $100, comparable to other online education classes? Would people pay more if you offered college credit?

Online education providers, take note! There's a large, underserved market for scientific and technical classes out here in the long tail.

Tags: education, programming
[ 16:31 Nov 05, 2011    More education | permalink to this entry | comments ]

Monitor an Arduino's serial output from Python

Debugging Arduino sensors can sometimes be tricky. While working on my Arduino sonar project, I found myself wanting to know what values the Arduino was reading from its analog port.

It's easy enough to print from the Arduino to its USB-serial line. First add some code like this in setup():

    Serial.begin(9600);

Then in loop(), if you just read the value "val":

    Serial.println(val);

Serial output from Python

That's all straightforward -- but then you need something that reads it on the PC side.

When you're using the Arduino Java development environment, you can set it up to display serial output in a few lines at the bottom of the window. But it's not terrifically easy to read there, and I don't want to be tied to the Java IDE -- I'm much happier doing my Arduino development from the command line. But then how do you read serial output when you're debugging? In general, you can use the screen program to talk to serial ports -- it's the tool of choice to log in to plug computers. For the Arduino, you can do something like this: screen /dev/ttyUSB0 9600

But I found that a bit fiddly for various reasons. And I discovered that it's easy to write something like this in Python, using the serial module.

You can start with something as simple as this:

import serial

ser = serial.Serial("/dev/ttyUSB0", 9600)
while True:
    print ser.readline()

Serial input as well as output

That worked great for debugging purposes. But I had another project (which I will write up separately) where I needed to be able to send commands to the Arduino as well as reading output it printed. How do you do both at once?

With the select module, you can monitor several file descriptors at once. If the user has typed something, send it over the serial line to the Arduino; if the Arduino has printed something, read it and display it for the user to see.

That loop looks like this:

while True :
    # Check whether the user has typed anything (timeout of .2 sec):
    inp, outp, err = select.select([sys.stdin, self.ser], [], [], .2)

    # If the user has typed anything, send it to the Arduino:
    if sys.stdin in inp :
        line = sys.stdin.readline()
        self.ser.write(line)

    # If the Arduino has printed anything, display it:
    if self.ser in inp :
line = self.ser.readline().strip()
print "Arduino:", line

Add in a loop to find the right serial port (the Arduino doesn't always show up on /dev/ttyUSB0) and a little error and exception handling, and I had a useful script that met all my Arduino communication needs: ardmonitor.

Tags: arduino, hardware, programming, python
[ 19:27 Oct 16, 2011    More hardware | permalink to this entry | comments ]

Banishing errant tooltips

Every now and then I have to run a program that doesn't manage its tooltips well. I mouse over some button to find out what it does, a tooltip pops up -- but then the tooltip won't go away. Even if I change desktops, the tooltip follows me and stays up on all desktops. Worse, it's set to stay on top of all other windows, so it blocks anything underneath it.

The places where I see this happen most often are XEphem (probably as an artifact of the broken Motif libraries we're stuck with on Linux); Adobe's acroread (Acrobat Reader), though perhaps that's gotten better since I last used it; and Wine.

I don't use Wine much, but lately I've had to use it for a medical imaging program that doesn't seem to have a Linux equivalent (viewing PETscan data). Every button has a tooltip, and once a tooltip pops up, it never goes aawy. Eventually I might have five of six of these little floating windows getting in the way of whatever I'm doing on other desktops, until I quit the wine program.

So how does one get rid of errant tooltips littering your screen? Could I write an Xlib program that could nuke them?

Finding window type

First we need to know what's special about tooltip windows, so the program can identify them. First I ran my wine program and produced some sticky tooltips.

Once they were up, I ran xwininfo and clicked on a tooltip. It gave me a bunch of information about the windows size and location, color depth, etc. ... but the useful part is this:

  Override Redirect State: yes

In X, override-redirect windows are windows that are immune to being controlled by the window manager. That's why they don't go away when you change desktops, or move when you move the parent window.

So what if I just find all override-redirect windows and unmap (hide) them? Or would that kill too many innocent victims?

Python-Xlib

I thought I'd have to write my little app in C, since it's doing low-level Xlib calls. But no -- there's a nice set of Python bindings, python-xlib. The documentation isn't great, but it was still pretty easy to whip something up.

The first thing I needed was a window list: I wanted to make sure I could find all the override-redirect windows. Here's how to do that:

from Xlib import display

dpy = display.Display()
screen = dpy.screen()
root = screen.root
tree = root.query_tree()

for w in tree.children :
    print w

w is a Window (documented here). I see in the documentation that I can get_attributes(). I'd also like to know which window is which -- calling get_wm_name() seems like a reasonable way to do that. Maybe if I print them, those will tell me how to find the override-redirect windows:

for w in tree.children :
    print w.get_wm_name(), w.get_attributes()

Window type, redux

Examining the list, I could see that override_redirect was one of the attributes. But there were quite a lot of override-redirect windows. It turns out many apps, such as Firefox, use them for things like menus. Most of the time they're not visible. But you can look at w.get_attributes().map_state to see that.

So that greatly reduced the number of windows I needed to examine:

for w in tree.children :
    att = w.get_attributes()
    if att.map_state and att.override_redirect :
        print w.get_wm_name(), att

I learned that tooltips from well-behaved programs like Firefox tended to set wm_name to the contents of the tooltip. Wine doesn't -- the wine tooltips had an empty string for wm_name. If I wanted to kill just the wine tooltips, that might be useful to know.

But I also noticed something more important: the tooltip windows were also "transient for" their parent windows. Transient for means a temporary window popped up on behalf of a parent window; it's kept on top of its parent window, and goes away when the parent does.

Now I had a reasonable set of attributes for the windows I wanted to unmap. I tried it:

for w in tree.children :
    att = w.get_attributes()
    if att.map_state and att.override_redirect and w.get_wm_transient_for():
        w.unmap()

It worked! At least in my first test: I ran the wine program, made a tooltip pop up, then ran my killtips program ... and the tooltip disappeared.

Multiple tooltips: flushing the display

But then I tried it with several tooltips showing (yes, wine will pop up new tooltips without hiding the old ones first) and the result wasn't so good. My program only hid the first tooltip. If I ran it again, it would hide the second, and again for the third. How odd!

I wondered if there might be a timing problem. Adding a time.sleep(1) after each w.unmap() fixed it, but sleeping surely wasn't the right solution.

But X is asynchronous: things don't necessarily happen right away. To force (well, at least encourage) X to deal with any queued events it might have stacked up, you can call dpy.flush().

I tried adding that after each w.unmap(), and it worked. But it turned out I only need one

dpy.flush()

at the end of the program, just exiting. Apparently if I don't do that, only the first unmap ever gets executed by the X server, and the rest are discarded. Sounds like flush() is a good idea as the last line of any python-xlib program.

killtips will hide tooltips from well-behaved programs too. If you have any tooltips showing in Firefox or any GTK programs, or any menus visible, killtips will unmap them. If I wanted to make sure the program only attacked the ones generated by wine, I could add an extra test on whether w.get_wm_name() == "".

But in practice, it doesn't seem to be a problem. Well-behaved programs handle having their tooltips unmapped just fine: the next time you call up a menu or a tooltip, the program will re-map it.

Not so in wine: once you dismiss one of those wine tooltips, it's gone forever, at least until you quit and restart the program. But that doesn't bother me much: once I've seen the tooltip for a button and found out what that button does, I'm probably not going to need to see it again for a while.

So I'm happy with killtips, and I think it will solve the problem. Here's the full script: killtips.

Tags: X11, python, programming, tooltips, annoyances
[ 10:36 Sep 27, 2011    More programming | permalink to this entry | comments ]

Count characters or words in the X selection from Python

This post is, above all, a lesson in doing a web search first. Even when what you're looking for is so obscure you're sure no one else has wanted it. But the script I got out of it might turn out to be useful.

It started with using Bitlbee for Twitter. I love bitlbee -- it turns a Twitter stream into just another IRC channel tab in the xchat I'm normally running anyway.

The only thing I didn't love about bitlbee is that, unlike the twitter app I'd previously used, I didn't have any way to keep track of when I neared the 140-character limit. There were various ways around that, mostly involving pasting the text into other apps before submitting it. But they were all too many steps.

It occurred to me that one way around this was to select-all, then run something that would show me the number of characters in the X selection. That sounded like an easy app to write.

Getting the X selection from Python

I was somewhat surprised to find that Python has no way of querying the X selection. It can do just about everything else -- even simulate X events. But there are several command-line applications that can print the selection, so it's easy enough to run xsel or xclip from Python and read the output.

I ended up writing a little app that brings up a dialog showing the current count, then hangs around until you dismiss it, querying the selection once a second and updating the count. It's called countsel.

Of course, if you don't want to write a Python script you can use commandline tools directly. Here are a couple of examples, using xclip instead of xsel: xterm -title 'lines words chars' -geometry 25x2 -e bash -c 'xclip -o | wc; read -n 1' pops up a terminal showing the "wc" counts of the selection once, and xterm -title 'lines words chars' -geometry 25x1 -e watch -t 'xclip -o | wc' loops over those counts printing them once a second.

Binding commands to a key is different for every window manager. In Openbox, I added this to rc.xml to call up my program whenever I type W-t (short for Twitter):

    <keybind key="W-t">
      <action name="Execute">
        <execute>/home/akkana/bin/countsel</execute>
      </action>
    </keybind>

Now, any time I needed to check my character count, I could triple-click or type Shift-Home, then hit W-t to call up the dialog and get a count. Then I could leave the dialog up, and whenever I wanted a new count, just Shift-Home or triple-click again, and the dialog updates automatically. Not perfect, but not bad.

Xchat plug-in for a much more elegant solution

Only after getting countsel working did it occur to me to wonder if anyone else had the same Bitlbee+xchat+twitter problem. And a web search found exactly what I needed: xchat-inputcount.pl, a wonderful xchat script that adds a character-counter next to the input box as you're typing. It's a teensy bit buggy, but still, it's far better than my solution. I had no idea you could add user-interface elements to xchat like that!

But that's okay. Countsel didn't take long to write. And I've added word counting to countsel, so I can use it for word counts on anything I'm writing.

Tags: x, programming, python
[ 11:32 Sep 09, 2011    More programming | permalink to this entry | comments ]

Read Excel XLS spreadsheets with Python

Someone mailed out information to a club I'm in as an .XLS file. Another Excel spreadsheet. Sigh.

I do know one way to read them. Fire up OpenOffice, listen to my CPU fan spin as I wait forever for the app to start up, open the xls file, then click in one cell after another as I deal with the fact that spreadsheet programs only show you a tiny part of the text in each cell. I'm not against spreadsheets per se -- they're great for calculating tables of interconnected numbers -- but they're a terrible way to read tabular data.

Over the years, lots of open-source programs like word2x and catdoc have sprung up to read the text in MS Word .doc files. Surely by now there must be something like that for XLS files?

Well, I didn't find any ready-made programs, but I found something better: Python's xlrd module, as well as a nice clear example at ScienceOSS of how to Read Excel files from Python.

Following that example, in six lines I had a simple program to print the spreadsheet's contents:

import xlrd

for filename in sys.argv[1:] :
    wb = xlrd.open_workbook(filename)
    for sheetname in wb.sheet_names() :
        sh = wb.sheet_by_name(sheetname)
        for rownum in range(sh.nrows) :
            print sh.row_values(rownum)

Of course, having gotten that far, I wanted better formatting so I could compare the values in the spreadsheet. Didn't take long to write, and the whole thing still came out under 40 lines: xlsrd. And I was able to read that XLS file that was mailed to the club, easily and without hassle.

I'm forever amazed at all the wonderful, easy-to-use modules there are for Python.

Tags: python, programming
[ 09:58 Aug 31, 2011    More programming | permalink to this entry | comments ]

Teaching programming with "program a person"

A few weeks ago, at the annual GetSET engineering summer camp for high school girls, I taught my usual one-day workshop on beginning programming in Javascript.

The big question every year is always how to make the class more interactive. The girls who come to GetSET are great -- smart and motivated -- but after six hours of lectures and working through exercises, anyone, of any age, is going to glaze over. Especially when it's their first introduction to programming and they only have a day to learn it. People learn better when they're asking questions, thinking and solving problems, not just listening or following instructions.

For years I've heard vague references to "programming a person" as an exercise for teaching the basic idea of programming. The idea is to get the students to come up with step-by-step instructions for someone to do something -- say, walk across the room and pick up a water bottle -- so they realize how specific you have to be. It also solves another problem: giving everyone a break from sitting still and focusing on a computer screen.

But how do you really do it? What kind of problems work best in practice? How much time should you allow? If you have a volunteer carrying out the instructions, how do you keep them from skipping steps? Surprisingly, I couldn't find anything written up to help an inexperienced would-be teacher of programming.

What I needed was a chance to try out some ideas, or watch someone with more of a clue on this sort of teaching. This year, I found opportunities for both.

First try: Toastmasters

One of the reasons I love Toastmasters, especially with a small and friendly club like Coherent Communicators, is that it offers a safe place to try new presentation techniques and get good feedback about what does and doesn't work. So I made my first try at a Toastmasters meeting a few weeks before the GetSET workshop.

I allowed 15-20 minutes for the exercise. I explained to the audience that I wanted them to get me to turn left, walk over to the easel at the side of the room, touch it, turn around, walk back to the lectern, pick up the gavel and pound it on the lectern. I would solicit a command from them, write it on the whiteboard, then carry out the command and ask for the next command.

The day's audience was a fairly even mix of techies and non. I had wondered whether the audience would be widely mixed in how specific their instructions were, but they were fairly uniform -- mostly along the lines of "Turn 90 degrees left." "Take 5 steps." "Take 2 more steps". Of course, there were a few joking suggestions from the techies, like "send an electrical impulse from your brain to your left quadriceps", that you wouldn't expect with a high school group, but mostly everyone was on the same page.

When I got near the easel, we hit "Raise your right arm". (Oops, not close enough yet.) "Um ... lean forward about a foot?" A good illustration of being specific ... just the sort of thing I was hoping for.

They got me back to the lectern, got me to pick up the gavel (I was letting them skip a few steps by this point) ... and improvised a little, getting me to knock my head rather than the lectern. That was fun, and got some laughs ... it worked well.

I had hoped to do a second run where I guided them into understanding a while loop ("while (not yet to the easel), take another step"). But seeing a yellow light from the timer, I opted for a quick explanation of how a loop would work rather than guiding the audience into it. I found out later that the timer had hit the wrong button and only given me 8 minutes rather than my requested 15-20 ... so 20 minutes actually would have been plenty of time to cover loops as well as basic instructions. Disappointing ... but I was surprised we'd gotten so much done in so little time.

Lessons learned:

• Draft a volunteer to write the instructions on the board. It was distracting and time-wasting to run back from the side of the room to write each new instruction.
• You can teach the basic concepts in less than ten minutes.

Try 2: "Program a blind robot"

For the real workshop, I had help in the form of Esther Heller, an experienced girl scout leader as well as many year GetSET veteran. Esther had done exercises like this before and was willing to take the lead; I was looking forward to learning from her. We had discussed two different variants, and decided to try both of them at different times during the day.

For the first variant, we waited until mid-morning when the class was bogging down a bit and looked like they needed a break. Esther called for two volunteers: one programmer and one robot. The girl playing the robot was blindfolded with a bandanna and escorted to the door of the room, while Esther whispered the task to the other girl. The task was something like walking over to a water bottle, picking it up, walking over to another girl and handing it to her -- though the rest of us didn't know that until it was completed.

The instructions suggested by the girls were quite similar to the ones I'd heard in Toastmasters. There was lots of "Take 5 steps" ... "take two more steps", guessing at how many steps it would take to get from one place to another. No one came up with anything like a loop or conditional. I'd wondered if anyone would try remote control -- "walk" then wait until the right moment to yell "STOP!" -- but no one did.

The blindfolding worked really well. I'd worried that with a volunteer chosen to be the robot, she might skip steps she hadn't been given. But if the "robot" is blindfolded and doesn't know the task, she can't skip steps; she can only do what she's programmed to. The only problem was that a blindfolded person told to walk straight ahead does not necessarily hold to a straight line, much to the consternation of the girl playing the programmer.

There was a lot of "turn right" ... "no, not that much, turn back left again" ... "now turn JUST A LITTLE to the right" that helped stress the need for specificity -- exactly what we were after. I had wondered beforehand whether anyone would ever suggest anything like "turn right by 30 degrees", but no one, either in Toastmasters or GetSET, ever did.

The exercise was successful and everybody seemed to have fun, so it broke up the morning well. We didn't get to loops or conditionals, though. I didn't record how long we spent, but it was probably in the neighborhood of 20 minutes.

Lessons:

• Blindfolding and choosing a volunteer definitely helps this exercise: it solves the problem of a volunteer who might skip steps.
• I wished the whole room knew what the task was ... but I'm not sure how to accomplish that. Either you have to escort the "robot" far enough away that she can't hear you explain it, or write it on the board after she's blindfolded. Extra time either way.

Try 3, in groups: "The muffin is ready"

At the end of the day, we tried Esther's favorite variant. You're watching TV, and you want to go to the kitchen, get an English muffin, toast it, put butter/jam/peanut butter/whatever on it, take it back to your seat and eat it. What are the steps?

Esther divided the girls into groups of 4-5 and passed out post-its on which to write the steps. There was some inertia getting started ... it was late in the day and everybody was tired. (That's not unique to this exercise -- it's always a challenge to come up with something that will hold the girls' interest for the last hour. It's a long day for everyone.)

Eventually they got rolling and got into it -- I saw some very long stacks of post-its from various groups. With ten minutes left to go in the session, Esther picked two volunteers from one group: one to read the instructions, one to execute them. She pointed out places where they skipped steps -- "Hey, wait, how can she get the muffins out of the cupboard without opening the cupboard first?" After a minute or two, Esther called on a new pair from a different group to continue where the first pair had left off.

As she worked through all the groups, you could see each group becoming more cognizant of steps they had skipped, and improvising them on the spot. Despite the end-of-day crankiness, you could see they were learning from the exercise.

Lessons:

• Splitting into groups allows for more discussion among the girls, and comparing various groups' answers is fun.
• Splitting into groups takes a lot of time, and you have to monitor to make sure all the groups are actually working on the problem and not just chatting.

So which is better? The muffin exercise was definitely more time consuming than the previous "robot" exercise, due to overhead of splitting into groups and bringing up volunteers from each group. On the other hand, I could see there was benefit in having them work in small groups, and in the touch of competition in comparing their group's answers with the ones from other groups.

It was hard to compare the two exercises directly to see which one worked better, because of end-of-day crankiness. But they both worked well -- I'm going to keep using some variant of this in future workshops, ideally with loops and conditionals added. Thanks, Esther, for your expertise ... and to the students and the rest of the volunteers for making it a successful class!

Tags: education, programming, toastmasters
[ 15:34 Aug 28, 2011    More education | permalink to this entry | comments ]

Deleting email from a mail server with Python

How do you delete email from a mail server without downloading or reading it all?

Why? Maybe you got a huge load of spam and you need to delete it. Maybe you have your laptop set up to keep a copy of your mail on the server so you can get it on your desktop later ... but after a while you realize it's not worth downloading all that mail again. In my case, I use an ISP that keeps copies of all mail forwarded from one alias to another, so I periodically need to clean out the copies.

There are quite a few reasons you might want to delete mail without reading it ... so I was surprised to find that there didn't seem to be any easy way to do so.

But POP3 is a fairly simple protocol. How hard could it be to write a Python script to do what I needed?

Not hard at all, in fact. The poplib package does most of the work for you, encapsulating both the networking and the POP3 protocol. It even does SSL, so you don't have to send your password in the clear.

Once you've authenticated, you can list() messages, which gives you a status and a list of message numbers and sizes, separated by a space. Just loop through them and delete each one.

Here's a skeleton program to delete messages:

server = "mail.example.com"
port = 995
user = "myname"
passwd = "seekrit"

pop = poplib.POP3_SSL(server, port)
pop.user(user)
pop.pass_(passwd)

poplist = pop.list()
if poplist[0].startswith('+OK') :
    msglist = poplist[1]
    for msgspec in msglist :
        # msgspec is something like "3 3941", 
        # msg number and size in octets
        msgnum = int(msgspec.split(' ')[0])
        print "Deleting msg %d\r" % msgnum,
        pop.dele(msgnum)
    else :
        print "No messages for", user
else :
    print "Couldn't list messages: status", poplist[0]
pop.quit()

Of course, you might want to add more error checking, loop through a list of users, etc. Here's the full script: deletemail.

Tags: email, python, programming
[ 16:41 Aug 25, 2011    More programming | permalink to this entry | comments ]

Beginning Python: Sorting lists of objects

The Beginning Python class has pretty much died down -- although there are still a couple of interested students posting really great homework solutions, I think most people have fallen behind, and it's time to wrap up the course.

So today, I didn't post a formal lesson. But I did have something to share about how I used Python's object-oriented capabilities to solve a problem I had copying new podcast files onto my MP3 player. I used Python's built-in list sort() function, along with the easy way it lets me define operators like < and > for any object I define.

You can read all about it in my post to the Courses list describing how I sorted my list of podcast objects. Or just go straight to the final program, pods.

Tags: python, programming, education
[ 18:48 Aug 19, 2011    More education | permalink to this entry | comments ]

Beginning Python, Lesson 9: More extras

Lesson 9 in my online Python course is up: Lesson 9: Extras (requested topics), including string operations, web development and GUI toolkits.

The web development and GUI toolkits are topics which were requested by students, while the string ops are things that just seemed too useful not to include.

Tags: python, programming, education
[ 16:45 Aug 12, 2011    More education | permalink to this entry | comments ]

Beginning Python, Lesson 8: Extras

Lesson 8 in my online Python course is up: Lesson 8: Extras, including exception handling, optional arguments, and running system commands. A motley collection of fun and useful topics that didn't quite fit anywhere in the earlier formal lessons, but you'll find a lot of use for them in writing real-world Python scripts. In the homework, I have some examples of some of my scripts using these techniques; I'm sure the students will have lots of interesting problems of their own.

Tags: python, programming, education
[ 13:56 Aug 05, 2011    More education | permalink to this entry | comments ]

Beginning Python, Lesson 7: Object-oriented programming

Lesson 7 in my online Python course is up: Lesson 7: Object-oriented programming.

This is the last formal lesson in the Beginning Python class. But I will be posting a few more "tips and tricks" lessons, little things that didn't fit in other lessons plus suggestions for useful Python packages students may want to check out as they continue their Python hacking.

Tags: python, programming, education
[ 09:28 Jul 30, 2011    More education | permalink to this entry | comments ]

Beginning Python, Lesson 6: Functions and Dictionaries

Lesson 6 in my online Python course is up: Lesson 6: Functions and Dictionaries.

We're getting near the end of the course -- partly because I think students may be saturated, though I may post one more lesson. I'll post on the list and see what the students think about it.

This afternoon, though, is pretty much booked up trying to get my mother's new Nook Touch e-book reader working with Linux. Would be easy ... except that she wants to be able to check out books from her local public library, which of course uses proprietary software from Adobe and other companies to do DRM. It remains to be seen if this will be possible ... of course, I'll post the results once we know.

Tags: python, programming, education
[ 16:49 Jul 22, 2011    More education | permalink to this entry | comments ]

Beginning Python, Lesson 5

Lesson 5 in my online Python course is up: Infinite loops, modulo, and random numbers.

It's a motley mix of topics, mostly because I wanted to have a fun homework project that actually did something interesting. I hope everyone enjoys it!

Tags: python, programming, education
[ 15:44 Jul 15, 2011    More education | permalink to this entry | comments ]

Beginning Python, Lesson 4

Lesson 4 in my online Python course is up: Modules and command-line arguments.

This lesson is a little longer than previous lessons, but that's partly because of a couple of digressions at the beginning. Hope I didn't overdo it! The homework includes an optional debugging problem for folks who want to dive a little deeper into this stuff.

Tags: python, programming, education
[ 19:20 Jul 08, 2011    More education | permalink to this entry | comments ]

Beginning Python, Lesson 3: Strings and Lists

Lesson 3 in my online Python course is up: Fun with Strings and Lists.

There may be some backlog on the mailing list -- my first attempt to post the lesson didn't show up at all, but my second try made it. Mail seems to be flowing now, but if you try to post something and it doesn't show up, let me know or tell us on irc.linuxchix.org, so we know if there's a continuing problem that needs to be fixed, not just a one-time glitch.

Meanwhile, I'm having some trouble getting new blog entries posted. Due to some network glitches, I had to migrate shallowsky.com to a different ISP, and it turns out the PyBlosxom 1.4 I'd been using doesn't work with more recent versions of Python; but none of my PyBlosxom plug-ins work in 1.5. Aren't software upgrades a joy? So I'm getting lots of practice debugging other people's Python code trying to get the plug-ins updated, and there probably won't be many blog entries until I've figured that out.

Once that's all straightened out, I should have a cool new PyTopo feature to report on, as well as some Arduino hacks I've had on the back burner for a while.

Tags: python, programming, education
[ 10:57 Jul 03, 2011    More education | permalink to this entry | comments ]

Beginning Python, Lesson 2 posted

I've just posted Lesson 2 in my online Python course, covering loops, if statements, and beer! You can read it in the list archives: Lesson 2: Loops, if, and beer, or, better, subscribe to the list so you can join the discussion.

I hope everybody has fun writing loops!

Tags: python, programming, education
[ 15:10 Jun 24, 2011    More education | permalink to this entry | comments ]

Beginning Programming in Python course starting

I'm about to start a new LinuxChix course: Beginning Programming in Python.

It will be held on the Linuxchix Courses mailing list: to follow the course, subscribe to the list. Lessons will be posted weekly, on Fridays, with the first lesson starting tomorrow, Friday, June 17.

This is intended a short course, probably only 4-5 weeks to start with, aimed mostly at people who are new to programming. Though of course anyone is welcome, even if you've programmed before. And experienced programmers are welcome to hang out, lurk and help answer questions. I might extended the course if people are still interested and having fun.

The course is free (just subscribe to the mailing list) and open to both women and men. Standard LinuxChix rules apply: Be polite, be helpful. And do the homework. :-)

Tags: python, programming, education
[ 08:51 Jun 16, 2011    More education | permalink to this entry | comments ]

Command-line Arduino development

I've been doing more Arduino development lately. But I don't use the Arduino Java development environment -- programming is so much easier when you have a real editor, like emacs or vim, and key bindings to speed everything up.

I've found very little documentation on how to do command-line Arduino development, and most of the Makefiles out there are old and no longer work. So I've written up a tutorial. It ended up too long for a blog post, so I've made it a separate article:

Command-line Arduino development.

Tags: hardware, arduino, cmdline, programming
[ 13:45 May 30, 2011    More programming | permalink to this entry | comments ]

Packaging Python for MeeGo (or other RPM-based distros)

Writing Python scripts for MeeGo is easy. But how do you package a Python script in an RPM other MeeGo users can install?

It turned out to be far easier than I expected. Python and Ubuntu had all the tools I needed.

First you'll need a .desktop file describing your app, if you don't already have one. This gives window managers the information they need to show your icon and application name so the user can run it. Here's the one I wrote for PyTopo: pytopo.desktop.

Of course, you'll also want a desktop icon. Most other applications on MeeGo seemed to use 48x48 pixel PNG images, so that's what I made, though it seems to be quite flexible -- an SVG is ideal.

With your script, desktop file and an icon, you're ready to create a package.

Create a setup.py file describing your package, as in the distutils simple example or the more detailed distutils setup script page. For a sample standalone script with a desktop file and icon, you can take a look at my PyTopo setup.py.

Starting from the Python setup script, Python's distutils can generate RPM or even Windows packages -- assuming you have the appropriate tools installed on your machine.

I'm on an Ubuntu (Debian-based) machine, and all the docs imply you have to be on an RPM-based distro to make an RPM. Happily, that's not true: Ubuntu has RPM tools you can install.

$ sudo apt-get install rpm

Then let Python do its thing:

$ python setup.py bdist_rpm

Python generates the spec file and everything else needed and builds a multiarch RPM that's ready to install on MeeGo. You can install it by copying it to the MeeGo device with scp dist/PyTopo-1.0-1.noarch.rpm meego@address.of.device:/tmp/. Then, as root on the device, install it with rpm -i /tmp/PyTopo-1.0-1.noarch.rpm. You're done!

To see a working example, you can browse my latest PyTopo source (only what's in SVN; it needs a few more tweaks before it's ready for a formal release). Or try the RPM I made for MeeGo: PyTopo-1.0-1.noarch.rpm. I'd love to hear whether this works on other RPM-based distros.

What about Debian packages?

Curiously, making a Debian package on Debian/Ubuntu is much less straightforward even if you're starting on a Debian/Ubuntu machine. Distutils can't do it on its own. There's a Debian Python package recipe, but it begins with a caution that you shouldn't use it for a package you want to submit. For that, you probably have to wade through the Complete Ubuntu Packaging Guide. Clearly, that will need a separate article.

Tags: python, meego, programming, rpm, redhat
[ 17:44 May 20, 2011    More programming | permalink to this entry | comments ]

Children of the Code -- Derived Python projects

I got some fun email today -- two different people letting me know about new projects derived from my Python code.

One is M-Poker, originally based on a PyQt tutorial I wrote for Linux Planet. Ville Jyrkkä has taken that sketch and turned it into a real poker program. And it uses PySide now -- the new replacement for PyQt, and one I need to start using for MeeGo development. So I'll be taking a look at M-Poker myself and maybe learning things from it. There are some screenshots on the blog A Hacker's Life in Finland.

The other project is xkemu, a Python module for faking keypresses, grown out of pykey, a Python version of my Crikey keypress generation program. xkemu-server.py looks like a neat project -- you can run it and send it commands to generate key presses, rather than just running a script each time.

(Sniff) My children are going out into the world and joining other projects. I feel so proud. :-)

Tags: programming, python
[ 20:04 May 13, 2011    More programming | permalink to this entry | comments ]

Firefox 4: Fixing middlemouse.content load, and hacking jars

Mostly the transition to Firefox4 has pretty smooth. But there's been one big hassle: middlemouse content load URL doesn't work as well as it used to.

Middlemouse content load is a great Firefox feature on Linux and other Unix platforms. You see a URL somewhere that doesn't have clickable URLs -- say, a plaintext mail message, or something somebody typed in IRC. You highlight it with the mouse -- no need to Copy explicitly -- X does that automatically whenever you highlight text). Then move to the Firefox window and click the middle mouse button somewhere in the content window -- anywhere as long as it's not over a link or text input -- and Firefox goes straight to the URL you pasted.

A few silly Linux distros, like Ubuntu, disable this feature by default. You can turn it back on by going to about:config, searching for middlemouse, and setting middlemouse.contentLoadURL to true.

Except, in Firefox 4, much of the time nothing happens. In Firefox 4, contentLoadURL only works if the URL you pasted is a complete URL, like http://example.com. This is completely silly, because most of the time, if you had a complete URL, it would have been clickable already in whatever window you originally found it in. When you need contentLoadURL is when someone types "Go to example.com if you want to see this". It's also great for when you get those horrible Facebook or Stumbleupon or Google URLs like http://www.facebook.com/l/bfd4f/example.com/somelink and you want just the real link (example.com/somelink), without the added cruft.

Hacking the jar

Hooray! It turns out the offending code is in browser.js, so it's hackable without needing to recompile all of Firefox. You just need to unpack omni.jar, patch browser.js, then zip up a new omni.jar.

In other words, something like this (on Ubuntu Natty, starting in an empty directory):

$ cp /usr/lib/firefox-4.0/omni.jar ~/omni-jar.sav
$ unzip /usr/lib/firefox-4.0/omni.jar
  [ edit or patch chrome/browser/content/browser/browser.js ]
$ rm -f /tmp/new-omni.jar; zip /tmp/new-omni.jar `find .`
$ sudo cp /tmp/new-omni.jar /usr/lib/firefox-4.0/omni.jar

Getting Firefox to notice code changes

Except, as I was testing this, I discovered: I could make changes and most of the time Firefox wouldn't see them. I would put in something obvious like alert("Hello, world");, verify that the alert was really in omni.jar, run Firefox, click the middle mouse button and -- no alert. Where was Firefox getting the code it was actually running, if not from omni.jar?

I'll spare you the agonizing details of the hunt and just say that eventually I discovered that if I ran Firefox from a different profile on the same machine, I got a different result. It turns out that if you remove either of two files, extensions.sqlite and XUL.mfasl, Firefox4 will re-read the new code in omni.jar.

Removing XUL.mfasl seems to be a little safer: extensions.sqlite contains some details of which extensions are enabled. Of course, back up both files first before experimenting with removing them.

Why these files are keeping a cache of code that's already in omni.jar is anybody's guess.

The Patch: fix contentLoadURL

Oh, and the change? Mikachu came up with a cleaner fix than mine, so this is his. It accepts partial URLs like example.com and also bookmarklet keywords:

--- browser.js.sav      2011-05-07 16:40:03.672540242 -0700
+++ omni/chrome/browser/content/browser/browser.js      2011-05-08 16:29:28.943313984 -0700
@@ -10597,12 +10597,10 @@
   clipboard.replace(/\s*\n\s*/g, "");
 
   let url = getShortcutOrURI(clipboard);
-  try {
-    makeURI(url);
-  } catch (ex) {
-    // Not a valid URI.
-    return;
-  }
+  var URIFixup = Components.classes["@mozilla.org/docshell/urifixup;1"]
+                           .getService(Components.interfaces.nsIURIFixup);
+  url = URIFixup.createFixupURI(url, 1).spec;
+  // 1 is FIXUP_FLAG_ALLOW_KEYWORD_LOOKUP
 
   try {
     addToUrlbarHistory(url);

Tags: firefox, mozilla, programming, browsers
[ 19:28 May 09, 2011    More tech/web | permalink to this entry | comments ]

Overview of MeeGo Development Tools

Over the past week I've been playing with the MeeGo ExoPC tablet, experimenting with the various options for building programs.

One advantage MeeGo has over Android or iOS is that there are quite a lot of language and toolkit options. If you have an existing program, especially if it runs on Linux, you can probably find a way to port it to MeeGo fairly without much extra coding.

But since MeeGo is still quite new, not all of the options work quite as well as you might hope -- yet. I'm sure they'll get better, but here's what the development climate looks like now.

C++, Qt and Meego SDK

C++ and Qt are the primary supported environment on MeeGo, via the MeeGo SDK, run in an IDE called QtCreator.

The documentation turns out to be somewhat incomplete; I'll write up a howto as a separate article. But once you get it installed (much easier than installing, say, Eclipse for Android), it runs nicely. You can use normal Qt, not a specialized environment, so existing Qt programs should be quick to port, and the IDE takes care of packaging, copying the app to your remote device and starting it running. Very painless.

Testing apps locally isn't so easy. They're set up to use QEMU, which only works if your development machine has hardware virtualization. Supposedly there's a way to make this work in virtualbox (which doesn't require hardware virtualization), but the docs don't address how.

Still, testing on the target device is easy. Unfortunately, not many of my existing programs are C++ and Qt. I mostly write Python, C, and Javascript/HTML web apps.

Update: I should have mentioned that QtCreator also lets you program in QML, an XML-like language that lets you design Qt user interface and even write application code.

Web Apps under C++ and QWebView

So what about those web apps? Since the C++ SDK was so well supported, I figured, why not use a QWebView so I can run my HTML and Javascript code?

Unfortunately QWebView turns out to be tricky to use, has very few sample apps available, and so far I've been unable to get it to work under MeeGo at all.

Nokia Web RunTime

And anyway, for pure web apps, there's a kit explicitly for that: WRT, the Nokia Web RunTime. But it's pretty heavyweight: the official download and documentation is all based around Eclipse (for HTML/Javascript apps? Seriously?) and a required set of Nokia libraries that I had trouble installing.

But WRT apps are packaged according to the W3C Widget Packaging and Configuration specification -- so it's probably possible to roll your own without the big Nokia WRT package. More research required.

Python, Qt and PySide

I have a lot of Python apps, so I was very happy to see that Python comes already installed on MeeGo. (Perl is also supported; Ruby theoretically is but it's not installed by default.)

Since Qt is the officially blessed user interface, Python-Qt seems like the way to go. Except -- Python in MeeGo has no Qt bindings installed. The package that provides them is called PySide, but depending on where you look you'll either be steered toward a lengthy source compile on the MeeGo device, or a zypper install from someone's personal repository.

None of that is difficult for developers, but you can't expect users to enable experimental repositories or compile from source. Is PySide going to be a standard part of MeeGo by the time it ships to users? Nobody's saying. It makes me leery of putting much energy into Python-Qt development until I hear more.

Python-GTK

A little frustrated with the options so far, I was idly poking around an interactive Python session and typed

>>> import gtk

And it worked! Even though Qt is the supported UI toolkit and nobody talks about GTK, it's installed on MeeGo -- complete with Python bindings. It gave two deprecation warnings, but what's a little deprecation among friends?

A simple test program ran just fine. So I whipped up a starter page for PyTopo, made a .desktop file and icon for it, copied the three files over with scp -- and everything worked great.

There's no standard way yet to make a MeeGo RPM from a Python program, so that will require a little hand-fiddling, but only a little. And sadly, python-webkit-gtk isn't included, so this isn't a good solution for porting web apps.

I'll be keeping an eye on PySide and continuing to experiment with PySide, WRT and C++. But in the meantime, it's great that it's so easy to port all my existing Python-GTK programs to MeeGo.

Tags: meego, programming
[ 10:33 May 07, 2011    More programming | permalink to this entry | comments ]

GIMP: Build-it

Are you a GIMP user or Summer of Code student who's been wanting to get involved, but having trouble building, or a bit intimidated by the build process?

I'll be running a session on IRC to help anyone build GIMP on Linux, as part of the OpenHatch "Build it" project.

The session will take place on #gimp on irc.gimp.org (also known as GimpNet), on Fri, Apr 15, 0300 UTC -- that's Thursday night in the Americas. To convert to your time zone, run this command on your local machine:

$ date -d 'Fri Apr 15 03:00 UTC'
Thu Apr 14 20:00:00 PDT 2011

Or try this link: world time server.

This is a time that's usually fairly quiet on #gimp -- European users don't fret, since it's pretty easy to get help there during more Europe-friendly time zones. I'll hang around for at least two hours; that should be plenty of time to build GIMP and all its prerequisites.

For folks new to IRC, note that irc.gimp.org is its own server -- this is not the #gimp channel on Freenode. You can learn more about IRC on the LinuxChix IRC for Beginners page, or, if you have trouble getting an IRC client configured, try this link for mibbit web chat.

Note: The #gimp IRC channel was recently under attack by trolls, and it's possible that it may not be usable at the time of the session. In that case, I will update this blog page with the name of an alternate channel to use, and any other necessarily details.

Preparation

If you want to get your system set up ahead of time, I've put the instructions needed to build on Ubuntu Lucid and other older Linux distros here: Gimp Building Tips (for Linux). I might be able to offer a little help with building on Macs, but no guarantees.

Mac and Windows users, or people running a very old Linux distro (more than a year old) might want to consider an alternate approach: install Virtualbox or VMware and install Ubuntu "Natty Narwhal" (currently still in beta) in a virtual machine.

Of course, this isn't the only time you can get help with building GIMP. There are folks around on #gimp most of the time who are happy to help with problems. But if you've been meaning to get started and want a good excuse, or you've been holding off on asking for help ... come hang out with us and try it!

Tags: gimp, programming, education
[ 11:50 Apr 13, 2011    More gimp | permalink to this entry | comments ]

Plug Computer + Arduino = Proximity Camera

In the last article, I wrote about how to get a very simple webcam demo running on a plug computer, also known as a Sheevaplug.

Okay, a webcam is sorta cool, but it's still a pretty easy thing to do from any laptop. I wanted to demonstrate some lower-level hardware control.

As I mentioned in the previous article, trying to run hardware directly from a plug computer is an exercise in frustration. So what do you do when you want to drive low-level hardware? Use an Arduino, of course!

Add the Arduino

Happily, the sheeva.with-linux kernels include the FTDI driver you need to talk to an Arduino. So you can plug the Arduino to the plug computer, then let the Arduino read the sensor and write its value to the serial port, which you can read on the plug.

First I tried a simple light sensor from Adafruit, using the circuit from the LadyAda photocell tutorial.

I wrote a very simple Arduino sketch to read the analog output: lightsensor.pde. I'm allergic to Java IDEs, so I compiled the sketch from the commandline using this lightsensor Arduino Makefile. Edit the Makefile to point to wherever you've installed the Arduino software.

Now, on the plug, I needed a Python script to read the numbers coming in on the serial line. I ran apt-get install python-serial, then wrote this script: readsensor.py.

The script loops, reading the sensor and writing the output to an HTML file called arduino.html. Visit that in a browser from your desktop or laptop, and watch it reload and change the number as you wave your hand or a flashlight over the photocell.

Ultrasonic rangefinder for proximity detection

Pretty cool ... if you're extremely geeky and have no life. Otherwise, it's maybe a bit limited. But can we use this Arduino technique to do something useful in combination with the webcam exercise? How about an ultrasonic sonar rangefinder?

The rangefinder comes with a little PC board, and you have to solder wires to it. I wanted to be able to plug and unplug -- the rangefinger also has digital outputs and I may want to experiment with those some day. So I soldered an 8-pin header to the board. (The rangefinder board only has 7 holes, so I had to cut off the 8th pin on the header.)

I ran power and ground wires to 5v and Gnd on the Arduino, and a wire from the rangefinder's analog out to the Arduino's Analog In 2. A little heatshrink keeps the three wires together.

Then I rubber-banded the rangefinder to the front of the webcam, and I was ready to test.

Use a sketch almost identical to the one for the light sensor: rangefinder.pde, and its rangefinder Arduino Makefile. I used pin 2 so I could leave the light sensor plugged in on Pin 1.

Now I ran that same readsensor.py script, paying attention to the numbers being printed out. I found that they generally read around 35-40 when I was sitting right in front of it (camera mounted on my laptop screen), and more like 150-250 when I got out of the way and pointed it across the room.

So I wrote a script, proximity.py, that basically does this:

  if data < 45 :
    if verbose :
      print "Snapping photo!"
    os.system("fswebcam --device /dev/video0 -S 1 output.jpg")

It also rewrites the HTML file to display the value it read from the rangefinder, though that part isn't so important.

Put it all together, and the proximity-sensitive camera snaps a photo any time something is right in front of it; otherwise, it keeps displaying the last photo and doesn't snap a new one. Sample uses: find out who's using your computer when you're away at lunch, or run a security camera at home, or set up a camera to snap shots of the exotic wildlife that's visiting your feeder or research station.

You could substitute an infra-red motion sensor and use it as a motion-operated security camera or bird feeder camera. I ordered one, but got bogged down trying to reverse-engineer the sensor (I should have just ordered one from Adafruit or Sparkfun).

I'm happy to say this all worked pretty well as a demo. But mostly, it's fun to know that I can plug in virtually any sensor and collect any sort of data I want. Adding the Arduino makes the plug computer much more fun and useful.

Tags: hardware, arduino, plug, programming
[ 20:23 Apr 11, 2011    More tech | permalink to this entry | comments ]

A simple plug computer webcam

I was asked to give a talk on plug computers ("sheevaplugs") at a local LUG. I thought at first I didn't have much to say about them, but after writing down an outline I realized that wouldn't be a problem.

But plugs aren't that interesting unless you have something fun to demonstrate. Sure, plugs can run a web server or a file server, but that doesn't make for a very fun demo -- "woo, look, I'm loading a web page!" What's more fun? Hardware.

The first step to running any hardware off a plug computer is to get an upgraded kernel. The kernels that come with these things can't drive any useful external gizmos.

I've often lamented how the folks who build plug computers seem oblivious to the fact that a large part of their potential customer base wants to drive hardware -- temperature and light sensors, weather stations, garage door openers, servos, whatever. By not including drivers for GPIO, 1-wire, video and so forth, they're shutting out anyone who doesn't feel up to building a kernel.

And make no mistake: building a kernel for a sheevaplug is quite a bit harder than building one for your laptop or workstation. Some of the hardware isn't supported by fully open source drivers, and most Linux distros don't offer a cross-compiler that can do the job. I covered some of the issues in my LinuxPlanet article on Cross-compiling Custom Kernels for Plug Computers.

Fortunately, the sheeva.with-linux kernels include a webcam driver. That seemed like a good start for a demo.

A simple webcam demo

My demo plug is running Debian Squeeze, which has a wealth of webcam software available. Although there are lots of packages to stream live video to a web server, they all have a lot of requirements, so I settled for a simple snapshot program, fswebcam.

The command I needed to snap a photo is:

fswebcam --device /dev/video0 -S 1 output.jpeg

The -S 1 skips a frame to account for the fact that my cheap and crappy webcam (a Gearhead Sound FX) tends to return wildly striped green and purple images otherwise.

So I run that in a loop, something like:

while /bin/true; do
  fswebcam --device /dev/video0 -S 1 output.jpeg
  sleep 2
done

Now that I have a continuously updating image, I need to run some sort of web server on the plug. Plugs are perfectly capable of running apache or lighttpd or whatever server you favor. But for this simple demo, I used a tiny Python server script: simpleserver.py.

Then all I have to do is a simple web page that includes <img src="output.jpg"> and point my computer at http://192.168.1.102:8080 to see the image. Either refresh the page to see the image update, or add something like

<meta http-equiv="Refresh" content='2'>

to make it refresh

The next parts of the demo added an Arduino to the mix. But this is already getting long and I'm out of time ... so the second part of this demo will follow in a day or two.

Tags: hardware, plug, programming
[ 21:18 Apr 10, 2011    More tech | permalink to this entry | comments ]

Finding Twitter references to you

Twitter is a bit frustrating when you try to have conversations there. You say something, then an hour later, someone replies to you (by making a tweet that includes your Twitter @handle). If you're away from your computer, or don't happen to be watching it with an eagle eye right then -- that's it, you'll never see it again. Some Twitter programs alert you to @ references even if they're old, but many programs don't.

Wouldn't it be nice if you could be notified regularly if anyone replied to your tweets, or mentioned you?

Happily, you can. The Twitter API is fairly simple; I wrote a Python function a while back to do searches in my Twitter app "twit", based on a code snippet I originally cribbed from Gwibber. But if you take out all the user interface code from twit and use just the simple JSON code, you get a nice short app. The full script is here: twitref, but the essence of it is this:

import sys, simplejson, urllib, urllib2

def get_search_data(query):
    s = simplejson.loads(urllib2.urlopen(
            urllib2.Request("http://search.twitter.com/search.json",
                            urllib.urlencode({"q": query}))).read())
    return s

def json_search(query):
    for data in get_search_data(query)["results"]:
        yield data

if __name__ == "__main__" :
    for searchterm in sys.argv[1:] :
        print "**** Tweets containing", searchterm
        statuses = json_search(searchterm)
        for st in statuses :
            print st['created_at']
            print "<%s> %s" % (st['from_user'], st['text'])
            print ""

You can run twitref @yourname from the commandline now and then. You can even call it as a cron job and mail yourself the output, if you want to make sure you see replies. Of course, you can use it to search for other patterns too, like twitref #vss or twitref #scale9x.

You'll need the simplejson Python library, which most distros offer as a package; on Ubuntu, install python-simplejson.

It's unclear how long any of this will continue to be supported, since Twitter recently announced that they disapprove of third-party apps using their API. Oh, well ... if Twitter stops allowing outside apps, I'm not sure how interested I'll be in continuing to use it.

On the other hand, their original announcement on Google Groups seems to have been removed -- I was going to link to it here and discovered it was no longer there. So maybe Twitter is listening to the outcry and re-thinking their position.

Tags: programming, twitter, python, JSON
[ 09:53 Mar 18, 2011    More programming | permalink to this entry | comments ]

On Linux Planet: Plotting mail logs with CairoPlot

My latest LinuxPlanet article is on plotting pretty graphs from Python with CairoPlot.

Of course, to demonstrate a graphing package I needed some data. So I decided to plot some stats parsed from my Postfix mail log file. We bounce a lot of mail (mostly spam but some false positives from mis-configured email servers) that comes in with bogus HELO addresses. So I thought I'd take a graphical look at the geographical sources of those messages.

The majority were from IPs that weren't identifiable at all -- no reverse DNS info. But after that, the vast majority turned out to be, surprisingly, from .il (Israel) and .br (Brazil).

Surprised me! What fun to get useful and interesting data when I thought I was just looking for samples for an article.

Tags: programming, python, visualization, cairoplot
[ 14:08 Mar 10, 2011    More programming | permalink to this entry | comments ]

Python for (Cartalk) Puzzlers

Last week's Car Talk had a fun puzzler called "Three Pieces of Paper":

Three different numbers are chosen at random, and one is written on each of three slips of paper. The slips are then placed face down on the table. The objective is to choose the slip upon which is written the largest number.

Here are the rules: You can turn over any slip of paper and look at the amount written on it. If for any reason you think this is the largest, you're done; you keep it. Otherwise you discard it and turn over a second slip. Again, if you think this is the one with the biggest number, you keep that one and the game is over. If you don't, you discard that one too.

What are the odds of winning? The obvious answer is one in three, but you can do better than that. After thinking about it a little I figured out the strategy pretty quickly (I won't spoil it here; follow the link above to see the answer). But the question was: how often does the correct strategy give you the answer?

It made for a good "things to think about when trying to fall asleep" insomnia game. And I mostly convinced myself that the answer was 50%. But probability problems are tricky beasts (witness the Monty Hall Problem, which even professional mathematicians got wrong) and I wasn't confident about it. Even after hearing Click and Clack describe the answer on this week's show, asserting that the answer was 50%, I still wanted to prove it to myself.

Why not write a simple program? That way I could run lots of trials and see if the strategy wins 50% of the time.

So here's my silly Python program:

#! /usr/bin/env python

# Cartalk puzzler Feb 2011

import random, time

random.seed()

tot = 0
wins = 0

while True:
    # pick 3 numbers:
    n1 = random.randint(0, 100)
    n2 = random.randint(0, 100)
    n3 = random.randint(0, 100)

    # Always look at but discard the first number.
    # If the second number is greater than the first, stick with it;
    # otherwise choose the third number.
    if n2 > n1 :
        final = n2
    else :
        final = n3

    biggest = max(n1, n2, n3)
    win = (final == biggest)
    tot += 1
    if win :
        wins += 1
    print "%4d %4d %4d %10d %10s %6d/%-6d = %10d%%" % (n1, n2, n3, final,
                                                       str(win),
                                                       wins, tot,
                                                       int(wins*100/tot))
    if tot % 1000 == 0:
        print "(%d ...)" % tot
        time.sleep(1)

It chooses numbers between 0 and 100, for no particular reason; I could randomize that, but it wouldn't matter to the result. I made it print out all the outcomes, but pause for a second after every thousand trials ... otherwise the text scrolls too fast to read.

And indeed, the answer converges very rapidly to 50%. Hurray!

After I wrote the script, I checked Car Talk's website. They have a good breakdown of all the possible outcomes and how they map to a probability. Of course, I could have checked that first, before writing the program. But I was thinking about this in the car while driving home, with no access to the web ... and besides, isn't it always more fun to prove something to yourself than to take someone else's word for it?

Tags: python, programming, probability, puzzles
[ 20:17 Feb 22, 2011    More programming | permalink to this entry | comments ]

New GIMP Arrow Designer

While writing a blog post on GIMP's confusing Auto button (to be posted soon), I needed some arrows, and discovered a bug in my Arrow Designer script when making arrows that are mostly vertical.

So I fixed it. You can get the new Arrow Designer 0.5 on my GIMP Arrow Designer page.

It's purely a coincidence that I discovered this a week before SCALE, where I'll be speaking on Writing GIMP Scripts and Plug-Ins. Arrow Designer is one of my showpieces for making interactive plug-ins with GIMP-Python, so I'm glad I noticed the bug when I did.

Tags: gimp, programming, python
[ 20:28 Feb 18, 2011    More gimp | permalink to this entry | comments ]

Feedme 0.7

I've been enjoying my Android tablet e-reader for a couple of months now ... and it's made me realize some of the shortcomings in FeedMe. So of course I've been making changes along the way -- quite a few of them, from handling multiple output file types (html, plucker, ePub or FictionBook) to smarter handling of start, end and skip patterns to a different format of the output directory.

It's been fairly solid for a few weeks now, so it's time to release ... FeedMe 0.7.

Tags: RSS, python, programming, palm
[ 21:32 Jan 31, 2011    More programming | permalink to this entry | comments ]

Getting rid of extra whitespace from Eclipse

Eclipse has been driving me batty with all the extra spaces it adds everywhere -- blank lines all have indents on them, and lots of code lines have extra spaces randomly tacked on to the end. I sure wouldn't want to share files like that with coworkers or post them as open source.

I found lots of suggestions on the web for eliminating extra whitespace, and several places to configure this within Eclipse, but most of them don't do anything. Here's the one that actually worked:

Window->Preferences
Jave->Editor->Save Actions
Enable Perform the selected actions on save.
Enable Additional actions.
Click Configure.
In the Code Organizing tab., enable Remove trailing whitespace for All lines.
Review all the other options there, since it will all happen automatically whenever you save -- make sure there isn't anything there you don't want.
Dismiss the Configure window.
Review the other options under Save Actions, since these will also happen automatically now.
Don't forget to click Apply in the Save Actions preference page.

Whew! There are other places to set this, in various Code style and Cleanup options, but all all the others require taking some action periodically, like Source->Clean up...

By the way, while you're changing whitespace preferences, you may also want the Insert spaces for tabs preference under General->Editors->Text Editors.

An easy way to check whether you've succeeded in exorcising the spaces -- eclipse doesn't show them all, even when you tell it to -- is to :set hlsearch in vim, then search for a space. (Here are some other ways to show spaces in vim.) In emacs, you can M-x set-variable show-trailing-whitespace to true, but that doesn't show spaces on blank lines; for that you might want whitespace.el or similar packages.

Tags: eclipse, android, programming
[ 14:42 Jan 25, 2011    More programming | permalink to this entry | comments ]

X Terminal Colors (and dark and light backgrounds)

At work, I'm testing some web programming on a server where we use a shared account -- everybody logs in as the same user. That wouldn't be a problem, except nearly all Linuxes are set up to use colors in programs like ls and vim that are only readable against a dark background. I prefer a light background (not white) for my terminal windows.

How, then, can I set things up so that both dark- and light-backgrounded people can use the account? I could set up a script that would set up a different set of aliases and configuration files, like when I changed my vim colors. Better, I could fix all of them at once by changing my terminal's idea of colors -- so when the remote machine thinks it's feeding me a light color, I see a dark one.

I use xterm, which has an easy way of setting colors: it has a list of 16 colors defined in X resources. So I can change them in ~/.Xdefaults.

That's all very well. But first I needed a way of seeing the existing colors, so I knew what needed changing, and of testing my changes.

Script to show all terminal colors

I thought I remembered once seeing a program to display terminal colors, but now that I needed one, I couldn't find it. Surely it should be trivial to write. Just find the escape sequences and write a script to substitute 0 through 15, right?

Except finding the escape sequences turned out to be harder than I expected. Sure, I found them -- lots of them, pages that conflicted with each other, most giving sequences that didn't do anything visible in my xterm.

Eventually I used script to capture output from a vim session to see what it used. It used <ESC>[38;5;Nm to set color N, and <ESC>[m to reset to the default color. This more or less agreed Wikipedia's ANSI escape code page, which says <ESC>[38;5; does "Set xterm-256 text coloor" with a note "Dubious - discuss". The discussion says this isn't very standard. That page also mentions the simpler sequence <ESC>[0;Nm to set the first 8 colors.

Okay, so why not write a script that shows both? Like this:

#! /usr/bin/env python

# Display the colors available in a terminal.

print "16-color mode:"
for color in range(0, 16) :
    for i in range(0, 3) :
        print "\033[0;%sm%02s\033[m" % (str(color + 30), str(color)),
    print

# Programs like ls and vim use the first 16 colors of the 256-color palette.
print "256-color mode:"
for color in range(0, 256) :
    for i in range(0, 3) :
        print "\033[38;5;%sm%03s\033[m" % (str(color), str(color)),
    print

Voilà! That shows the 8 colors I needed to see what vim and ls were doing, plus a lovely rainbow of other possible colors in case I ever want to do any serious ASCII graphics in my terminal.

Changing the X resources

The next step was to change the X resources. I started by looking for where the current resources were set, and found them in /etc/X11/app-defaults/XTerm-color:

$ grep color /etc/X11/app-defaults/XTerm-color
irrelevant stuff snipped
*VT100*color0: black
*VT100*color1: red3
*VT100*color2: green3
*VT100*color3: yellow3
*VT100*color4: blue2
*VT100*color5: magenta3
*VT100*color6: cyan3
*VT100*color7: gray90
*VT100*color8: gray50
*VT100*color9: red
*VT100*color10: green
*VT100*color11: yellow
*VT100*color12: rgb:5c/5c/ff
*VT100*color13: magenta
*VT100*color14: cyan
*VT100*color15: white
! Disclaimer: there are no standard colors used in terminal emulation.
! The choice for color4 and color12 is a tradeoff between contrast, depending
! on whether they are used for text or backgrounds.  Note that either color4 or
! color12 would be used for text, while only color4 would be used for a
! Originally color4/color12 were set to the names blue3/blue
!*VT100*color4: blue3
!*VT100*color12: blue
!*VT100*color4: DodgerBlue1
!*VT100*color12: SteelBlue1

So all I needed to do was take the ones that don't show up well -- yellow, green and so forth -- and change them to colors that work better, choosing from the color names in /etc/X11/rgb.txt or my own RGB values. So I added lines like this to my ~/.Xdefaults:

!! color2 was green3
*VT100*color2: green4
!! color8 was gray50
*VT100*color8: gray30
!! color10 was green
*VT100*color10: rgb:00/aa/00
!! color11 was yellow
*VT100*color11: dark orange
!! color14 was cyan
*VT100*color14: dark cyan

... and so on.

Now I can share accounts, and I no longer have to curse at those default ls and vim settings!

Update: Tip from Mikachu: ctlseqs.txt is an excellent reference on terminal control sequences.

Tags: color, X11, linux, programming, python, tips
[ 09:56 Jan 18, 2011    More linux | permalink to this entry | comments ]

Fontasia v 0.5

I had a nice relaxing holiday season. A little too relaxing -- I didn't get much hacking done, and spent more time fighting with things that didn't work than making progress fixing things.

But I did spend quite a bit of time with my laptop, currently running Arch Linux, trying to get the fonts to work as well as they do in Ubuntu. I don't have a definite solution yet to my Arch font issues, but all the fiddling with fonts did lead me to realize that I needed an easier way to preview specific fonts in bold.

So I added Bold and Italic buttons to fontasia, and called it Fontasia 0.5. I'm finding it quite handy for previewing all my fixed-width fonts while trying to find one emacs can display.

Tags: fonts, programming, python
[ 22:00 Jan 04, 2011    More programming | permalink to this entry | comments ]

Developing for Android

I wrote yesterday about my quest for an app for reading news feeds and other timely information from the web. And how existing ebook readers didn't meet that need. That meant I would have to write something.

Android development is done in Java, using Eclipse as an IDE. Let me just state up front that (a) I dislike Java (and have forgotten most of what I once knew about it) and (b) I hate IDEs -- they make you use their crippled editor instead of your own, they control what you can put where on the screen, and they're always popping up windows that get in your way.

Okay, not an auspicious beginning. But let's try to be open-minded, follow the instructions and see what happens.

I installed Eclipse from eclipse.org after being advised that the version on Ubuntu is out of date. Then I installed all the various Android plug-ins and SDKs and set them up (there is no single page that lists all the steps, so I did a lot of googling). It took maybe an hour or so to get it all installed to the point where it could produce its "Hello world".

And then ... wow! Hello world worked right off the bat, and the emulator worked for basic testing. Hmm, okay ... how about if we use HTML as a format ... is there a standard HTML display component? Sure enough -- I added a WebView to my app and right away I had a working HTML reader. Okay, how about a row of buttons and a status bar on top? Sure, no problem.

The standard Android online docs aren't great -- they're a wonderful example of how to write seemingly comprehensive documentation that somehow manages to tell you nothing of what you actually need to know. But that's not as bad as it sounds, because there are lots of forums and tutorials to fill in the gaps. Stack Overflow is particularly good for Android tips.

And yes, I did some swearing at Eclipse and spent too much time googling how to disable features, like the "Content Assist" that periodically freezes the whole UI for a minute or so in the middle of your typing a line of code, while it thinks about some unhelpful and irrelevant hints to offer you in a popup. Turn it off in the prefs under Java/Editor. (Eclipse's actual useful hints, like the ones you get when you hover over something that's red because of an error, will still work. I found them very helpful.)

More specifically: Java/Editor/Content Assist/Hovers, and turn off Combined Hover and maybe anything else that happens without a modifier key. You probably also want to turn off Enable Auto Activation under Java/Editor/Content Assist. And possibly others -- I kept turning things off until the popups and delays went away, and I haven't found anything explaining how all these parameters relate.

Okay, so there were snags, and it's frustrating how there are almost no open source apps for this open source OS. (Yes, my app will be.) But here's the thing: in about 4 days, starting from nothing, I had a little RSS reader that did everything I needed. I've been adding features since then. Android doesn't have a reasonable battery status monitor? Fine, my reader can show battery percentage in the status bar. Android doesn't dim the screen enough? Fine, I can dim it further inside the application (an idea borrowed from Aldiko).

After less than a week of work I have an RSS reader that's better than my Palms running Plucker ever were. And that says a lot about the ease of the Android programming environment. I'm impressed!

Update: The source, an apk, and a brief discussion of how I use my feed reader are now here: FeedViewer.

Tags: android, programming
[ 15:26 Dec 21, 2010    More programming | permalink to this entry | comments ]

Android/Eclipse Spellchecker is a bit confused

I've been doing some Android development, using the standard Eclipse development tools. A few days ago, I pasted some code that included a comment about different Android versions, and got a surprise:

What do you think -- should I change all the "Android" references to "Undried"?

Tags: humor, android, eclipse, programming
[ 10:09 Dec 07, 2010    More humor | permalink to this entry | comments ]

Escaping HTML characters in Emacs (and how to do replaces in elisp)

Last week I found myself writing another article that includes code snippets in HTML.

So what, you ask? The problem is, when you're writing articles in HTML, every time you include a code snippet inside a <pre> tag you invariably forget that special characters like < > & have special meanings in HTML, and must be escaped. Every < has to change to &lt;, and so forth, after you paste the code.

In vi/vim, replacing characters is straightforward. But I usually write longer articles in emacs, for various unimportant reasons, and although emacs has global replace, it only works from wherever you are now (called "point" in emacs lingo) to the end of the file. So if you're trying to fix something you pasted in the middle of the article, you can't do it with normal emacs replace.

Surely this is a wheel that has already been re-invented a thousand times, I thought! But googling and asking emacs experts turned up nothing. Looks like I'd have to write it.

And that turned out to be more difficult than I expected, for the same reason: emacs replace-string works the same way from a program as it does interactively, and replaces from point to the end of the file, and there's no way to restrict it to a more limited range.

Several helpful people on #emacs chimed in with ideas, but most of them didn't pan out. But ggole knew a way to do it that was both clean and reliable (thanks!).

Here's the elisp function I ended up with. It uses save-excursion to put the cursor back where it started before you ran the function, narrow-to-region to make replace-string work only on the region, and save-restriction get rid of that narrow-to-region after we're done. Nice!

(defun unhtml (start end)
  (interactive "r")
  (save-excursion
    (save-restriction
      (narrow-to-region start end)
      (goto-char (point-min))
      (replace-string "&" "&")
      (goto-char (point-min))
      (replace-string "<" "&lt;")
      (goto-char (point-min))
      (replace-string ">" "&gt;")
      )))

And yes, I used it just now on that elisp snippet.

Tags: emacs, editors, programming
[ 19:08 Dec 01, 2010    More linux/editors | permalink to this entry | comments ]

Garmin GPX timestamp bizarreness

My last entry mentioned some work I'd done to one of my mapping programs, Ellie, to gather statistics from the track logs I get from my Garmin GPS.

In the course of working on Ellie, I discovered something phenomenally silly about the GPX files from my Garmin Vista CX, as uploaded with gpsbabel.

Track log points, quite reasonably, have time stamps in "Zulu time" (essentially the same as GMT, give or take some fraction of a second). They look like this:

<trkpt lat="35.289519913" lon="-115.227057561">
  <ele>1441.634277</ele>
  <time>2010-10-14T17:51:35Z</time>
</trkpt>

But the waypoints you set for specific points of interest, even if they're in the same GPX file, have timestamps that have no time zone at all. They look like this:

<wpt lat="35.334813371" lon="-115.178730609">
  <ele>1489.917480</ele>
  <name>001</name>
  <cmt>14-OCT-10 11:18:51AM</cmt>
  <desc>14-OCT-10 11:18:51AM</desc>
  <sym>Flag, Blue</sym>
</wpt>

Notice the waypoint's time isn't actually in a time field -- it's duplicated in two fields, cmt (comment) and desc (description). So it's not really intended to be a time stamp -- but it sure would be handy if you could use it as one.

You might be able to correlate waypoints with track points by comparing coordinates ... unless you spent more than an hour hanging around a particular location, or came back several hours later (perhaps starting and ending your hike at the same place). In that case ... you'd better know what the local time zone was, including daylight savings time.

What a silly omission, considering that the GPS obviously already knows the Zulu time and could just as easily use that!

Tags: mapping, programming, timezones
[ 21:09 Nov 03, 2010    More mapping | permalink to this entry | comments ]

New versions of mapping programs: Pytopo and Ellie

On our recent Mojave trip, as usual I spent some of the evenings reviewing maps and track logs from some of the neat places we explored.

There isn't really any existing open source program for offline mapping, something that works even when you don't have a network. So long ago, I wrote Pytopo, a little program that can take map tiles from a Windows program called Topo! (or tiles you generate yourself somehow) and let you navigate around in that map.

But in the last few years, a wonderful new source of map tiles has become available: OpenStreetMap. On my last desert trip, I whipped up some code to show OSM tiles, but a lot of the code was hacky and empirical because I couldn't find any documentation for details like the tile naming scheme.

Well, that's changed. Upon returning to civilization I discovered there's now a wonderful page explaining the Slippy map tilenames very clearly, with sample code and everything. And that was the missing piece -- from there, all the things I'd been missing in pytopo came together, and now it's a useful self-contained mapping script that can download its own tiles, and cache them so that when you lose net access, your maps don't disappear along with everything else.

Pytopo can show GPS track logs and waypoints, so you can see where you went as well as where you might want to go, and whether that road off to the right actually would have connected with where you thought you were heading.

It's all updated in svn and on the Pytopo page.

Ellie

Most of the pytopo work came after returning from the desert, when I was able to google and find that OSM tile naming page. But while still out there and with no access to the web, I wanted to review the track logs from some of our hikes and see how much climbing we'd done. I have a simple package for plotting elevation from track logs, called Ellie. But when I ran it, I discovered that I'd never gotten around to installing the pylab Python plotting package (say that three times fast!) on this laptop.

No hope of installing the package without a net ... so instead, I tweaked Ellie so that so that without pylab you can still print out statistics like total climb. While I was at it I added total distance, time spent moving and time spent stopped. Not a big deal, but it gave me the numbers I wanted. It's available as ellie 0.3.

Tags: mapping, programming, python
[ 18:24 Oct 30, 2010    More mapping | permalink to this entry | comments ]

Snakes on a Couch! Using Python with CouchDB

Part II of my CouchDB tutorial is out at Linux Planet. In it, I use Python and CouchDB to write a simple application that keeps track of which restaurants you've been to recently, and to suggest new places to eat where you haven't been.

Snakes on a Couch, Part 2: Where do you want to eat?

Tags: writing, python, programming, database, couchdb
[ 20:00 Oct 15, 2010    More writing | permalink to this entry | comments ]

GIMP Wallpaper script improvements

Dave was using some old vacation photos to test filesystem performance, and that made me realize that I had beautiful photos from the same trip that I hadn't yet turned into desktop backgrounds.

Sometimes I think that my GIMP Wallpaper script is the most useful of the GIMP plug-ins I've written. It's such a simple thing ... but I bet I use it more than any of my other plug-ins, and since I normally make backgrounds for at least two resolutions (my 1680x1050 desktop and my 1366x768 laptop), it certainly saves me a lot of time and hassle.

But an hour into my background-making, I started to have nagging doubts. I wasn't renaming these images, just keeping the original filenames from the camera, like pict0828.jpg. What if if some of these were overwriting images of the same name? The one thing my script doesn't do is check for that, and gimp_file_save doesn't pop up any warnings. I've always meant to add a check for it.

Of course, once the doubts started, I had to stop generating backgrounds and start generating code. And I'm happy with the result: wallpaper-0.4.py warns and won't let you save over an old background image, but keeps all the logic in one dialog rather than popping up extra warnings.

Now I can generate backgrounds without worrying that I'm stomping on earlier ones.

Tags: gimp, programming, gtk
[ 21:25 Sep 26, 2010    More gimp | permalink to this entry | comments ]

Snakes on a Couch! Using Python with CouchDB

I've been learning CouchDB, the hot NoSQL database, as part of my new job. It's interesting -- a very different mindset compared to classic databases like MySQL.

There's a fairly good Python package for it, python-couchdb ... but the documentation is somewhat incomplete and there's very little else written about it, and virtually no sample code to steal.

That makes it a perfect topic for a Linux Planet tutorial! So here it is, Part 1:

Snakes on a Couch! Using Python with CouchDB.

I have a rather fun application for the database I introduce in the article, but you'll have to wait until Part 2, two weeks from now, to see the details.

Tags: writing, python, programming, database, couchdb
[ 10:55 Sep 23, 2010    More writing | permalink to this entry | comments ]

Fontasia v 0.3

A couple of weeks ago I posted about fontasia, my new font-chooser app. It's gone through a couple of revisions since then, and Mikael Magnusson contributed several excellent improvements, like being able to render each font in the font list.

I'd been holding off on posting 0.3, hoping to have time to do something about the font buttons -- they really need to be smaller, so there's space for more categories. But between a new job and several other commitments, I haven't had time to implement that. And the fancy font list is so cool it really ought to be shared.

So here it is: fontasia 0.3.

Tags: fonts, programming, python
[ 09:31 Sep 03, 2010    More programming | permalink to this entry | comments ]

Fontasia: View and categorize your fonts

We were talking about fonts again on IRC, and how there really isn't any decent font viewer on Linux that lets you group fonts into categories.

Any time you need to choose a font -- perhaps you know you need one that's fixed-width, script, cartoony, western-themed -- you have to go through your entire font list, clicking one by one on hundreds of fonts and saving the relevant ones somehow so you can compare them later. If you have a lot of fonts installed, it can take an hour or more to choose the right font for a project.

There's a program called fontypython that does some font categorization, but it's hard to use: it doesn't operate on your installed fonts, only on fonts you copy into a special directory. I never quite understood that; I want to categorize the fonts I can actually use on my system.

I've been wanting to write a font categorizer for a long time, but I always trip up on finding documentation on getting Python to render fonts. But this time, when I googled, I found jan bodnar's ZetCode Pango tutorial, which gave me all I needed and I was off and running.

Fontasia is initially a font viewer. It shows all your fonts in a list on the left, with a preview on the right. But it also lets you add categories: just type the category name in the box and click Add category and a button for that category will appear, with the current font added to it. A font can be in multiple categories.

Once you've categorized your fonts, a menu at the top of the window lets you show just the fonts in a particular category. So if you're working on a project that needs a Western-style font, show that category and you'll see only relevant fonts.

You can also show only the fonts you've categorized -- that way you can exclude fonts you never use -- I don't speak Tamil or Urdu so I don't really need to see those fonts when I'm choosing a font. Or you can show only the uncategorized fonts: this is useful when you add some new fonts to your system and need to go through them and categorize them.

I'm excited about fontasia. It's only a few days old and already used it several times for real-world font selection problems.

If you want to try it, it's here: Fontasia: View and categorize fonts.

Tags: fonts, programming, python
[ 11:20 Aug 17, 2010    More programming | permalink to this entry | comments ]

Writing scripts for your Canon camera with CHDK

On Linux Planet yesterday: an article on how to write scripts for chdk, the Canon Hack Development Kit -- Part 3 in my series on CHDK.

Time-Lapse Photography with your Inexpensive Canon Camera (CHDK p. 3)

I found that CHDK scripting wasn't quite as good as I'd hoped -- some of the functions, especially the aperture and shutter setting, were quite flaky on my A540 so it really didn't work to write a bracketing script. But it's fantastic for simple tasks like time-lapse photography, or taking a series of shots like the Grass Roots Mapping folk do.

If you're at OSCON and you like scripting and photos, check out my session on Thursday afternoon at 4:30: Writing GIMP Plug-ins and Scripts, in which I'll walk through several GIMP scripts in Python and Script-Fu and show some little-known tricks you can do with Python plug-ins.

Tags: photo, writing, programming, mapping, conferences, oscon, speaking
[ 09:31 Jul 21, 2010    More photo | permalink to this entry | comments ]

Interactive arrow design in GIMP

How many times have you wanted an easy way of making arrows in GIMP?

I need arrows all the time, for screenshots and diagrams. And there really isn't any easy way to do that in GIMP. There's a script-fu for making arrows in the Plug-in registry, but it's fiddly and always takes quite a few iterations to get it right. More often, I use a collection of arrow brushes I downloaded from somewhere -- I can't remember exactly where I got my collection, but there are lots of options if you google gimp arrow brushes -- then use the free rotate tool to rotate the arrow in the right direction.

The topic of arrows came up again on #gimp yesterday, and Alexia Death mentioned her script-fu in GIMP Fx Foundary that "abuses the selection" to make shapes, like stars and polygons. She suggested that it would be easy to make arrows the same way, using the current selection as a guide to where the arrow should go.

And that got me thinking about Joao Bueno's neat Python plug-in demo that watches the size of the selection and updates a dialog every time the selection changes. Why not write an interactive Python script that monitors the selection and lets you change the arrow by changing the size of the selection, while fine-tuning the shape and size of the arrowhead interactively via a dialog?

Of course I had to write it. And it works great! I wish I'd written this five years ago.

This will also make a great demo for my OSCON 2010 talk on Writing GIMP Scripts and Plug-ins, Thursday July 22. I wish I'd had it for Libre Graphics Meeting last month.

It's here: GIMP Arrow Designer.

Tags: gimp, programming, python, oscon2010
[ 10:25 Jul 10, 2010    More gimp | permalink to this entry | comments ]

Tee in Python

I needed a way to send the output of a Python program to two places simultaneously: print it on-screen, and save it to a file.

Normally I'd use the Linux command tee for that: prog | tee prog.out saves a copy of the output to the file prog.out as well as printing it. That worked fine until I added something that needed to prompt the user for an answer. That doesn't work when you're piping through tee: the output gets buffered and doesn't show up when you need it to, even if you try to flush() it explicitly.

I investigated shell-based solutions: the output I need is on sterr, while Python's raw_input() user prompt uses stdout, so if I could get the shell to send stderr through tee without stdout, that would have worked. My preferred shell, tcsh, can't do this at all, but bash supposedly can. But the best examples I could find on the web, like the arcane prog 2>&1 >&3 3>&- | tee prog.out 3>&- didn't work.

I considered using /dev/tty or opening a pty, but those calls only work on Linux and Unix and the program is otherwise cross-platform.

What I really wanted was a class that acts like a standard Python file object, but when you write to it it writes to two places: the log file and stderr.

I found an example of someone trying to write a Python tee class, but it didn't work: it worked for write() but not for print >>

I am greatly indebted to KirkMcDonald of #python for finding the problem. In the Python source implementing >>, PyFile_WriteObject (line 2447) checks the object's type, and if it's subclassed from the built-in file object, it writes directly to the object's fd instead of calling write().

The solution is to use composition rather than inheritance. Don't make your file-like class inherit from file, but instead include a file object inside it. Like this:

import sys

class tee :
    def __init__(self, _fd1, _fd2) :
        self.fd1 = _fd1
        self.fd2 = _fd2

    def __del__(self) :
        if self.fd1 != sys.stdout and self.fd1 != sys.stderr :
            self.fd1.close()
        if self.fd2 != sys.stdout and self.fd2 != sys.stderr :
            self.fd2.close()

    def write(self, text) :
        self.fd1.write(text)
        self.fd2.write(text)

    def flush(self) :
        self.fd1.flush()
        self.fd2.flush()

stderrsav = sys.stderr
outputlog = open(logfilename, "w")
sys.stderr = tee(stderrsav, outputlog)

And it works! print >>sys.stderr, "Hello, world" now goes to the file as well as stderr, and raw_input still works to prompt the user for input.

In general, I'm told, it's not safe to inherit from Python's built-in objects like file, because they tend to make assumptions instead of making virtual calls to your overloaded methods. What happened here will happen for other objects too. So use composition instead when extending Python's built-in types.

Tags: programming, python
[ 08:48 Apr 16, 2010    More programming | permalink to this entry | comments ]

Configuring git colors

I spent a morning wrestling with git after writing a minor GIMP fix that I wanted to check in. Deceptively simple ideas, like "Check the git log to see the expected format of check-in messages", turned out to be easier said than done.

Part of the problem was git's default colors: colors calculated to be invisible to anyone using a terminal with dark text on a light background. And that sent me down the perilous path of git configuration.

git-config does have a manual page. But it lacks detail: you can't get from there to knowing what to change so that the first line of commits in git log doesn't show up yellow.

But that's okay, thought I: all I need to do is list the default settings, then change anything that's a light color like yellow to a darker color. Easy, right?

Well, no. It turns out there's no way to get the default settings -- because they aren't part of git's config; they're hardwired into the C code.

But you can find most of them with a seach for GIT_COLOR in the source. The most useful lines are these the ones in diff.c, builtin-branch.c and wt-status.c.

gitconfig

The next step is to translate those C lines to git preferences, something you can put in a .gitconfig. Here's a list of all the colors mentioned in the man page, and their default values -- I used "normal" for grep and interactive where I wasn't sure of the defaults.

[color "diff"]
	plain = normal
	meta = bold
	frag = cyan
	old = red
	new = green
	commit = yellow
	whitespace = normal red
[color "branch"]
	current = green
	local = normal
	remote = red
	plain = normal
[color "status"]
	header = normal
	added = red
	updated = green
	changed = red
	untracked = red
	nobranch = red
[color "grep"]
	match = normal
[color "interactive"]
	prompt = normal
	header = normal
	help = normal
	error = normal

The syntax and colors are fairly clearly explained in the manual: allowable colors are normal, black, red, green, yellow, blue, magenta, cyan and white. After the foreground color, you can optionally list a background color. You can also list an attribute, chosen from bold, dim, ul, blink and reverse -- only one at a time, no combining of attributes.

So if you really wanted to, you could say something like

[color "status"]
	header = normal blink
	added = magenta yellow
	updated = green reverse
	changed = red bold
	untracked = blue white
	nobranch = red white bold

Minimal changes for light backgrounds

What's the minimum you need to get everything readable? On the light grey background I use, I needed to change the yellow, cyan and green entries:

[color "diff"]
	frag = cyan
	new = green
	commit = yellow
[color "branch"]
	current = green
[color "status"]
	updated = green

Disclaimer: I haven't tested all these settings -- because I haven't yet figured out where all of them apply. That's another area where the manual is a bit short on detail ...

Tags: git, programming, color
[ 22:26 Feb 02, 2010    More programming | permalink to this entry | comments ]

On Linux Planet: a simple Poker game in Python-Qt

I've written in the past about Python GUI programming using the GTK and Tk toolkits, and several KDE fans felt that I was slighting the much nicer looking Qt.

So my latest article on Linux Planet, Make Pretty GUI Apps Fast with Python-Qt, shows how to develop a little poker game using the python-qt toolkit.

I didn't want to dwell on it in the article (and didn't have space anyway), but pyqt turned out to be a bit of a pain. There's no official documentation -- or at least nothing that's obviously official -- and a lot of the examples on google are out of date because of API changes. None of the tutorial examples explain much, and they never demonstrate the practical features I'd want to do in a real app. It was surprisingly hard to come up with an application idea that worked well, looked good and was still easy to explain.

And don't get me started on this whole "Slots and signals are revolutionarily different even though they look just like the callbacks every other toolkit has used for the last three decades" meme. I'm sure there is a subtle technical difference -- but if there's a difference that matters to the average UI programmer, their documentation sure doesn't make it clear.

All that aside, PyQt (and Qt in general) does produce very pretty apps and is worth trying for that reason.

The suit images in the article are adapted from some suits I found on Wikimedia Commons (the "Naipe" set). I wanted them to look more 3-dimensional, so I applied my blobipy GIMP script as well as scaling and resizing them. I really liked those shiny-looking Tango heart and spade emblems (also on the Wikimedia Commons page) but I couldn't find a diamond or club to match.

The poker program I wrote has menus and a second round of dealing, where you can mark off the cards you want to keep. I couldn't fit all that in a 700-word article, but the complete program is available here: qpoker.py or you can get it in a tarball along with the suit images at qpoker.tar.gz.

Tags: programming, writing, qt, python-qt
[ 09:53 Jan 28, 2010    More programming | permalink to this entry | comments ]

Displaying images from Javascript file inputs

(despite Firefox's attempts to prevent that)

My Linux Planet article last week was on printing pretty calendars. But I hit one bug in Photo Calendar. It had a HTML file chooser for picking an image ... and when I chose an image and clicked Select to use it. it got the pathname wrong every time.

I poked into the code (Photo Calendar's code turned out to be exceptionally clean and well documented) and found that it was expecting to get the pathname from the file input element's value attribute. But input.File.value was just returning the filename, foo.jpg, instead of the full pathname, /home/user/Images/yosemite/foo.jpg. So when the app tried to make it into a file:/// URL, it ended up pointing to the wrong place.

It turned out the cause was a security change in Firefox 3. The issue: it's considered a security hole to expose full pathnames on your computer to Javascript code coming from someone else's server. The Javascript could give bad guys access to information about the directory structures on your disk. That's a perfectly reasonable concern, and it makes sense to consider it as a security hole.

The problem is that this happens even when you're running a local app on your local disk. Programs written in any other language and toolkit -- a Python program using pygtk, say, or a C++ Qt program -- have access to the directories on your disk, but you can't use Javascript inside Firefox to do the same thing. The only ways to make an exception seems to be an elaborate procedure requiring the user to change settings in about:config. Not too helpful.

Perhaps this is even reasonable, given how common cross-site scripting bugs have been in browsers lately -- maybe running a local script really is a security risk if you have other tabs active. But it leaves us with the problem of what to do about apps that need to do things like choose a local image file, then display it.

And it turns out there is: a data URL. Take the entire contents of the file (ouch) and create a URL out of those contents, then set the src attribute of the image to that.

Of course, that makes for a long, horrifying, unreadable URL -- but the user never has to see that part. I suspect it's also horribly memory intensive -- the image has to be loaded into memory anyway, to display it, but is Firefox also translating all of that to a URL-legal syntax? Obviously, any real app using this technique had better keep an eye on memory consumption. But meanwhile, it fixes Photo Calendar's file button.

Here's what the code looks like:

  img = document.getElementById("pic");
  fileinput = document.input.File;
  if (img && fileinput)
    img.src = fileinput.files[0].getAsDataURL();

Here's a working minimal demo of using getAsDataURL() with a file input.

Tags: javascript, web, programming
[ 13:57 Jan 17, 2010    More programming | permalink to this entry | comments ]

Python-GTK regression: How to catch mouse button release

We just had the second earthquake in two days, and I was chatting with someone about past earthquakes and wanted to measure the distance to some local landmarks. So I fired up PyTopo as the easiest way to do that. Click on one point, click on a second point and it prints distance and bearing from the first point to the second.

Except it didn't. In fact, clicks weren't working at all. And although I have hacked a bit on parts of pytopo (the most recent project was trying to get scaling working properly in tiles imported from OpenStreetMap), the click handling isn't something I've touched in quite a while.

It turned out that there's a regression in PyGTK: mouse button release events now need you to set an event mask for button presses as well as button releases. You need both, for some reason. So you now need code that looks like this:

drawing_area.connect("button-release-event", button_event)
drawing_area.set_events(gtk.gdk.EXPOSURE_MASK |
                        # next line wasn't needed before:
                        gtk.gdk.BUTTON_PRESS_MASK |
                        gtk.gdk.BUTTON_RELEASE_MASK )

An easy fix ... once you find it.

I filed bug 606453 to see whether the regression was intentional.

I've checked in the fix to the PyTopo svn repository on Google Code. It's so nice having a public source code repository like that! I'm planning to move Pho to Google Code soon.

Tags: programming, python, pygtk, mapping
[ 13:20 Jan 08, 2010    More programming | permalink to this entry | comments ]

Debug logging in Javascript under Firefox

While debugging Javascript, I've occasionally come across references to a useful function called console.log. Supposedly you can log errors with a line like:

  console.log(""key press, char code " + e.charCode);

Since the only native way of logging debug statements in Javascript is with a pop-up alert() box, having a less obtrusive way to print is something any JS programmer could use.

The catch? It didn't do anything -- except print console is not defined.

Today a friend was crowing about how wonderful Javascript debugging was in Google's Chrome browser -- because it has functions like console.log.

After some searching and poking around, we determined that Firefox also has console.log -- it's just well hidden and a bit hard to get going.

First, you need the Firebug extension. If you're developing Javascript, you probably already have it. If not, you need it.

Run Firebug and click to the Console tab. Now click on the tiny arrow that shows up at the right edge of that tab, as shown. Turns out there's a whole menu of options under there -- one of which is Enabled.

But wait, that's not all. In my case, the console was already Enabled according to the menu. To get the console working, I had to

1. Disable the console
2. Re-enable it
3. Shift-reload the page being debugged

My friend also said that if she didn't enable the console in Firebug, then her script died when she called console.log. That didn't happen for me -- all that happened was that I got error messages in the error console (the one accessed from Firefox's Tools menu, different from the Firebug console). But it's a good idea to check for its existence if you're going to use debugging statements in your code. Like this:

  if (typeof console != "undefined") {
    console.log( "key press, char code " + e.charCode
                + ", key code " + e.keyCode
                + ", " + e.ctrlKey + ", " + e.altKey
		+ ", " + e.metaKey );
  }

Here are some more things you can do with Firebug's console.

Tags: firefox, javascript, programming, tips
[ 15:41 Nov 28, 2009    More tech/web | permalink to this entry | comments ]

Character Sets and Encodings in Linux, part 2

Continuing the discussion of those funny characters you sometimes see in email or on web pages, today's Linux Planet article discusses how to convert and handle encoding errors, using Python or the command-line tool recode:

Mastering Characters Sets in Linux (Weird Characters, part 2).

Tags: writing, linux, unicode, i18n, charsets, ascii, programming, python
[ 14:06 Nov 25, 2009    More writing | permalink to this entry | comments ]

Building a Py-Webkit-GTK presentation tool

I almost always write my presentation slides using HTML. Usually I use Firefox to present them; it's the browser I normally run, so I know it's installd and the slides all work there. But there are several disadvantages to using Firefox:

• In fullscreen mode, it has a small "minimized urlbar" at the top of the screen that I've never figured out to banish -- not only is it visible to users, but it also messes up the geometry of the slides (they have to be 762 pixels high rather than 768);
• It's very heavyweight, bad when using a mini laptop or netbook;
• Any personal browsing preferences, like no-animation, flashblock or noscript, apply to slides too unless explicitly disabled, which I've forgotten to do more than once before a talk.

Last year, when I was researching lightweight browsers, one of the ones that impressed me most was something I didn't expect: the demo app that comes with pywebkitgtk (package python-webkit on Ubuntu). In just a few lines of Python, you can create your own browser with any UI you like, with a fully functional content area. Their current demo even has tabs.

So why not use pywebkitgtk to create a simple fullscreen webkit-based presentation tool?

It was even simpler than I expected. Here's the code:

#!/usr/bin/env python
# python-gtk-webkit presentation program.
# Copyright (C) 2009 by Akkana Peck.
# Share and enjoy under the GPL v2 or later.

import sys
import gobject
import gtk
import webkit

class WebBrowser(gtk.Window):
    def __init__(self, url):
        gtk.Window.__init__(self)
        self.fullscreen()

        self._browser= webkit.WebView()
        self.add(self._browser)
        self.connect('destroy', gtk.main_quit)

        self._browser.open(url)
        self.show_all()

if __name__ == "__main__":
    if len(sys.argv) <= 1 :
        print "Usage:", sys.argv[0], "url"
        sys.exit(0)

    gobject.threads_init()
    webbrowser = WebBrowser(sys.argv[1])
    gtk.main()

That's all! No navigation needed, since the slides include javascript navigation to skip to the next slide, previous, beginning and end. It does need some way to quit (for now I kill it with ctrl-C) but that should be easy to add.

Webkit and image buffering

It works great. The only problem is that webkit's image loading turns out to be fairly poor compared to Firefox's. In a presentation where most slides are full-page images, webkit clears the browser screen to white, then loads the image, creating a noticable flash each time. Having the images in cache, by stepping through the slide show then starting from the beginning again, doesn't help much (these are local images on disk anyway, not loaded from the net). Firefox loads the same images with no flash and no perceptible delay.

I'm not sure if there's a solution. I asked some webkit developers and the only suggestion I got was to rewrite the javascript in the slides to do image preloading. I'd rather not do that -- it would complicate the slide code quite a bit solely for a problem that exists only in one library.

There might be some clever way to hack double-buffering in the app code. Perhaps something like catching the 'load-started' signal, switching to another gtk widget that's a static copy of the current page (if there's a way to do that), then switching back on 'load-finished'.

But that will be a separate article if I figure it out. Ideas welcome!

Update, years later: I've used this for quite a few real presentations now. Of course, I keep tweaking it: see my scripts page for the latest version.

Tags: programming, hack, python, web, speaking
[ 16:12 Nov 11, 2009    More programming | permalink to this entry | comments ]

Friendlier error messages for shell newbies

Helping people get started with Linux shells, I've noticed they tend to make two common mistakes vastly more than any others:

1. Typing a file path without a slash, like etc/fstab
2. typing just a filename, without a command in front of it

The first boils down to a misunderstanding of how the Linux file system hierarchy works. (For a refresher, you might want to check out my Linux Planet article Navigating the Linux Filesystem.)

The second problem is due to forgetting the rules of shell grammar. Every shell sentence needs a verb, just like every sentence in English. In the shell, the command is the verb: what do you want to do? The arguments, if any, are the verb's direct object: What do you want to do it to?

(For grammar geeks, there's no noun phrase for a subject because shell commands are imperative. And yes, I ended a sentence with a preposition, so go ahead and feel superior if you believe that's incorrect.)

The thing is, both mistakes are easy to make, especially when you're new to the shell, perhaps coming from a "double-click on the file and let the computer decide what you should do with it" model. The shell model is a lot more flexible and (in my opinion) better -- you, not the computer, gets to decide what you should do with each file -- but it does take some getting used to.

But as a newbie, all you know is that you type a command and get some message like "Permission denied." Why was permission denied? How are you to figure out what the real problem was? And why can't the shell help you with that?

And a few days ago I realized ... it can! Bash, zsh and similar shells have a fairly flexible error handling mechanism. Ubuntu users have seen one part of this, where if you type a command you don't have installed, Ubuntu gives you a fancy error message suggesting what you might have meant and/or what package you might be missing:

$ catt /etc/fstab
No command 'catt' found, did you mean:
 Command 'cat' from package 'coreutils' (main)
 Command 'cant' from package 'swap-cwm' (universe)
catt: command not found

What if I tapped into that same mechanism and wrote a more general handler that could offer helpful suggestions when it looked like the user forgot the command or the leading slash?

It turns out that Ubuntu's error handler uses a ridiculously specific function called command_not_found_handle that can't be used for other errors. Some helpful folks I chatted with on #bash felt, as I did, that such a specific mechanism was silly. But they pointed me to a more general error trapping mechanism that turned out to work fine for my purposes.

It took some fussing and fighting with bash syntax, but I have a basic proof-of-concept. Of course it could be expanded to cover a lot more types of error cases -- and more types of files the user might want to open.

Here are some sample errors it catches:

$ schedule.html
bash: ./schedule.html: Permission denied

schedule.html is an HTML file. Did you want to run: firefox schedule.html

$ screenshot.jpg
bash: ./screenshot.jpg: Permission denied

screenshot.jpg is an image file. Did you want to run:
    pho screenshot.jpg
    gimp screenshot.jpg

$ .bashrc
bash: ./.bashrc: Permission denied

.bashrc is a text file. Did you want to run:
    less .bashrc
    vim .bashrc

$ ls etc/fstab
/bin/ls: cannot access etc/fstab: No such file or directory

Did you forget the leading slash?
etc/fstab doesn't exist, but /etc/fstab does.

You can find the code here: Friendly shell errors and of course I'm happy to take suggestions or contributions for how to make it friendlier to new shell users.

Tags: linux, shell, help, education, programming
[ 14:07 Nov 08, 2009    More linux | permalink to this entry | comments ]

Gathering RSS files for a Palm PDA: FeedMe

For years I've been reading daily news feeds on a series of PalmOS PDAs, using a program called Sitescooper that finds new pages on my list of sites, downloads them, then runs Plucker to translate them into Plucker's open Palm-compatible ebook format.

Sitescooper has an elaborate series of rules for trying to get around the complicated formatting in modern HTML web pages. It has an elaborate cache system to figure out what it's seen before. When sites change their design (which most news sites seem to do roughly monthly), it means going in and figuring out the new format and writing a new Sitescooper site file. And it doesn't understand RSS, so you can't use the simplified RSS that most sites offer. Finally, it's no longer maintained; in fact, I was the last maintainer, after the original author lost interest.

Several weeks ago, bma tweeted about a Python RSS reader he'd hacked up using the feedparser package. His reader targeted email, not Palm, but finding out about feedparser was enough to get me started. So I wrote FeedMe (Carla Schroder came up with the all-important name).

I've been using it for a couple of weeks now and I'm very happy with the results. It's still quite rough, of course, but it's already producing better files than Sitescooper did, and it seems more maintainable. Time will tell.

Of course it needs to be made more flexible, adjusted so that it can produce formats besides Plucker, and so on. I'll get to it.

And the only site I miss now, because it doesn't offer an RSS feed, is Linux Planet. Maybe I'll find a solution for that eventually.

Tags: RSS, python, programming, palm, plucker
[ 20:08 Oct 20, 2009    More programming | permalink to this entry | comments ]

GTK dialogs in GIMP (and updated wallpaper script)

I've been getting tired of my various desktop backgrounds, and realized that I had a lot of trip photos, from fabulous places like Grosvenor Arch (at right), that I'd never added to my background collection.

There's nothing like lots of repetitions of the same task to bring out the shortcomings of a script, and the wallpaper script I threw together earlier this year was no exception. I found myself frequently irritated by not having enough information about what the script was doing or being able to change the filename. Then I could have backgrounds named grosvenor.jpg rather than img2691.jpg.

Alas, I can't use the normal GIMP Save-as dialog, since GIMP doesn't make that dialog available to plug-ins. (That's a deliberate choice, though I've never been clear on the reason behind it.) If I wanted to give that control to the user, I'd have to make my own dialogs.

It's no problem to make a GTK dialog from Python. Just create a gtk.Dialog, add a gtk.Entry to it, call dialog.run(), then check the return value and get the entry's text to see if it changed. No problem, right?

Ha! If you think that, you don't work with computers. The dialog popped up fine, it read the text entry fine ... but it wouldn't go away afterward. So after the user clicked OK, the plug-in tried to save and GIMP popped up the JPEG save dialog (the one that has a quality slider and other controls, but no indication of filename) under my text entry dialog, which remained there.

All attempts at calling dialog.hide() and dialog.destroy() and similar mathods were of no avail. A helpful person on #pygtk worked with me but ended up as baffled as I was. What was up?

The code seemed so simple -- something like this:

    response = dialog.run()
    if response == gtk.RESPONSE_OK :
        pathname = pathentry.get_text()
        dialog.hide()
        dialog.destroy()
        pdb.gimp_file_save(newimg, newimg.active_layer, pathname, pathname,
                           run_mode=0)

In the end, GIMP guru Sven pointed me to the answer. The problem was that my dialog wasn't part of the GTK main loop. In retrospect, this makes sense: the plug-in is an entirely different process, so I shouldn't be surprised that it would have its own main loop. So when I hide() and destroy(), those events don't happen right away because there's no loop in the plug-in process that would see them.

The plug-in passes control back to GIMP to do the gimp_file_save(). GIMP's main loop doesn't have access to the hide and destroy signals I just sent. So the gimp_file_save runs, popping up its own dialog (under mine, because the JPEG save dialog is transient to the original image window while my python dialog isn't). That finishes, returns control to the plug-in, the plug-in exits and at that point GTK cleans up and finally destroys the dialog.

The solution is to loop over GTK events in the plug-in before calling gimp_file_save, like this:

    response = dialog.run()
    if response == gtk.RESPONSE_OK :
        pathname = pathentry.get_text()
        dialog.hide()
        dialog.destroy()
        while gtk.events_pending() :
            gtk.main_iteration()
        pdb.gimp_file_save(newimg, newimg.active_layer, pathname, pathname,
                           run_mode=0)

That loop gives the Python process a chance to clean up the dialog before passing control to GIMP and its main loop. GTK in the subprocess is happy, the user is happy, and I'm happy because now I have a much more efficient way of making lots of desktop backgrounds for lots of different machines.

The updated script, along with a lot more information on how to use it and how to set up tool presets for it.

Tags: programming, gimp, gtk
[ 22:21 Sep 15, 2009    More gimp | permalink to this entry | comments ]

Using apt-file to track down build errors

Someone was asking for help building XEphem on the XEphem mailing list. It was a simple case of a missing include file, where the only trick is to find out what package you need to install to get that file. (This is complicated on Ubuntu, which the poster was using, by the way they fragment the X developement headers into a maze of a xillion tiny packages.)

The solution -- apt-file -- is so simple and easy to use, and yet a lot of people don't know about it. So here's how it works.

The poster reported getting these compiler errors:

ar rc libz.a adler32.o compress.o crc32.o uncompr.o deflate.o trees.o zutil.o inflate.o inftrees.o inffast.o
ranlib libz.a
make[1]: Leaving directory `/home/gregs/xephem-3.7.4/libz'
gcc -I../../libastro -I../../libip -I../../liblilxml -I../../libjpegd -I../../libpng -I../../libz -g -O2 -Wall -I../../libXm/linux86 -I/usr/X11R6/include   -c -o aavso.o aavso.c
In file included from aavso.c:12:
../../libXm/linux86/Xm/Xm.h:56:27: error: X11/Intrinsic.h: No such file or directory
../../libXm/linux86/Xm/Xm.h:57:23: error: X11/Shell.h: No such file or directory
../../libXm/linux86/Xm/Xm.h:58:23: error: X11/Xatom.h: No such file or directory
../../libXm/linux86/Xm/Xm.h:59:34: error: X11/extensions/Print.h: No such file or directory
In file included from ../../libXm/linux86/Xm/Xm.h:60,
                 from aavso.c:12:
../../libXm/linux86/Xm/XmStrDefs.h:1373: error: expected `=', `,', `;', `asm' or `__attribute__' before `char'
In file included from ../../libXm/linux86/Xm/Xm.h:60,
                 from aavso.c:12:
../../libXm/linux86/Xm/XmStrDefs.h:5439:28: error: X11/StringDefs.h: No such file or directory
In file included from ../../libXm/linux86/Xm/Xm.h:61,
                 from aavso.c:12:
../../libXm/linux86/Xm/VirtKeys.h:108: error: expected `)' before `*' token
In file included from ../../libXm/linux86/Xm/Display.h:49,
                 from ../../libXm/linux86/Xm/DragC.h:48,
                 from ../../libXm/linux86/Xm/Transfer.h:44,
                 from ../../libXm/linux86/Xm/Xm.h:62,
                 from aavso.c:12:
../../libXm/linux86/Xm/DropSMgr.h:88: error: expected specifier-qualifier-list before `XEvent'
../../libXm/linux86/Xm/DropSMgr.h:100: error: expected specifier-qualifier-list before `XEvent'

How do you go about figuring this out?

When interpreting compiler errors, usually what matters is the *first* error. So try to find that. In the transcript above, the first line saying "error:" is this one:

../../libXm/linux86/Xm/Xm.h:56:27: error: X11/Intrinsic.h: No such file or directory

So the first problem is that the compiler is trying to find a file called Intrinsic.h that isn't installed.

On Debian-based systems, there's a great program you can use to find files available for install: apt-file. It's not installed by default, so install it, then update it, like this (the update will take a long time):

$ sudo apt-get install apt-file
$ sudo apt-file update

Once it's updated, you can now find out what package would install a file like this:

$  apt-file search Intrinsic.h
libxt-dev: /usr/include/X11/Intrinsic.h
tendra: /usr/lib/TenDRA/lib/include/x5/t.api/X11/Intrinsic.h

In this case two two packages could install a file by that name. You can usually figure out from looking which one is the "real" one (usually the one with the shorter name, or the one where the package name sounds related to what you're trying to do). If you're stil not sure, try something like apt-cache show libxt-dev tendra to find out more about the packages involved.

In this case, it's pretty clear that tendra is a red herring, and the problem is likely that the libxt-dev package is missing. So apt-get install libxt-dev and try the build again.

Repeat the process until you have everything you need for the build.

Remember apt-file if you're not already using it. It's tremendously useful in tracking down build dependencies.

Tags: open source, linux, programming, debian, ubuntu
[ 10:25 Sep 06, 2009    More linux | permalink to this entry | comments ]

Reading Palm Datebook files on Linux

Over the years, I've kept a few sets of records in the Datebook app on my PalmOS PDA -- health records and such. I've been experimenting with a few python plotting packages (pycha, CairoPlot and a few others) and I wanted to try plotting one of my Datebook databases.

Not so fast. It seems that it's been a year or more since I last crunched any of this data -- and in the time since then, pilot-link has bumped its version numbers and is now shipping libpisock.so.9 instead of .8.

So what? Well, the problem is that Linux hasn't offered any way to read Palm Datebook files for years. The pilot-link package offered on most distros used to include a program called pilot-datebook, but it was deleted from the source several years ago. Apparently it was hard to maintain.

Back when it first disappeared, I built the previous version of the source, stuck the pilot-datebook binary in ~/bin/linux and have been using it ever since. Which worked fine -- until libpisock.so.8 was no longer there. (Linking .9 to .8 didn't work either.) This is all the more ironic because I don't need pilot-datebook to talk to the PDA with libpisock -- all I want to do is parse the format of a file I've already uploaded.

Off to hunt for an old version of the source. I started at pilot-link.org, but gave up after a while -- they don't seem to have source there except for the latest couple of versions, nor do they have any documentation. Ironically, in their FAQ the very first question is "How can I read the databook entries from a Palm backup?" but the FAQ page is broken and the "answer" is actually another unrelated FAQ question.

Anyway, no help there. I tried googling for old tarballs but there doesn't seem to be anything like archive.org for source code. All I found was the original pilot-datebook page, with a tarball that you insert into a copy of pilot-link 0.9.5 then modify the Makefile. Might work but that's really old.

So I fell back on old distributions. I guessed that Ubuntu Dapper was old enough that it might still have pilot-datebook. So I went to the Dapper pilot-link source and downloaded the source tarball (curiously, they don't offer src debs -- you have to download the tarball and patches separately).

Of course, it doesn't build on Ubuntu Jaunty. It had various entertaining errors ranging from wanting a mysterious tcl.m4 file not present in the code ... to not being able to find <iostream.h< because all the C++ stdlib files have recently been renamed to remove the .h ... to a change in the open() system call where I needed to add permissions argument for O_CREAT.

But I did get it working! So now I have a pilot-datebook program that builds and runs on Ubuntu Jaunty, and parses my DatebookDB.pdb file.

Since I bet I'm not the only one in the world who occasionally wants to read a Palm Datebook file, I've put my working version of the source here: pilot-link_0.11.8.jaunty.tar.gz.

After the usual configure and make, if all you want is pilot-datebook, cd src/pilot-datebook then copy both pilot-datebook and the directory .libs to wherever you want to install them.

And yeah, it would be better to write a standalone program that just parsed the format. But it's hard to justify that for what's essentially a dead platform. The real solution is to quit using a Palm for this, import the data into some common format and keep it on my Linux workstation from now on.

Tags: palm, linux, programming, patch
[ 11:39 Aug 31, 2009    More linux | permalink to this entry | comments ]

Linux Bloat 102

Part 2 of my Linux bloat article looks at information you can get from the kernel via some useful files in /proc, at three scripts that display that info, and also at how to use exmap, an app and kernel module that shows you a lot more about what resources your apps are using. How Do You Really Measure Linux Bloat?

Tags: writing, programming, linux, performance, bloat
[ 19:52 Aug 27, 2009    More writing | permalink to this entry | comments ]

Crikey 0.8.3: bug fixes and several new syntaxes

Sometimes I love open source. A user contacted me about my program Crikey!, which lets you generate key events to do things like assign a key that will type in a string you don't want to type in by hand. He had some thorny problems where crikey was failing, and a few requests, like sending Alt and other modifier keys.

We corresponded a bit, figured out exactly how things should work and some test cases, went through a couple iterations of changes where I got lots of detailed and thoughtful feedback and more test cases, and now Crikey can do an assortment of new useful stuff.

New features: crikey now handles number codes like \27, modifier keys like \A for alt, does a better job with symbols like \(Return\), and handles a couple of new special characters like \e for escape. It also works better at sending window manager commands, like "\A\t" to change the active window.

I've added some better documentation on all the syntaxes it understands, both on the web page and in the -h and -l (longhelp) command-line arguments, and made a release: crikey 0.8.3.

Plus: a list of great regression tests that I can use when testing future updates (in the file TESTING in the tarball).

Tags: programming, crikey, X11, linux
[ 16:38 Aug 19, 2009    More programming | permalink to this entry | comments ]

Linux Bloat 101

Continuing my Linux Planet series on Linux performance monitoring, the latest article looks at bloat and how you can measure it: Finding and Trimming Linux Bloat.

This one just covers the basics. The followup article, in two weeks, will dive into more detail on how to analyze what resources programs are really using.

Tags: writing, programming, linux, performance, bloat
[ 10:27 Aug 13, 2009    More writing | permalink to this entry | comments ]

On Teaching Programming (GetSET 2009)

I survived another GetSET Javascript-in-a-day workshop for GetSET 2009.

It went okay, but not as well as I'd hoped. This year's class was more distractable than classes of past years -- and, judging by their career goals, less interested in computers or engineering, unfortunate in a program run by the Society of Women Engineers.

In the morning, we had a hard time getting them to focus long enough to learn basics like what a variable was. After a caucus at lunchtime, we decided to skip the next exercise (looping over an array of colors) and spend some time drilling on the basics, and keep at it 'til they got it. It took a while but we eventually got through. We needed more examples in the morning, more interaction, some visceral way of explaining programming basics so they really get it.

They do better working as a group on a concrete problem, like the final whiteboard exercise, "How do we figure out whether the click was on the flower?". That always ends up being a highlight of the class, even though it involves (gasp) doing math. This year was no exception, but it did take a while to get through. Using variables lost them completely ("is the mouse's X coordinate bigger than or less than the flower's X?") but when we used actual numnbers and ran through several examples, things eventually clicked. "The flower starts at (2, 5) and is 200 pixels wide. If the mouse click is at (34, 45), who thinks it's inside the flower? Raise your hands. Who thinks it's not? Now what if I click at (300, 24)?" A couple of them got it right away, but it took a long time to bring the whole class along.

I'm not still sure how to use that method for more basic concepts like "what is a variable?". Perhaps some sort of role-playing? Watching William Phelps guide the girls through planet motions in our astronomy workshop Wednesday, each girl playing the role of a solar system object, inspired me. I'd used role-playing like that with little kids, but William says it works even with adults to get concepts across, and after seeing him with the high schoolers I believe it. But how to adapt that to programming concepts? A recent Slate article on teaching programming had some interesting ideas I want to try.

Printed handouts for GetSET may be a waste of time. Nobody was even bothering to look at them, despite the fact that they had complete instructions for everything we were doing. Do schools not give students printed assignments or homework any more? Last year, they used the printed exercises but not the quick reference guides; this year they wouldn't even read the exercises. On the other hand, it might be worth it for the handful in each class who really love programming. I always hope some of them take the handouts home and try some of the extras on their own.

Finally, the class would be so much easier if we could teach it on a less pointy-clicky OS! Or at least on machines where IE isn't the default browser. The first 3-4 exercises go painfully slowly, guiding a roomful of girls through many GUI navigation steps:

• Open the GetSET folder on your desktop
• Find the file named whatever.html
• Right-click on it, find Open With, and choose Wordpad
• Now find the window where you were just looking at the GetSET files (because everything on Windows tends to open with huge windows, it's now covered by Wordpad)
• Drag whatever.html into Firefox

Then the helpers have to go around the room ensuring that the girls have the correct file loaded in both Wordpad and Firefox. This took way too long with only four people to check the whole class, especially since we had to do it for every exercise. Invariably some girls will doubleclick instead of right-clicking or dragging, and will end up in whatever HTML editor Microsoft is pushing this year, or with an IE window instead of Firefox (and then the Error Console won't be there when she looks for it later).

Suggestions like "Keep that window open, you'll need it throughout the class" or "Try making that window smaller, so you can see both windows at once" don't help. The girls are too used to the standard Windows model of one screen-filling window at a time. Keeping two apps visible at once is too foreign. A few them are good at using the taskbar to switch among apps, but for the rest, loading new files is awkward and error prone.

In postmortems two years ago we talked about having them work on one file throughout the whole workshop. That would solve the problem, but I'm still working on how to do it without a lot of "Now comment out the code you just wrote, so you won't get the prompt every time, then scroll down to the next block of code and uncomment it."

I couldn't help thinking how on Linux, we could just tell them to type leafpad whatever.html; firefox whatever.html and be done. Or even give them an alias that would do it. Hmm ... I wonder if I could make a Windows .bat file that would open the same file in Wordpad and Firefox both? Must try that.

Tags: education, programming
[ 19:12 Aug 07, 2009    More education | permalink to this entry | comments ]

Twit: Now with pattern searches

During OSCON a couple of weeks ago, I kept wishing I could do Twitter searches for a pattern like #oscon in a cleaner way than keeping a tab open in Firefox where I periodically hit Refresh.

Python-twitter doesn't support searches, alas, though it is part of the Twitter API. There's an experimental branch of python-twitter with searching, but I couldn't get it to work. But it turns out Gwibber is also written in Python, and I was able to lift some JSON code from Gwibber to implement a search. (Gwibber itself, alas, doesn't work for me: it bombs out looking for the Gnome keyring. Too bad, looks like it might be a decent client.)

I hacked up a "search for OSCON" program and used it a little during the week of the conference, then got home and absorbed in catching up and preparing for next week's GetSET summer camp, where I'm running an astronomy workshop and a Javascript workshop for high school girls. That's been keeping me frazzled, but I found a little time last night to clean up the search code and release Twit 0.3 with search and a few other new command-line arguments.

No big deal, but it was nice to take a hacking break from all this workshop coordinating. I'm definitely happier program than I am organizing events, that's for sure.

Tags: twitter, python, programming
[ 17:23 Aug 03, 2009    More programming | permalink to this entry | comments ]

Newbie Greasemonkey script writing

I was reading a terrific article on the New York Times about Watching Whales Watching Us. At least, I was trying to read it -- but the NYT website forces font faces and sizes that, on my system, end up giving me a tiny font that's too small to read. Of course I can increase font size with Ctrl-+ -- but it gets old having to do that every time I load a NYT page.

The first step was to get Greasemonkey working on Firefox 3.5. "Update scripts" doesn't find a new script, and if you go to Greasemonkey's home page, the last entry is from many months ago and announces Firefox 3.1 support. But curiously, if you go to the Greasemonkey page on the regular Mozilla add-ons site, it does support 3.5.

I've had Greasemonkey for quite some time, but every time I try to get started writing a script I have trouble getting started. There are dozens of Greasemonkey tutorials on the web, but most of them are oriented toward installing scripts and don't address "What do you type into the fields of the Greasemonkey New User Script dialog?"

Fortunately, I did find one that explained it: The beginner's guide to Greasemonkey scripting. I gave my script a name (NYT font) and a namespace (my own domain), added http://*nytimes.com/* for Includes, and nothing for Excludes.

Click OK, and Greasemonkey offers a "choose editor" dialog. I chose emacs, which mostly worked though the emacs window unaccountably came up with a split window that I had to dismiss with C-x 1.

Now what to type in the editor? Firebug came to the rescue here.

I went back to the NYT page with the too-small fonts and clicked on Firebug. The body style showed that they're setting

font-family: Georgia, serif
font-size: 84.5%

84.5%? Where does that come from? What happens if I change that to 100%? Fortunately, I can test that right there in the Firebug window. 100% made the fonts fairly huge, but 90% was about right.

I went back to greasemonkey's editor window and added:

document.body.style.fontSize = "90%";

Saved the file, and that was all I needed! Once I hit Reload on the NYT page I got a much more readable font size.

Tags: web, browsers, firefox, greasemonkey, programming
[ 11:30 Jul 12, 2009    More tech/web | permalink to this entry | comments ]

Twittering -- and writing Twitter clients

I finally dragged myself into 2009 and tried Twitter.

I'd been skeptical, but it's actually fairly interesting and not that much of a time sink. While it's true that some people tweet about every detail of their lives -- "I'm waiting for a bus" / "Oh, hooray, the bus is finally here" / "I got a good seat in the second row of the bus" / "The bus just passed Second St. and two kids got on" / "Here's a blurry photo from my phone of the Broadway Av. sign as we pass it" -- it's easy enough to identify those people and un-follow them.

And there are tons of people tweeting about interesting stuff. It's like a news ticker, but customizable -- news on the latest protests in Iran, the latest progress on freeing the Mars Spirit Rover, the latest interesting publication on dinosaur fossils, and what's going on at that interesting conference halfway around the world.

The trick is to figure out how you want the information delivered. I didn't want to have to leave a tab open in Firefox all the time. There was an xchat plug-in that sounded perfect -- I have an xchat window up most of the time I'm online -- but it turned out it works by picking one of the servers you're connected to, making a private channel and posting things there. That seemed abusive to the server -- what if everyone on Freenode did that?

So I wanted a separate client. Something lightweight and simple. Unfortunately, all the Twitter clients available for Linux either require that I install a lot of infrastructure first (either Adobe Air or Mono), or they just plain didn't work (a Twitter client where you can't click on links? Come on!)

But then I tried out the Python-Twitter bindings, and they were so easy to use I decided to write them up for my next Linux Planet article, which came out today: Write Your Own Linux Twitter Client In Less Time Than It Takes To Find One!.

The article shows how to use the bindings to write a bare-bones client. But of course, I've been hacking on the client all along, so the one I'm actually using has a lot more features like *ahem* letting you click on links. And letting you block threads, though I haven't actually tested that since I haven't seen any threads I wanted to block since my first day.

You can download the current version of Twit, and anyone who's interested can follow me on Twitter. I don't promise to be interesting -- that's up to you to decide -- but I do promise not to tweet about every block of my bus ride.

Tags: writing, programming, python, twitter
[ 15:09 Jul 09, 2009    More writing | permalink to this entry | comments ]

A Beginner's Guide to Free Software Programming Languages

Linux Planet asked me for an intro article for prospective programmers, explaining the pros and cons of various programming languages. Here it is: A Beginner's Guide to Free Software Programming Languages

Tags: writing, programming
[ 12:00 Jun 28, 2009    More writing | permalink to this entry | comments ]

Pytopo 0.8 released

On my last Mojave trip, I spent a lot of the evenings hacking on PyTopo.

I was going to try to stick to OpenStreetMap and other existing mapping applications like TangoGPS, a neat little smartphone app for downloading OpenStreetMap tiles that also runs on the desktop -- but really, there still isn't any mapping app that works well enough for exploring maps when you have no net connection.

In particular, uploading my GPS track logs after a day of mapping, I discovered that Tango really wasn't a good way of exploring them, and I already know Merkaartor, nice as it is for entering new OSM data, isn't very good at working offline. There I was, with PyTopo and a boring hotel room; I couldn't stop myself from tweaking a bit.

Adding tracklogs was gratifyingly easy. But other aspects of the code bother me, and when I started looking at what I might need to do to display those Tango/OSM tiles ... well, I've known for a while that some day I'd need to refactor PyTopo's code, and now was the time.

Surprisingly, I completed most of the refactoring on the trip. But even after the refactoring, displaying those OSM tiles turned out to be a lot harder than I'd hoped, because I couldn't find any reliable way of mapping a tile name to the coordinates of that tile. I haven't found any documentation on that anywhere, and Tango and several other programs all do it differently and get slightly different coordinates. That one problem was to occupy my spare time for weeks after I got home, and I still don't have it solved.

But meanwhile, the rest of the refactoring was done, nice features like track logs were working, and I've had to move on to other projects. I am going to finish the OSM tile MapCollection class, but why hold up a release with a lot of useful changes just for that?

So here's PyTopo 0.8, and the couple of known problems with the new features will have to wait for 0.9.

Tags: programming, python, gtk, pygtk, mapping
[ 19:49 Jun 20, 2009    More programming | permalink to this entry | comments ]

Python: show all methods in a given object or module

A silly little thing, but something that Python books mostly don't mention and I can never find via Google:

How do you find all the methods in a given class, object or module?

Ideally the documentation would tell you. Wouldn't that be nice? But in the real world, you can't count on that, and examining all of an object's available methods can often give you a good guess at how to do whatever you're trying to do.

Python objects keep their symbol table in a dictionary called __dict__ (that's two underscores on either end of the word). So just look at object.__dict__. If you just want the names of the functions, use object.__dict__.keys().

Thanks to JanC for suggesting dir(object) and help(object), which can be more helpful -- not all objects have a __dict__.

Tags: programming, python, tips
[ 11:44 Jun 19, 2009    More programming | permalink to this entry | comments ]

Programming With PyGTK, part 3: Key events and object oriented Python

Part 3 of "Graphical Python Programming With PyGTK" uses object-oriented Python to clean up the code from Part 2, and also adds handling of key events to get rid of that silly Quit button. PythonGTK Programming part 3: Screensaver, Objects, and User Input

Tags: writing, python, programming
[ 11:18 Jun 14, 2009    More writing | permalink to this entry | comments ]

A GPX file manager

Someone on the OSM newbies list asked how he could strip waypoints out of a GPX track file. Seems he has track logs of an interesting and mostly-unmapped place that he wants to add to openstreetmap, but there are some waypoints that shouldn't be included, and he wanted a good way of separating them out before uploading.

Most of the replies involved "just edit the XML." Sure, GPX files are pretty simple and readable XML -- but a user shouldn't ever have to do that! Gpsman and gpsbabel were also mentioned, but they're not terribly easy to use either.

That reminded me that I had another XML-parsing task I'd been wanting to write in Python: a way to split track files from my Garmin GPS.

Sometimes, after a day of mapping, I end up with several track segments in the same track log file. Maybe I mapped several different trails; maybe I didn't get a chance to upload one day's mapping before going out the next day. Invariably some of the segments are of zero length (I don't know why the Garmin does that, but it always does). Applications like merkaartor don't like this one bit, so I usually end up editing the XML file and splitting it into segments by hand. I'm comfortable with XML -- but it's still silly.

I already have some basic XML parsing as part of PyTopo and Ellie, so I know the parsing very easy to do. So, spurred on by the posting on OSM-newbies, I wrote a little GPX parser/splitter called gpxmgr. gpxmgr -l file.gpx can show you how many track logs are in the file; gpxmgr -w file.gpx can write new files for each non-zero track log. Add -p if you want to be prompted for each filename (otherwise it'll use the name of the track log, which might be something like "ACTIVE\ LOG\ #2").

How, you may wonder, does that help the original poster's need to separate out waypoints from track files? It doesn't. See, my GPS won't save tracklogs and waypoints in the same file, even if you want them that way; you have to use two separate gpsbabel commands to upload a track file and a waypoint file. So I don't actually know what a tracklog-plus-waypoint file looks like. If anyone wants to use gpxmgr to manage waypoints as well as tracks, send me a sample GPX file that combines them both.

Tags: mapping, programming, python
[ 19:43 Jun 01, 2009    More mapping | permalink to this entry | comments ]

JS Jup: now, with variable animation speed

I wrote last week about the sorts of programmer compulsions that lead to silly apps like my animated Javascript Jupiter. I got it working well enough and stopped, knowing there were more features that would be easy to add but trying to ignore them.

My mom, immediately upon seeing it, unerringly zeroed in on the biggest missing feature I'd been trying to ignore. "Can you make it go faster or slower?"

I put it off for a while, but of course I had to do it -- so now there are Faster and Slower buttons. It still goes by hour jumps, so the fastest you can go is an hour per millisecond. Fun to watch. Or you can slow it down to 1 hour per 3600000 milliseconds if you want to see it animate in real time. :-)

Tags: programming, astronomy, javascript
[ 10:42 May 31, 2009    More programming | permalink to this entry | comments ]

Programming With PyGTK, part 2: pretty screensaver-type graphics

Part 2 of Graphical Python Programming With PyGTK gets into how to do some cool Qix screensaver-style graphics, in: Graphical Python Programming With PyGTK, part 2: Write Your Own Screensaver.

There's also a digg link.

Tags: writing, python, programming
[ 17:09 May 28, 2009    More writing | permalink to this entry | comments ]

Javascript Jupiter

It's a sickness, I tell you.

It's not like I needed another Jupiter's moons application. I've already written more or less the same app for four platforms.

I don't use the Java web version, Juplet, very much any more, because I often have Java disabled or missing. And I don't use my Zaurus any more so Juplet for Zaurus isn't very relevant. But I can always call up my Xlib or PalmOS Jupiter's moons app if I need to check on those Galilean moons. They work fine. Another version would be really pointless. A waste of time.

So it should have been no big deal when, during the course of explaining to someone the difference between Java and Javascript, it suddenly occurred to me that it would be awfully easy to re-implement that Java Juplet web page using Javascript, HTML and CSS. I mean, a rational person would just say "oh, yeah, I suppose that's true" and go on with life.

But what I'm trying to say is that programming isn't a career path, or a hobby, or a field of academic study. It's a disease. It's a compulsion, where, sometimes, just realizing that something could be done renders you unable to think about anything else until you just ... try ... just a few minutes ... see how well it works ... oh, wow, that really looks a lot better than the Java version, wouldn't it look even nicer if you just added in this one other little tweak ... but wait, now it's so close to working, I bet it wouldn't be all that hard to take the Java class and turn it into ...

... and before you know it, it's tomorrow and you have something that's almost a working app, and it's just really a shame to get that far and not finish it at least to the point where you can share it.

But then, Javascript and web pages are so easy to work on that it really isn't that much extra work to add in some features that the old version didn't have, like an animate button ...

... and your Saturday morning is gone forever, and there's not much you can do about that, but at least you have a nice animated Jupiter's moons (and shadows) page when the sickness passes and you can finally think about other things.

Tags: programming, astronomy, javascript
[ 20:10 May 23, 2009    More programming | permalink to this entry | comments ]

Graphical Python Programming With PyGTK

This week's Linux Planet article is another one on Python and graphical toolkits, but this time it's a little more advanced: Graphical Python Programming With PyGTK.

This one started out as a fun and whizzy screensaver sort of program that draws lots of pretty colors -- but I couldn't quite fit it all into one article, so that will have to wait for the sequel two weeks from now.

Tags: writing, python, programming
[ 18:53 May 14, 2009    More writing | permalink to this entry | comments ]

GUI Programming in Python For Beginners

Latest on Linux Planet: another introductory programming article, this time on Python's tkinter library: GUI Programming in Python For Beginners. (As usual, there's a Digg link and also a Reddit one.)

Tags: writing, python, programming
[ 15:36 Mar 26, 2009    More writing | permalink to this entry | comments ]

Ellie: Plot GPS elevation profiles

Ever since I got the GPS I've been wanting something that plots the elevation data it stores. There are lots of apps that will show me the track I followed in latitude and longitude, but I couldn't find anything that would plot elevations.

But GPX (the XML-based format commonly used to upload track logs) is very straightforward -- you can look at the file and read the elevations right out of it. I knew it wouldn't be hard to write a script to plot them in Python; it just needed a few quiet hours. Sounded like just the ticket for a rainy day stuck at home with a sore throat.

Sure enough, it was fairly easy. I used xml.dom.minidom to parse the file (I'd already had some experience with it in gimplabels for converting gLabels templates), and pylab from matplotlib for doing the plotting. Easy and nice looking.

I even threw in the nice "conditional main" code from Matt Harrison's SCALE7x Python talk, so it should be callable from other Python code.

Here's the page and a screenshot: Ellie: plot elevation from a GPS track.

Tags: programming, mapping, python
[ 16:57 Mar 03, 2009    More programming | permalink to this entry | comments ]

langgrep: search only in scripts of a specified language

I was making a minor tweak to my garmin script that uses gpsbabel to read in tracklogs and waypoints from my GPS unit, and I needed to look up the syntax of how to do some little thing in sh script. (One of the hazards of switching languages a lot: you forget syntax details and have to look things up a lot, or at least I do.)

I have quite a collection of scripts in various languages in my ~/bin (plus, of course, all the scripts normally installed in /usr/bin on any Linux machine) so I knew I'd have lots of examples. But there are scripts of all languages sharing space in those directories; it's hard to find just sh examples. For about the two-hundredth time, I wished, "Wouldn't it be nice to have a command that can search for patterns only in files that are really sh scripts?"

And then, the inevitable followup ... "You know, that would be really easy to write."

So I did -- a little python hack called langgrep that takes a language, grep arguments and a file list, looks for a shebang line and only greps the files that have a shebang matching the specified language.

Of course, while writing langgrep I needed langgrep, to look up details of python syntax for things like string.find (I can never remember whether it's string.find(s, pat) or s.find(pat); the python libraries are usually nicely object-oriented but strings are an exception and it's the former, string.find). I experimented with various shell options -- this is Unix, so of course there are plenty of ways of doing this in the shell, without writing a script. For instance:

grep find `egrep -l '#\\!.*python' *`
grep find `file * | grep python | sed 's/:.*//'`
i in foo; file $i|grep python && grep find $i; done    # in sh/bash

These are all pretty straightforward, but when I try to make them into tcsh aliases things get a lot trickier. tcsh lets you make aliases that take arguments, so you can use !:1 to mean the first argument, !2-$ to mean all the arguments starting with the second one. That's all very well, but when you put them into a shell alias in a file like .cshrc that has to be parsed, characters like ! and $ can mean other things as well, so you have to escape them with \. So the second of those three lines above turns into something like

alias greplang "grep \!:2-$ `file * | grep \!:1 | sed 's/:.*//'`"

except that doesn't work either, so it probably needs more escaping somewhere. Anyway, I decided after a little alias hacking that figuring out the right collection of backslash escapes would probably take just as long as writing a python script to do the job, and writing the python script sounded more fun.

So here it is: my langgrep script. (Awful name, I know; better ideas welcome!) Use it like this (if python is the language you're looking for, find is the search pattern, and you want -w to find only "find" as a whole word):

langgrep python -w find ~/bin/*

Tags: programming, python, shell
[ 09:57 Feb 28, 2009    More programming | permalink to this entry | comments ]

Use GIMP Tool Presets for setting multiple crop aspect ratios

Making desktop backgrounds in GIMP is a bit tedious if you have several machines with screens of different sizes. The workflow goes something like this:

First, choose Crop tool and turn on Fixed: Aspect Ratio. Then loop over images:

1. Load an image
2. Go to Tool Options
3. Type in the aspect ratio: 4:3, 8:5, 5:4, 1366:768 etc.
4. Go to the image and crop.
5. Image->Scale (I have this on Shift-S, can't remember whether that was a default binding or not).
6. Ctrl-K to delete the current width (Ctrl-U also works, but beeps; I'm not sure why)
7. Type in the desired width (1024 or 1680 or 1366 or whatever) (I always hit Tab here, though it's probably not necessary)
8. Click Scale or type Alt-S (unfortunately, Return doesn't work in this dialog).
9. Save As to the appropriate name and path for the current resolution
10. Undo (the Scale), Undo (the Crop)
11. Load a new image (continue loop)

But you can use Save Options (Tool Presets) to avoid step 3, typing in the aspect ratio. Here's how:

• Set up the aspect ratio you want in the Crop tool, with Fixed checked
• Click on Save Options (the floppy disk icon in the lower left of Tool Options)
• Choose a name (choose New Entry first if you've already saved options).

Repeat, for each aspect ratio you might want to use.

Now clicking on Restore Options gives you a menu of all your commonly used aspect ratios -- much faster than typing them in every time. Too bad there's no way to use this shortcut for the Scale step, or to do Crop and Scale in one operation.

Nice shortcut! But having done that, I realized I could shorten it even more: I could make a selection (not a crop) with one of my preset aspect ratios, then run a script that would figure out from the aspect ratio which absolute size I wanted, crop and scale, and either save it to the right place, or make a new image so I could save without needing to Redo or Save a Copy. That was an easy script to write, so here it is: wallpaper.py.

Tags: gimp, tips, programming, hack
[ 22:45 Feb 05, 2009    More gimp | permalink to this entry | comments ]

Crikey 0.8

Mostly this week has been consumed with preparations for LCA ... but programming is a sickness. When you get email from someone suggesting something relatively simple and obviously useful, well ... it's simply impossible not to pull out that emacs window and start typing.

And so it was when I got a request for a backspace character in crikey. Of course backspace and delete seem like perfectly reasonable and useful characters to want; don't know why I didn't think of putting them in before. So I did.

But while I was in there, suddenly it occurred to me that it really wouldn't be much harder to let users specify any key by symbol. (Did I mention being a programmer is a sickness?) And then I realized that specifying control characters with a caret, like ^H, would also be quite useful. (Did I mention that ...)

So anyway, now there's a Crikey 0.8 and it's time to get back to packing and endless fiddling with my talk slides. Except, wait, I need to update my netscheme script to work right with the new laptop, and ...

Did I mention that programming is a sickness?

Tags: programming, crikey
[ 20:56 Jan 14, 2009    More programming | permalink to this entry | comments ]

Debian/Ubuntu repositories for Pho

I've been wanting for a long time to make Debian and Ubuntu repositories so people can install pho with apt-get, but every time I try to look it up I get bogged down.

But I got mail from a pho user who really wanted that, and even suggested a howto. That howto didn't quite do it, but it got me moving to look for a better one, which I eventually found in the Debian Repository Howto.

It wasn't complete either, alas, so it took some trial-and-error before it actually worked. Here's what finally worked:

I created two web-accessible directories, called hardy and etch. I copied all the files created by dpgk-buildpkg on each distro -- .deb, .dsc, .tar.gz, and .changes (I don't think this last file is used by anything) -- into each directory (renaming them to add -etch and -hardy as appropriate). Then:

% cd hardy/
% dpkg-scanpackages . /dev/null | gzip > Packages.gz
% dpkg-scansources . /dev/null | gzip > Sources.gz
% cd ../etch/
% dpkg-scanpackages . /dev/null | gzip > Packages.gz
% dpkg-scansources . /dev/null | gzip > Sources.gz

It gives an error,
** Packages in archive but missing from override file: **
but seems to work anyway.

Now you can use one of the following /etc/apt/sources.list lines:
deb http://shallowsky.com/apt/hardy ./
deb http://shallowsky.com/apt/etch ./

After an apt-get update, it saw pho, but it warned me

WARNING: The following packages cannot be authenticated!
  pho
Install these packages without verification [y/N]?

There's some discussion in the SecureAPT page on the Debian wiki, but it's a bit involved and I'm not clear if it helps me if I'm not already part of the official Debian keychain.

This page on Release check of non Debian sources was a little more helpful, and told me how to create the Release and Release.gpg file -- but then I just get a different error,

 The following signatures couldn't be verified because the public key is not available: NO_PUBKEY

And worse, it's an error now, not just a warning, preventing any apt-get update.

Going back to the SecureApt page, under Setting up a secure apt repository they give the two steps the other page gave for creating Release and Release.gpg, with a third step: "Publish the key fingerprint, that way your users will know what key they need to import in order to authenticate the files in the archive."

So apparently if users don't take steps to import the key manually, they can't update at all. Whereas if I leave out the Release and Release.gpg files, all they have to do is type y when they see the warning. Sounds like it's better to leave off the key. I wish, though, that there was a middle ground, where I could offer the key for those who wanted it without making it harder for those who don't care.

Tags: programming, pho, debian, ubuntu, linux
[ 20:14 Jan 13, 2009    More linux | permalink to this entry | comments ]

Cleaning up the edges of Moonroot's transparent images

I wrote moonroot more to figure out how to do it than to run it myself. But on the new monitor I have so much screen real estate that I've started using it -- but the quality of the images was such an embarrassment that I couldn't stand it. So I took a few minutes and cleaned up the images and made a moonroot 0.6 release.

Turned out there was a trick I'd missed when I originally made the images, years ago. XPM apparently only allows 1-bit transparency. When I was editing the RGB image and removing the outside edge of the circle, some of the pixels ended up semi-transparent, and when I saved the file as .xpm, they ended up looking very different (much darker) from what I had edited.

Here are two ways to solve that in GIMP:

1. Use the "Hard edge" option on the eraser tool (and a hard-edged brush, of course, not a fuzzy one).
2. Convert the image to indexed, in which case GIMP will only allow one bit's worth of transparency. (That doesn't help for full-color images, but for a greyscale image like the moon, there's no loss of color since even RGB images can only have 8 bits per channel.)

Either way, the way to edit a transparent image where you're trying to make the edges look clean is to add a solid-color background layer (I usually use white, but of course it depends on how you're going to use the image) underneath the layer you're trying to edit. (In the layers dialog, click the New button, chose White for the new layer, click the down-arrow button to move it below the original layer, then click on the original layer so your editing will all happen there.)

Once you're editing a circle with sharp edges, you'll probably need to adjust the colors for some of the edge pixels too. Unfortunately the Smudge tool doesn't seem to work on indexed images, so you'll probably spend a lot of time alternating between the Color Picker and the Pencil tool, picking pixel colors then dabbing them onto other pixels. Key bindings are the best way to do that: o activates the Color Picker, N the Pencil, P the Paintbrush. Even if you don't normally use those shortcuts it's worth learning them for the duration of this sort of operation.

Or use the Clone tool, where the only keyboard shortcut you have to remember is Ctrl to pick a new source pixel. (I didn't think of that until I was already finished, but it works fine.)

Tags: programming, astronomy, gimp
[ 14:48 Nov 16, 2008    More gimp | permalink to this entry | comments ]

Pho 0.9.6 finally released!

Pho 0.9.6-pre3 has been working great for me for about a month, and I've been trying to find the time to do a release. I finally managed it this weekend, after making a final tweak to change the default PHO_REMOTE command from gimp-remote to gimp since gimp-remote is obsolete and is no longer built by default.

The big changes from 0.9.5 are Keywords mode, slideshow mode, the new PHO_REMOTE environment variable, swapping -f and -F, and a bunch of performance work and minor bug fixing.

I built deb packages for Ubuntu (Hardy, but they should work on Intrepid too) and Debian (Etch), as well as the usual source tarball, and they're available at the usual place:

http://shallowsky.com/software/pho.

Tags: programming, pho, image viewer
[ 17:11 Nov 09, 2008    More programming | permalink to this entry | comments ]

A word count bookmarklet

This posting ended up being published as a Linux Planet Quick Tip. You can read about my nifty word counting bookmarklet there: Quick Firefox Tip: Word Count Bookmarklet.

Tags: firefox, mozilla, bookmarklets, writing, programming
[ 22:41 Nov 03, 2008    More tech/web | permalink to this entry | comments ]

Requesting no window decorations (and moonroot 0.4)

Someone on #openbox this morning wanted help in bringing up a window without decorations -- no titlebar or window borders.

Afterward, Mikael commented that the app should really be coded not to have borders in the first place.

Me: You can do that?

Turns out it's not a standard ICCCM request, but one that mwm introduced, MWM_HINTS_DECORATIONS. Mikael pointed me to the urxvt source as an example of an app that uses it.

My own need was more modest: my little moonroot Xlib program that draws the moon at approximately its current phase. Since the code is a lot simpler than urxvt, perhaps the new version, moonroot 0.4, will be useful as an example for someone (it's also an example of how to use the X Shape extension for making non-rectangular windows).

Tags: programming, xlib, astronomy, window managers
[ 11:06 Oct 20, 2008    More programming | permalink to this entry | comments ]

New Pho 0.9.6-pre3

I've released Pho 0.9.6-pre3. The only change is to fix a sporadic bug where pho would sometimes jump back to the first image after deleting the last one, rather than backing up to the next-to-last image. I was never able to reproduce the bug reliably, but I cleaned up the image list next/prev code quite a bit and haven't seen the bug since then. I'd appreciate having a few testers exercising this code as much as possible.

Otherwise pho is looking pretty solid for a 0.9.6 release.

Tags: programming, pho, image viewer
[ 09:57 Oct 02, 2008    More programming | permalink to this entry | comments ]

Fast Pixel Ops in GIMP-Python

Last night Joao and I were on IRC helping someone who was learning to write gimp plug-ins. We got to talking about pixel operations and how to do them in Python. I offered my arclayer.py as an example of using pixel regions in gimp, but added that C is a lot faster for pixel operations. I wondered if reading directly from the tiles (then writing to a pixel region) might be faster.

But Joao knew a still faster way. As I understand it, one major reason Python is slow at pixel region operations compared to a C plug-in is that Python only writes to the region one pixel at a time, while C can write batches of pixels by row, column, etc. But it turns out you can grab a whole pixel region into a Python array, manipulate it as an array then write the whole array back to the region. He thought this would probably be quite a bit faster than writing to the pixel region for every pixel.

He showed me how to change the arclayer.py code to use arrays, and I tried it on a few test layers. Was it faster? I made a test I knew would take a long time in arclayer, a line of text about 1500 pixels wide. Tested it in the old arclayer; it took just over a minute to calculate the arc. Then I tried Joao's array version: timing with my wristwatch stopwatch, I call it about 1.7 seconds. Wow! That might be faster than the C version.

The updated, fast version (0.3) of arclayer.py is on my arclayer page.

If you just want the trick to using arrays, here it is:

from array import array

[ ... setting up ... ]
        # initialize the regions and get their contents into arrays:
        srcRgn = layer.get_pixel_rgn(0, 0, srcWidth, srcHeight,
                                     False, False)
        src_pixels = array("B", srcRgn[0:srcWidth, 0:srcHeight])

        dstRgn = destDrawable.get_pixel_rgn(0, 0, newWidth, newHeight,
                                            True, True)
        p_size = len(srcRgn[0,0])               
        dest_pixels = array("B", "\x00" * (newWidth * newHeight * p_size))

[ ... then inside the loop over x and y ... ]
                        src_pos = (x + srcWidth * y) * p_size
                        dest_pos = (newx + newWidth * newy) * p_size
                        
                        newval = src_pixels[src_pos: src_pos + p_size]
                        dest_pixels[dest_pos : dest_pos + p_size] = newval

[ ... when the loop is all finished ... ]
        # Copy the whole array back to the pixel region:
        dstRgn[0:newWidth, 0:newHeight] = dest_pixels.tostring() 

Good stuff!

Tags: gimp, python, programming, performance
[ 21:02 Aug 16, 2008    More gimp | permalink to this entry | comments ]

GetSET: Teaching Javascript to high school girls

Every summer I volunteer as an instructor for a one-day Javascript programming class at the GetSET summer technology camp for high school girls. GetSET is a great program run by the Society of Women Engineers. it's intended for minority girls from relatively poor neighborhoods, and the camp is free to the girls (thanks to some great corporate sponsors). They're selected through a competitive interview process so they're all amazingly smart and motivated, and it's always rewarding being involved.

Teaching programming in one day to people with no programming background at all is challenging, of course. You can't get into any of the details you'd like to cover, like style, or debugging techniques. By the time you get through if-else, for and while loops, some basic display methods, the usual debugging issues like reading error messages, and typographical issues like "Yes, uppercase and lowercase really are different" and "No, sorry, that's a colon, you need a semicolon", it's a pretty full day and the students are saturated.

I got drafted as lead presenter several years ago, by default by virtue of being the only one of the workshop leaders who actually programs in Javascript. For several years I'd been asking for a chance to rewrite the course to try to make it more fun and visual (originally it used a lot of form validation exercises), and starting with last year's class I finally got the chance. I built up a series of graphics and game exercises (using some of Sara Falamaki's Hangman code, which seemed perfect since she wrote it when she was about the same age as the girls in the class) and it went pretty well. Of course, we had no idea how fast the girls would go or how much material we could get through, so I tried to keep it flexible and we adjusted as needed.

Last year went pretty well, and in the time since then we've exchanged a lot of email about how we could improve it. We re-ordered some of the exercises, shifted our emphasis in a few places, factored some of the re-used code (like windowWidth()) into a library file so the exercise files weren't so long, and moved more of the visual examples earlier.

I also eliminated a lot of the slides. One of the biggest surprises last year was the "board work". I had one exercise where the user clicks in the page, and the student has to write the code to figure out whether the click was over the image or not. I had been nervous about that exercise -- I considered it the hardest of the exercises. You have to take the X and Y coordinates of the mouse click, the X and Y coordinates of the image (the upper left corner of the <div> or <img> tag), and the size of the image (assumed to be 200x200), and turn that all into a couple of lines of working Javascript code. Not hard once you understand the concepts, but hard to explain, right?

I hadn't made a slide for that, so we went to the whiteboard to draw out the image, the location of the mouse click, the location of the image's upper left corner, and figure out the math ... and the students, who had mostly been sitting passively through the heavily slide-intensive earlier stuff, came alive. They understood the diagram, they were able to fill in the blanks and keep track of mouse click X versus image X, and they didn't even have much trouble turning that into code they typed into their exercise. Fantastic!

Remembering that, I tried to use a lot fewer slides this year. I felt like I still needed to have slides to explain the basic concepts that they actually needed to use for the exercises -- but if there was anything I thought they could figure out from context, or anything that was just background, I cut it. I tried for as few slides as possible between exercises, and more places where we could elicit answers from the students. I think we still have too many slides and not enough "board work" -- but we're definitely making progress, and this year went a lot better and kept them much better engaged. We're considering next year doing the first several exercises on the board first, then letting them type it in to their own copies to verify that it works.

We did find we needed to leave code examples visible: after showing slides saying something like "Ex 7: Write a loop that writes a line of text in each color", I had to back up to the previous slide where I'd showed what the code actually looked like. I had planned on their using my "Javascript Quick Reference" handout for reference and not needing that information on the slides; but in fact, I think they were confused about the quickref and most never even opened it. Either that information needs to be in the handout, or it needs to be displayed on the screen as they work, or I have to direct them to the quickref page explicitly ("Now turn to page 3 in ...") or put that information in the exercises.

The graphical flower exercises were a big hit this year (I showed them early and promised we'd get to them, and when we did, just before lunch, several girls cheered) and, like last year, some of the girls who finished them earlier decided on their own that they wanted to change them to use other images, which was also a big hit. Several other girls decided they wanted more than 10 flowers displayed, and others hit on the idea of changing the timeout to be a lot shorter, which made for some very fun displays. Surprisingly, hardly anyone got into infinite loops and had to kill the browser (always a potential problem with javascript, especially when using popups like alert() or prompt()).

I still have some issues I haven't solved, like what to do about semicolons and braces. Javascript is fairly agnostic about them. Should I tell the girls that they're required? (I did that this year, but it's confusing because then when you get to "if" statements you have to explain why that's different.) Not mention them at all? (I'm leaning toward that for next year.)

And it's always a problem figuring out what the fastest girls should do while waiting for the rest to finish. This year, in addition to trying to make each exercise shorter, we tried having the girls work on them in groups of two or three, so they could help each other. It didn't quite work out that way -- they all worked on their own copies of the exercises but they did seem to collaborate more, and I think that's the best balance. We also encourage the ones who finish first to help the girls around them, which mostly they do on their own anyway.

And we really do need to find a better editor we can use on the Windows lab machines instead of Wordpad. Wordpad's font is too small on the projection machine, and on the lab machines it's impossible for most of us to tell the difference between parentheses, brackets and braces, which leads to lots of time-wasting subtle bugs. Surely there's something available for Windows that's easy to use, freely distributable, makes it easy to change the font, and has parenthesis and brace matching (syntax highlighting would be nice too). Well, we have a year to look for one now.

All in all, we had a good day and most of the girls gave the class high marks. Even the ones who concluded "I learned I shouldn't be a programmer because it takes too much attention to detail" said they liked the class. And we're fine with that -- not everybody wants to be a programmer, and the point isn't to force them into any specific track. We're happy if we can give them an idea of what computer programming is really like ... then they'll decide for themselves what they want to be.

Tags: education, javascript, programming, speaking
[ 11:54 Aug 09, 2008    More education | permalink to this entry | comments ]

Crikey in Python, and generating key events with XTest

A user on the One Laptop Per Child (OLPC, also known as the XO) platform wrote to ask me how to use crikey on that platform.

There are two stages to getting crikey running on a new platform:

1. Build it, and
2. Figure out how to make a key run a specific program.

The crikey page contains instructions I've collected for binding keys in various window managers, since that's usually the hard part. On normal Linux machines the first step is normally no problem. But apparently the OLPC comes with gcc but without make or the X header files. (Not too surprising: it's not a machine aimed at developers and I assume most people developing for the machine cross-compile from a more capable Linux box.)

We're still working on that (if my correspondant gets it working, I'll post the instructions), but while I was googling for information about the OLPC's X environment I stumbled upon a library I didn't know existed: python-xlib. It turns out it's possible to do most or all of what crikey does from Python. The OLPC is Python based; if I could write crikey in Python, it might solve the problem. So I whipped up a little key event generating script as a test.

Unfortunately, it didn't solve the OLPC problem (they don't include python-xlib on the machine either) but it was a fun exercises, and might be useful as an example of how to generate key events in python-xlib. It supports both event generating methods: the X Test extension and XSendEvent. Here's the script: /pykey-0.1.

But while I was debugging the X Test code, I had to solve a bug that I didn't remember ever solving in the C version of crikey. Sure enough, it needed the same fix I'd had to do in the python version. Two fixes, actually. First, when you send a fake key event through XTest, there's no way to specify a shift mask. So if you need a shifted character like A, you have to send KeyPress Shift, KeyPress a. But if that's all you send, XTest on some systems does exactly what the real key would do if held down and never released: it autorepeats. (But only for a little while, not forever. Go figure.)

So the real answer is to send KeyPress Shift, KeyPress a, KeyRelease a, KeyRelease Shift. Then everything works nicely. I've updated crikey accordingly and released version 0.7 (though since XTest isn't used by default, most users won't see any change from 0.6). In the XSendEvent case, crikey still doesn't send the KeyRelease event -- because some systems actually see it as another KeyPress. (Hey, what fun would computers be if they were consistent and always predictable, huh?)

Both C and Python versions are linked off the crikey page.

Tags: programming, crikey, X11, python
[ 14:50 May 25, 2008    More programming | permalink to this entry | comments ]

Setting app name and class in Xlib

I was looking at Dave's little phase-of-the-moon Mac application, and got the urge to play with moonroot, the little xlib ditty I wrote several years ago to put a moon (showing the right phase) on the desktop.

I fired it up, and got the nice moon-shaped window ... but with a titlebar. I didn't want that! Figuring out how to get rid of the titlebar in openbox was easy, just

<application name="moonroot">
    <decor>no</decor>
    <desktop>all</desktop>
</application>

... but it didn't work! A poke with xwininfo showed the likely cause: instead of "moonroot", the window was listed as "Unnamed window". Whoops!

A little poking around revealed three different ways to set "name" for a window: XStoreName, XSetClassHint (which sets both class name and app name), and XSetWMName. Available online documentation on these functions was not very helpful in explaining the differences; fortunately someone hanging out on the openbox channel knew the difference (thanks, Crazy_Hopper). Thus:

• XSetWMName sets a property called XA_NAME which is primarily used to update the window's titlebar. Note that this may be more than the app name (for instance, Firefox puts the title of the current page in the titlebar). To use XSetWMName, you have to set up and populate an XTextProperty structure, which first requires that you set up a string list then run XStringListToTextProperty -- not difficult but it's several annoying steps.
• XStoreName is a shortcut to XSetWMName, a way to set the XA_NAME (titlebar name) in one step.
• XSetClassHint sets two properties at once: a name hint and a class hint. This is the name and class that the window manager uses for directives like suppressing the titlebar.

I didn't see much in the way of example code for what an app ought to do with these, so I'll post mine here:

    char* appname;
    XClassHint* classHint;
[ ... ]
    if (argv && argc > 1)
        appname = basename(argv[0]);
    else
        appname = "moonroot";

    /* set the titlebar name */
    XStoreName(dpy, win, appname);

    /* set the name and class hints for the window manager to use */
    classHint = XAllocClassHint();
    if (classHint) {
        classHint->res_name = appname;
        classHint->res_class = "MoonRoot";
    }
    XSetClassHint(dpy, win, classHint);
    XFree(classHint);

And if anyone is interested in my silly moon program, it's at moonroot-0.3.tar.gz. moonroot gives you a large moon, moonroot -s gives a smaller one. I'm not terribly happy with its accuracy and wasted too much time today fiddling with it and verifying that it's doing the right time conversions. All I can figure is that the approximation in Meeus' Astronomical Algorithms is way too approximate (it's sometimes off by more than a day) and I should just rewrite all my moon programs to calculate moon phase the hard (and slow) way.

Tags: programming, xlib, astronomy
[ 20:15 Mar 18, 2008    More programming | permalink to this entry | comments ]

Script to add tags

Python is so cool. I love how I'll be working on a script and suddenly think "Oh, it should also do X, but I bet that'll be a lot more work", and then it occurs to me that I can do exactly that by adding about 2 more lines of python. And I add them and it works the first time.

Anyway, it turned out to be very easy to go through all existing blog articles and add tags for the current category hierarchy, being careful to preserve each file's last-modified date since that's what pyblosxom uses for the date of the entry. add-tags.py

Tags: blogging, programming
[ 18:37 Feb 29, 2008    More blogging | permalink to this entry | comments ]

Got tags working with pybloxsom

Entries on this blog are arranged by category. But all too often I have something that really belongs equally well in two categories. Since pyblosxom's categories follow the hierarchy on disk, there's no way to have an entry in two categories. Enter tags.

Tags are a way of assigning any number of keywords to each blog entry. Search engines apparently pay attention to tags, and most tagged blogs also let you search by tag.

I wanted my tags to follow whatever canonical tag format the big blogging sites use, so search engines would index them. Unfortunately, this isn't well documented anywhere. Wikipedia has a tags entry that mentions a couple of common formats; the HTML format given in that entry (<a rel="tag" ...>) turns out to be the format used on most popular sites like livejournal and blogspot, so that's what I wanted to use. Later, someone pointed me to a much better tag explanation on technorati, which is useful whether or not you decide to register with technorati.

Next: how to implement searching? The simplest pyblosxom tags plug-in is called simply tags.py. All the others are much more complex and do tons of things I'm not interested in. But tags.py doesn't support static mode, and points to a modified tags.py that's supposedly modified to work with static blogs.

Alas, when I tried that version, it didn't work (and an inquiry on the pybloxsom list got a response from someone who agreed it didn't work). So I hacked around and eventually got it working. Here's a diff for what I changed or just the tags-static.py plug-in.

Additional steps I needed that weren't mentioned in tags.py:

• Add "#tags foo,bar" directives as the second line of an entry, right under the title; anywhere else in the file it will be ignored.
• You may ned to create the tag directories http://yoursite/tags/$tagname yourself (pyblosxom created the directories for me on the web server, but not on the machine where I first tested).
• In addition to the config file entries discussed below, if you use an extension other than .txt (or maybe even if you don't) you also need to set py[ 'taggable_files' ] = [ "ext" ]
• In your story.html template, include $tag_links wherever you want the tags line to go. But make "Tags: " or something similar be part of the pretext, so it won't be included on un-tagged entries.

I also wrote a little python index.cgi for my blog's /tags directory, so you can see the list of tags used so far. Strangely, tags.py didn't create any such index, and it was easier to make a cgi than to figure out how to do it from a blosxom plug-in.

And as long as I'm posting pyblosxom diffs, here's the little filename diff for 1.4.3 that I apply to pyblosxom whenever I update it, to let me use the .blx extension rather than .txt for my blog source files. (That way I can configure my editor to treat blog files as html, which they are -- they aren't plaintext.)

Anyway, it all seems to be working now, and in theory I can tag all future articles. I'll probably go back and gradually add tags to older articles, but that's a bigger project and there's no rush.

Tags: blogging, programming
[ 15:04 Feb 27, 2008    More blogging | permalink to this entry | comments ]

PyTopo and PyGTK pixbuf memory leakage

On a recent Mojave desert trip, we tried to follow a minor dirt road that wasn't mapped correctly on any of the maps we had, and eventually had to retrace our steps. Back at the hotel, I fired up my trusty PyTopo on the East Mojave map set and tried to trace the road. But I found that as I scrolled along the road, things got slower and slower until it just wasn't usable any more.

PyTopo was taking up all of my poor laptop's memory. Why? Python is garbage collected -- you're not supposed to have to manage memory explicitly, like freeing pixbufs. I poked around in all the sample code and man pages I had available but couldn't find any pygtk examples that seemed to be doing any explicit freeing.

When we got back to civilization (read: internet access) I did some searching and found the key. It's even in the PyGTK Image FAQ, and there's also some discussion in a mailing list thread from 2003.

Turns out that although Python is supposed to handle its own garbage collection, the Python interpreter doesn't grok the size of a pixbuf object; in particular, it doesn't see the image bits as part of the object's size. So dereferencing lots of pixbuf objects doesn't trigger any "enough memory has been freed that it's time to run the garbage collector" actions.

The solution is easy enough: call gc.collect() explicitly after drawing a map (or any other time a bunch of pixbufs have been dereferenced).

So there's a new version of PyTopo, 0.6 that should run a lot better on small memory machines, plus a new collection format (yet another format from the packaged Topo! map sets) courtesy of Tom Trebisky.

Oh ... in case you're wondering, the ancient USGS maps from Topo! didn't show the road correctly either.

Tags: programming, python, gtk, pygtk
[ 21:21 Oct 12, 2007    More programming | permalink to this entry | comments ]

Egg Timer in Python and TkInter

I left the water on too long in the garden again. I keep doing that: I'll set up something where I need to check back in five minutes or fifteen minutes, then I get involved in what I'm doing and 45 minutes later, the cornbread is burnt or the garden is flooded.

When I was growing up, my mom had a little mechanical egg timer. You twist the dial to 5 minutes or whatever, and it goes tick-tick-tick and then DING! I could probably find one of those to buy (they're probably all digital now and include clocks and USB plugs and bluetooth ports) but since the problem is always that I'm getting distracted by something on the computer, why not run an app there?

Of course, you can do this with shell commands. The simple solution is:

(sleep 300; zenity --info --text="Turn off the water!") &

But the zenity dialogs are small -- what if I don't notice it? -- and besides, I have to multiply by 60 to turn a minute delay into sleep seconds. I'm lazy -- I want the computer to do that for me!
Update: Ed Davies points out that "sleep 5m" also works.

A slightly more elaborate solution is at. Say something like: at now + 15 minutes and when it prompts for commands, type something like:

export DISPLAY=:0.0
zenity --info --text="Your cornbread is ready"

to pop up a window with a message. But that's too much typing and has the same problem of the small easily-ignored dialogs. I'd really rather have a great big red window that I can't possibly miss.

Surely, I thought, someone has already written a nice egg-timer application! I tried aptitude search timer and found several apps such as gtimer, which is much more complicated than I wanted (you can define named events and choose from a list of ... never mind, I stopped reading there). I tried googling, but didn't have much luck there either (lots of Windows and web apps, no Linux apps or cross-platform scripts).

Clearly just writing the damn thing was going to be easier than finding one. (Why is it that every time I want to do something simple on a computer, I have to write it? I feel so sorry for people who don't program.)

I wanted to do it in python, but what to use for the window that pops up? I've used python-gtk in the past, but I've been meaning to check out TkInter (the gui toolkit that's kinda-sorta part of Python) and this seemed like a nice opportunity since the goal was so simple.

The resulting script: eggtimer. Call it like this:

eggtimer 5 Turn off the water

and in five minutes, it will pop up a huge red window the size of the screen with your message in big letters. (Click it or hit a key to dismiss it.)

First Impressions of TkInter

It was good to have an excuse to try TkInter and compare it with python-gtk. TkInter has been recommended as something normally installed with Python, so the user doesn't have to install anything extra. This is apparently true on Windows (and maybe on Mac), but on Ubuntu it goes the other way: I already had pygtk, because GIMP uses it, but to use TkInter I had to install python-tk.

For developing I found TkInter irritating. Most of the irritation concerned the poor documentation: there are several tutorials demonstrating very basic uses, but not much detailed documentation for answering questions like "What class is the root Tk() window and what methods does it have?" (The best I found -- which never showed up in google, but was referenced from O'Reilly's Programming Python -- was here.) In contrast, python-gtk is very well documented.

Things I couldn't do (or, at least, couldn't figure out how to do, and googling found only postings from other people wanting to do the same thing):

• Button didn't respond to any of the obvious keys, like Return or Space, and in fact setting key handlers on the button didn't work -- I ended up setting a key handler on the root window.
• I couldn't find a way to set the root window size and background explicitly, so I had to set approximate window size by guessing at the size of the internal padding of the button.
• There's an alternate to the root Tk() window called Toplevel, which is documented and does allow setting window size. Unfortunately, it also pops up an empty dialog without being told to (presumably a bug).
• All of the tutorials I found for creating dialogs was wrong, and I finally gave up on dialogs and just used a regular window.
• I couldn't fork and return control to the shell, because TkInter windows don't work when called from a child process (for reasons no one seems to be able to explain), so you have to run it in the background with & if you want your shell prompt back.

I expect I'll be sticking with pygtk for future projects. It's just too hard figuring things out with no documentation. But it was fun having an excuse to try something new.

Tags: programming, python, tkinter
[ 13:35 Sep 04, 2007    More programming | permalink to this entry | comments ]

Find the nearest matching color name

Someone showed up on #gimp today with a color specified as an HTML hex color specifier, and wanted to know how to find the nearest color name.

Easy, right? There have got to be a bazillion pages that do that, plus at least a couple of Linux apps.

But I googled for a while and couldn't find a single one. There are lots of pages that list all the RGB colors, or convert decimal red, green and blue into HTML #nnn hex codes, or offer aesthetic advice about pleasing combinations of colors for themes (including this lovely page on butterfly-inspired color themes, courtesy of Rik) but nothing I could find that gave color names. Apparently there used to be a Linux app that did that, a piece of Gnome 1 called GColorSel, but it's gone now.

I got to thinking (always dangerous!) ... /etc/X11/rgb.txt has a list of color names with their RGB color equivalents. It would be really easy to write something that just read down the list finding the ones closest to the specified color.

Uh-oh ... of course, once that thought occurred to me, I was doomed. Programmer's disease. I had to write it. So I did, and here it is: Find the Nearest Matching Color Name. It checks against both rgb.txt and the much smaller list of 17 CSS color names.

Tags: programming
[ 22:18 Jun 11, 2007    More programming | permalink to this entry | comments ]

A Culture of Regressions (or, Why I no longer work on Mozilla)

A couple of friends periodically pester me to write about why I stopped contributing to Mozilla after so many years with the project. I've held back, feeling like it's airing dirty laundry in public.

But a discussion on mozilla.dev.planning over the last week, started by Nelson Bolyard, aired it for me: it was their culture of regressions.

I love Mozilla technology. I'm glad it exists, and I still use it for my everyday browsing. But trying to contribute to Mozilla just got too frustrating. I spent more time chasing down and trying to fix other people's breakages than I did working on anything I wanted to work on.

That might be okay, barely, when you're getting paid for it. But when you're volunteering your own time, who wants to spend it fixing code checked in by some other programmer who just can't be bothered to clean up his own mess?

It's the difference between spending a day cleaning your own house ... and spending every day cleaning other people's houses.

Nelson said it eloquently in this exchange:

(Robert Kaiser writes)
As we are open source, everyone can access and test that code, and find and file the regressions, so that they get fixed over time.

(Boris Zbarsky writes)
That last conclusion doesn't necessarily follow. To get them fixed you need someone fixing them.

(Nelson Bolyard writes)
We're very unlikely to get volunteers to spent large amounts of effort, rewriting formerly working code to get it to work again, after it was broken by someone else's checkin. This demotivates developers and drives them away. They think "why should I keep working on this when others can break my code and I must pay for their mistakes?" and "I worked hard to get that working, and now person X has broken it. Let HIM fix it."

This was exactly how I felt, and it's the reason I quit working on Mozilla.

A little later in the thread, Boris Zbarsky reports that the trunk has been so broken with regressions that it's been unusable for him for weeks or months. (When you have someone as committed and sharp as Boris unable to use your software, you know there's something wrong with your project's culture.) He writes: "For example, on my machine (Linux) about one in three SVG testcases in Bugzilla causes trunk Gecko to hang X ..."

Justin Dolske replies, "Oh, Linux," and asks if it's related to turning on Cairo. Boris replies affirmatively. Just another example where a change was checked in that caused serious regressions keeping at least one important contributor from using the browser on a regular basis; yet it's still there and hasn't been backed out. Of course, it's "only Linux".

David Baron appears to take Nelson's concerns seriously, and suggests criteria for closing the tree and making everyone stop work to track down regressions. As he correctly comments, closing the tree is very serious and inefficient, and should be avoided in all but the most serious cases.

But Nelson repeats the real question:

(Nelson Bolyard writes)
Under what circumstances does a Sheriff back out a patch due to functional regressions? From what you wrote above, I gather it's "never". :(

Alas, the thread peters out after that; there's no reply to Nelson's question.

The problem with Mozilla isn't that there are regressions. Mistakes happen. The problem is that regressions never get fixed, because the project's culture encourages regressions. The prevailing attitude is that it's okay to check in changes that break other people's features, as long as your new feature is cool enough or the right people want it. If you break something, well, hey, someone will figure out a fix eventually. Or not. Either way, it's not your problem.

Working on new features is fun, and so is getting the credit for being the one to check them in. Fixing bugs, writing API documentation, extensive testing -- these things aren't fun, they're hard work, and there isn't much glory in them either (you don't get much appreciation or credit for it). So why do them if you don't have to? Let someone else worry about it, as long as the project lets you get away with it!

A project with a culture of responsibility would say that the person who broke something should fix it, and that broken stuff should stay out of the tree. If programmers don't do that themselves just because it's the right thing to do, the project could enforce it: just insist that regression-causing changes that can't be fixed right away be backed out. Fix the regressions out of the tree where they aren't causing problems for other people. Get help from people to test it and to integrate it with those other modules you forgot about the first time around.

Yes, even if it's a change that's needed -- even if it's something a lot of people want. If it's a good change, there will always be time to check it in later.

When it's really working.

Tags: programming
[ 10:07 May 29, 2007    More programming | permalink to this entry | comments ]

A Kitfox Extension

For a bit over a year I've been running a patched version of Firefox, which I call Kitfox, as my main browser. I patch it because there are a few really important features that the old Mozilla suite had which Firefox removed; for a long time this kept me from using Firefox (and I'm not the only one who feels that way), but when the Mozilla Foundation stopped supporting the suite and made Firefox the only supported option, I knew my only choice was to make Firefox do what I needed. The patches were pretty simple, but they meant that I've been building my own Firefox all this time.

Since all my changes were in JavaScript code, not C++, I knew this was probably all achievable with a Firefox extension. But never around to it; building the Mozilla source isn't that big a deal to me. I did it as part of my job for quite a few years, and my desktop machine is fast enough that it doesn't take that long to update and rebuild, then copy the result to my laptop.

But when I installed the latest Debian, "Etch", on the laptop, things got more complicated. It turns out Etch is about a year behind in its libraries. Programs built on any other system won't run on Etch. So I'd either have to build Mozilla on my laptop (a daunting prospect, with builds probably in the 4-hour range) or keep another system around for the purpose of building software for Etch. Not worth it. It was time to learn to build an extension.

There are an amazing number of good tutorials on the web for writing Firefox extensions (I won't even bother to link to any; just google firefox extension and make your own choices). They're all organized as step by step examples with sample code. That's great (my favorite type of tutorial) but it left my real question unanswered: what can you do in an extension? The tutorial examples all do simple things like add a new menu or toolbar button. None of them override existing Javascript, as I needed to do.

Canonical URL to the rescue. It's an extension that overrides one of the very behaviors I wanted to override: that of adding "www." to the beginning and ".com" or ".org" to the end of whatever's in the URLbar when you ctrl-click. (The Mozilla suite behaved much more usefully: ctrl-click opened the URL in a new tab, just like ctrl-clicking on a link. You never need to add www. and .com or .org explicitly because the URL loading code will do that for you if the initial name doesn't resolve by itself.) Canonical URL showed me that all you need to do is make an overlay containing your new version of the JavaScript method you want to override. Easy!

So now I have a tiny Kitfox extension that I can use on the laptop or anywhere else. Whee!

Since extensions are kind of a pain to unpack, I also made a source tarball which includes a simple Makefile: kitfox-0.1.tar.gz.

Tags: tech, web, mozilla, firefox, programming
[ 10:59 May 27, 2007    More tech/web | permalink to this entry | comments ]

Pho 0.9.5 Released

Pho 0.9.5-pre5 has been working nicely since I released it two weeks ago. And meanwhile, I've already started working on the next version. So I've released it as 0.9.5 with no changes (except for version string and some updates to the documentation and debian config files).

I made a .deb on Ubuntu Edgy, but haven't actually tested it yet (anyone who sees problems, please let me know) and I'll try to make a straight Debian package on Sarge sometime soon.

So what's this stuff I've been working on for the next version? Image categorization. I shoot so many photos, and categorizing them by keyword can be a lot of work. Although Pho's "Notes 0 through 9" are helpful for a small number of notes, it's tough keeping track of which note corresponds to which keyword when I'm categorizing a directory full of photos from a trip. The next Pho release (which will have a much shorter release cycle than 0.9.5 did, honest!) will have an optional Keywords dialog where you can type in keywords and associate them with photos. I know there are apps such as f-spot, gthumb and Picasa, but they all seem much more heavyweight than what I need, and Pho only needs a tiny bit of work to get there.

While I'm working on dialogs, I'm also cleaning up modality: Pho dialogs will now stay visible so they can't get lost behind the image, and the question dialog ("Really delete?" or "Do you want to quit?" will be modal.

But that's all coming in the next version. For now, 0.9.5 is the stable version: get it from the Pho page.

Tags: programming
[ 15:56 Mar 22, 2007    More programming | permalink to this entry | comments ]

Pho 0.9.5-pre5

Pho's been static for a long time -- it's been working well enough that I keep forgetting that there were a couple of bugs that need fixing for a 0.9.5 release.

I had some time tonight, so I dug in and fixed the bugs I remembered: some issues with zooming in and out, a bug with aspect ratio when switching out of fullscreen mode, and the fact that Note 0 didn't work.

While I was at it, I added an environment variable, PHO_ARGS, where you can preset your default values. I find that I always want -p (presentation mode), so now I can specify that with PHO_ARGS=p, and use pho -P when I want window borders. I also updated the man page.

I'll test this for a little while and if nobody finds any serious bugs, maybe I can finally release 0.9.5.

Get Pho here.

Tags: programming
[ 23:06 Mar 05, 2007    More programming | permalink to this entry | comments ]

Random Wallpaper

I was talking about desktop backgrounds -- wallpaper -- with some friends the other day, and it occurred to me that it might be fun to have my system choose a random backdrop for me each morning.

Finding backgrounds is no problem: I have plenty of images stored in ~/Backgrounds -- mostly photos I've taken over the years, with a smattering of downloads from sites like the APOD. So all I needed was a way to select one file at random from the directory.

This is Unix, so there's definitely a commandline way to do it, right? Well, surprisingly, I couldn't find an easy way that didn't involve any scripting. Some shells have a random number generator built in ($RANDOM in bash) but you still have to do some math on the result.

Of course, I could have googled, since I'm sure other people have written random-wallpaper scripts ... but what's the fun in that? If it has to be a script, I might as well write my own.

Rather than write a random wallpaper script, I wanted something that could be more generally useful: pick one random line from standard input and print it. Then I could pass it the output of ls -1 $HOME/Backgrounds, and at the same time I'd have a script that I could also use for other purposes, such as choosing a random quotation, or choosing a "flash card" question when studying for an exam.

The obvious approach is to read all of standard input into an array, count the lines, then pick a random number between one and $num_lines and print that array element. It took no time to whip that up in Python and it worked fine. But it's not very efficient -- what if you're choosing a line from a 10Mb file?

Then Sara Falamaki (thanks, Sara!) pointed me to a page with a neat Perl algorithm. It's Perl so it's not easy to read, but the algorithm is cute. You read through the input line by line, keeping track of the line number. For each line, the chance that this line should be the one printed at the end is the reciprocal of the line number: in other words, there's one chance out of $line_number that this line is the one to print.

So if there's only one line, of course you print that line; when you get to the second line, there's one chance out of two that you should switch; on the third, one chance out of three, and so on.

A neat idea, and it doesn't require storing the whole file in memory. In retrospect, I should have thought of it myself: this is basically the same algorithm I used for averaging images in GIMP for my silly Chix Stack Mars project, and I later described the method in the image stacking section of my GIMP book. To average images by stacking them, you give the bottom layer 100% opacity, the second layer 50% opacity, the third 33% opacity, and so on up the stack. Each layer makes an equal contribution to the final result, so what you see is the average of all layers.

The randomline script, which you can inspect here, worked fine, so I hooked it up to accomplish the original problem: setting a randomly chosen desktop background each day. Since I use a lightweight window manager (fvwm) rather than gnome or kde, and I start X manually rather than using gdm, I put this in my .xinitrc:

(xsetbg -fullscreen -border black `find $HOME/Backgrounds -name "*.*" | randomline`) &

Update: I've switched to using hsetroot, which is a little more robust than xsetbg. My new command is:

hsetroot -center `find -L $HOME/Backgrounds -name "*.*" | randomline`

So, an overlong article about a relatively trivial but nontheless nifty algorithm. And now I have a new desktop background each day. Today it's something prosaic: mud cracks from Death Valley. Who knows what I'll see tomorrow?

Update, years later: I've written a script for the whole job, randombg, because on my laptop I want to choose from a different set of backgrounds depending on whether I'm plugged in to an external monitor or using the lower resolution laptop display.
But meanwhile, I've just been pointed to the shuf command, which does pretty much what my randomline script did. So you don't actually need any scripts, just

hsetroot -fill `find ~/Images/Backgrounds/1680x1050/ -name '*.jpg' | shuf -n 1`

Tags: programming, pipelines, shell
[ 13:02 Feb 28, 2007    More programming | permalink to this entry | comments ]

PyTopo 0.5

Belated release announcement: 0.5b2 of my little map viewer PyTopo has been working well, so I released 0.5 last week with only a few minor changes from the beta. I'm sure I'll immediately find six major bugs -- but hey, that's what point releases are for. I only did betas this time because of the changed configuration file format.

I also made a start on a documentation page for the .pytopo file (though it doesn't really have much that wasn't already written in comments inside the script).

Tags: programming, python, gtk, pygtk, mapping
[ 21:10 Aug 25, 2006    More programming | permalink to this entry | comments ]

Javascript Warnings Everywhere

I'm working on some little Javascript demos (for a workshop at this summer's Get SET girls' technology camp) so I've had the Javascript Console up for most of my browsing over the last few days. I also have Mozilla's strict Javascript checking on (user_pref("javascript.options.strict", true); in prefs.js or user.js) since I don't want to show the girls code that generates warnings. (Strict mode also reports errors in CSS.)

It's been eye opening how many sites give warnings. You know that nice clean ultra-simple Google search page? One CSS error and one JS warning. But that's peanuts to the pages of errors I see on most sites, and they're not all missing "var" declarations. I have to hit the "Clear" button frequently if I want to be able to see the errors on the pages I'm working on.

And my own sites? Yes, I admit it, I've seen some errors in my own pages too. Though it makes me feel better that there aren't very many of them (mostly CSS problems, not JS). I'm going to keep the JS Console visible more often so I'll see these errors and correct them.

Tags: programming
[ 23:06 Jul 31, 2006    More programming | permalink to this entry | comments ]

Cleaner, More Flexible Python Map Viewing

A few months ago, someone contacted me who was trying to use my PyTopo map display script for a different set of map data, the Topo! National Parks series. We exchanged some email about the format the maps used.

I'd been wanting to make PyTopo more general anyway, and already had some hacky code in my local version to let it use a local geologic map that I'd chopped into segments. So, faced with an Actual User (always a good incentive!), I took the opportunity to clean up the code, use some of Python's support for classes, and introduce several classes of map data.

I called it 0.5 beta 1 since it wasn't well tested. But in the last few days, I had occasion to do some map exploring, cleaned up a few remaining bugs, and implemented a feature which I hadn't gotten around to implementing in the new framework (saving maps to a file).

I think it's ready to use now. I'm going to do some more testing: after visiting the USGS Open House today and watching Jim Lienkaemper's narrated Virtual Tour of the Hayward Fault, I'm all fired up about trying again to find more online geologic map data. But meanwhile, PyTopo is feature complete and has the known bugs fixed. The latest version is on the PyTopo page.

Tags: programming, python, gtk, pygtk, mapping
[ 17:25 Jun 03, 2006    More programming | permalink to this entry | comments ]

Aid for Java Victims

Over dinner, I glanced at the cover of the latest Dr. Dobb's (a new article on Ruby on Rails), then switched to BBC World News. The first Beeb headline was Aid flow begins for Java victims.

I guess I was a little distracted from dinner preparations ... my first thought was "Are they going to give them all copies of Ruby and Rails?"

Then, of course, I remembered the earthquake. Oh, right, those Java victims!

(Not to make light of the situation there, which sounds grim. And just as I was writing this, I got email from the USGS Earthquake Notification Service reporting another aftershock in Indonesia, this one magnitude 5.6. I hope it doesn't make matters worse.)

Tags: programming
[ 21:05 May 29, 2006    More programming | permalink to this entry | comments ]

MySQL Losing its Socket!

This morning I was all ready to continue working on an ongoing web project when I discovered that mysql wasn't running.

That's funny, it was running fine yesterday! I tried /etc/init.d/mysql start, but it failed. The only error message was, "Please take a look at the syslog."

So I hied myself over to /var/log, to discover that mysql.log and mysql.err were both there, but empty.

Some poking around /etc/mysql/my.cnf revealed that logging is commented out by default, because: "# Be aware that this log type is a performance killer."

I uncommented logging and tried again, but /var/log/mysql.err remained blank, and all that was in mysql.log was three lines related basically giving its runtime arguments and the name of the socket file.

Back to the drawing board. I was well aware that I had changed the mysql settings yesterday. See, mysqld on Ubuntu likes to create its socket as /var/run/mysqld/mysqld.sock, but other apps, like Ruby, all expect to find it in /tmp/mysql.sock. It's easy enough to change Ruby's expectations. But then I found out that although the cmdline client mysql also expects the socket in /var/run/mysqld, it depends on something called mysqladmin that wants the socket in /tmp. (I may have those two reversed. No matter: the point is that you can't use the client to talk to the database because it and the program it depends on disagree about the name of the socket. This is probably a Dapper bug.)

Okay, so I had to pick one. I decided that /tmp/mysql.sock was easier to remember and more standard with non-Debian setups. I knew where to change it in the server (/etc/mysql/my.cnf is there and well commented) but the mysql client doesn't use that, and it took some googling and help from clueful friends to find out that what it wanted was a new file called /etc/my.cnf (how's that for a nice clear configuration file name?) containing one line:

socket          = /tmp/mysql.sock

That done, mysql started and ran and everything worked. Woo!

Except that it didn't the following morning after a reboot, and didn't give any error messages as to why.

Off I went on a merry chase across init files: /etc/init.d/mysql calls /etc/mysql/debian-start (which made me realize that debian has added yet another config file, debian.cnf, which has yet another copy of the line specifying the socket filename) which calls /usr/share/mysql/debian-start.inc.sh as well as calling various other programs. But those were all red herrings: the trick to debugging the problem was to run mysqld directly (not via /etc/init.d/mysql start: it actually does print error messages, but they were being hidden by using the init.d script.

The real problem turned out to be that I had changed the location of the socket file, but not the pid file, in /etc/mysql/my.cnf, which was also located by default in /var/run/mysqld. Apparently that directory is created dynamically at each boot, and it isn't created unless it's needed for the socket file (whether the pid file needs it doesn't matter). So since I'd moved the socket file to /tmp, /var/run/mysqld wasn't created, mysqld couldn't create its pid file and it bailed. Solution: edit my.cnf to use /tmp for the pid file.

Tags: programming
[ 12:29 Mar 17, 2006    More programming | permalink to this entry | comments ]

Ruby on Rails on Ubuntu

Back when I laboriously installed Ruby and Rails on Ubuntu "Hoary Hedgehog" (which involved basically ignoring all the Ubuntu packages and building everything, including Ruby itself, from source), I was cheered by the notes in Ubuntu's forums and bugzilla indicating that as of the next release ("Breezy Badger") all the right versions would be there, and all this source compilation would no longer be necessary.

I didn't get around to trying it until today. Breezy and its successor "Dapper Drake" do indeed have a rails package as well as a Ruby package, and I happily installed them. All looked great -- until I actually tried to use them on a real-world application. It turns out that the Ruby and Rails packages don't include gems, Ruby's package manager (similar to the CPAN system familiar to Perl programmers). And gems is required for doing anything useful in Rails.

Drat! After several false starts, I eventually found the instructions on this page. Except that installs way more than seems necessary for what I need to do, and if you copy/paste lines from that page you may end up with a bunch of packages you don't want, like an out of date version of mysql.

So here are simplified instructions for using Ruby on Rails on Ubuntu Breezy or Dapper.

As yourself:

wget http://rubyforge.org/frs/download.php/5207/rubygems-0.8.11.tgz
tar zxvf rubygems-0.8.11.tgz

As root:

cd rubygems-0.8.11
ruby setup.rb
gem install rubygems-update
gem install rails

Say yes to all dependency questions during the gem install of rails. Add your web server and database of choice (you probably already have them installed, anyway) and you should be good to go.

You may note that the page I referenced tells you to install two versions of rails: the Ubuntu package plus the one from gems. At least on Dapper, you don't need both. Installing rails pulls in the packages:
irb irb1.8 libpgsql-ruby1.8 libreadline-ruby1.8 libredcloth-ruby1.8 libruby1.8 rake rdoc rdoc1.8 ruby ruby1.8
I haven't experimented with which of these packages are and are not needed. If you run into problems, some set of packages from this list may solve them.

Update: it seems that none of these are required. Many of them are dummy packages anyway, which contain no files related to the actual package name. For instance, the rake package contains, not rake, but a single bash completion file related to rake. So you should be fine ignoring all of them, installing just Ruby and nothing else.

(I filed bug 34840 requesting that Ubuntu ship a usable version of Rails, since it didn't seem to be filed already.)

Tags: programming
[ 21:56 Mar 13, 2006    More programming | permalink to this entry | comments ]

Set C Styles Automatically in Emacs

I just figured out a nifty emacs trick: how to set C styles automatically based on the location of the file I'm editing. That way, emacs will autoindent with the proper style whether I'm editing my own code, GIMP code, Linux kernel code, etc. assuming that I keep the source for these projects in predictable places.

First, define a derived mode for each style you might want to use. Each minor mode inherits from c-mode, but adds a c-set-style call. (Obviously, this assumes that the styles you want to use are already defined.)

(define-derived-mode gnu-c-mode c-mode "GNU C mode"
  (c-set-style "gnu"))
(define-derived-mode linux-c-mode c-mode "GNU C mode"
  (c-set-style "linux"))

Then add entries to your auto-mode-alist for each directory you want to treat specially:

(setq auto-mode-alist
[ . . . ]
      (cons '("gimp.*/" . gnu-c-mode)
      (cons '("linux-.*/" . linux-c-mode)
            auto-mode-alist) ) )

Your normal default style will still work for any files not matching the patterns specified in auto-mode-alist.

It might be possible to skip the step of defining derived modes by switching on the file's pathname in the C mode hook, but I've never figured out how to get the pathname of the file being loaded in any reliable way.

Tags: programming
[ 12:42 Sep 30, 2005    More programming | permalink to this entry | comments ]

New Pho

Pho 0.9.5-pre4 seems to be working pretty well and fixes a couple of bugs in pre3, so I posted a tarball. I really need to quit this pre- stuff and just release 0.9.5. Soon, really!

Tags: programming
[ 11:51 Jul 26, 2005    More programming | permalink to this entry | comments ]

A Fast Volume Control App

I updated my Debian sid system yesterday, and discovered today that gnome-volume-control has changed their UI yet again. Now the window comes up with two tabs, Playback and Capture; the default tab, Playback, has only one slider in it, PCM, and all the important sliders, like Volume, are under Capture. (I'm told this is some interaction with how ALSA sees my sound chip.)

That's just silly. I've never liked the app anyway -- it takes forever to come up, so I end up missing too much of any clip that starts out quiet. All I need is a simple, fast window with a single slider controlling master volume. But nothing like that seems to exist, except panel applets that are tied to the panels of particular window managers.

So I wrote one, in PyGTK. vol is a simple script which shows a slider, and calls aumix under the hood to get and set the volume. It's horizontal by default; vol -h gives a vertical slider.

Aside: it's somewhat amazing that Python has no direct way to read an integer out of a string containing more than just that integer: for example, to read 70 out of "70,". I had to write a function to handle that. It's such a terrific no-nonsense language most of the time, yet so bad at a few things. (And when I asked about a general solution in the python channel at [large IRC network], I got a bunch of replies like "use int(str[0:2])" and "use int(str[0:-1])". Shock and bafflement ensued when I pointed out that 5, 100, and -27 are all integers too and wouldn't be handled by those approaches.)

Tags: programming, python, gtk, pygtk
[ 14:54 Jun 21, 2005    More programming | permalink to this entry | comments ]

PyTopo 0.3

I needed to print some maps for one of my geology class field trips, so I added a "save current map" key to PyTopo (which saves to .gif, and then I print it with gimp-print). It calls montage from Image Magick.

Get yer PyTopo 0.3 here.

Tags: programming, python, imagemagick, mapping
[ 16:56 Apr 13, 2005    More programming | permalink to this entry | comments ]

Python Expose vs. Focus

A few days ago, I mentioned my woes regarding Python sending spurious expose events every time the drawing area gains or loses focus.

Since then, I've spoken with several gtk people, and investigated several workarounds, which I'm writing up here for the benefit of anyone else trying to solve this problem.

First, "it's a feature". What's happening is that the default focus in and out handlers for the drawing area (or perhaps its parent class) assume that any widget which gains keyboard focus needs to redraw its entire window (presumably because it's locate-highlighting and therefore changing color everywhere?) to indicate the focus change. Rather than let the widget decide that on its own, the focus handler forces the issue via this expose event. This may be a bad decision, and it doesn't agree with the gtk or pygtk documentation for what an expose event means, but it's been that way for long enough that I'm told it's unlikely to be changed now (people may be depending on the current behavior).

Especially if there are workarounds -- and there are.

I wrote that this happened only in pygtk and not C gtk, but I was wrong. The spurious expose events are only passed if the CAN_FOCUS flag is set. My C gtk test snippet did not need CAN_FOCUS, because the program from which it was taken, pho, already implements the simplest workaround: put the key-press handler on the window, rather than the drawing area. Window apparently does not have the focus/expose misbehavior.

I worry about this approach, though, because if there are any other UI elements in the window which need to respond to key events, they will never get the chance. I'd rather keep the events on the drawing area.

And that becomes possible by overriding the drawing area's default focus in/out handlers. Simply write a no-op handler which returns TRUE, and set it as the handler for both focus-in and focus-out. This is the solution I've taken (and I may change pho to do the same thing, though it's unlikely ever to be a problem in pho).

In C, there's a third workaround: query the default focus handlers, and disconnect() them. That is a little more efficient (you aren't calling your nop routines all the time) but it doesn't seem to be possible from pygtk: pygtk offers disconnect(), but there's no way to locate the default handlers in order to disconnect them.

But there's a fourth workaround which might work even in pygtk: derive a class from drawing area, and set the focus in and out handlers to null. I haven't actually tried this yet, but it may be the best approach for an app big enough that it needs its own UI classes.

One other thing: it was suggested that I should try using AccelGroups for my key bindings, instead of a key-press handler, and then I could even make the bindings user-configurable. Sounded great! AccelGroups turn out to be very easy to use, and a nice feature. But they also turn out to have undocumented limitations on what can and can't be an accelerator. In particular, the arrow keys can't be accelerators; which makes AccelGroup accelerators less than useful for a widget or app that needs to handle user-initiated scrolling or movement. Too bad!

Tags: programming, python, gtk, pygtk
[ 20:52 Apr 09, 2005    More programming | permalink to this entry | comments ]

PyTopo is usable; pygtk is inefficient

While on vacation, I couldn't resist tweaking pytopo so that I could use it to explore some of the areas we were visiting.

It seems fairly usable now. You can scroll around, zoom in and out to change between the two different map series, and get the coordinates of a particular location by clicking. I celebrated by making a page for it, with a silly tux-peering-over-map icon.

One annoyance: it repaints every time it gets a focus in or out, which means, for people like me who use mouse focus, that it repaints twice for each time the mouse moves over the window. This isn't visible, but it would drag the CPU down a bit on a slow machine (which matters since mapping programs are particularly useful on laptops and handhelds).

It turns out this is a pygtk problem: any pygtk drawing area window gets spurious Expose events every time the focus changes (whether or not you've asked to track focus events), and it reports that the whole window needs to be repainted, and doesn't seem to be distinguishable in any way from a real Expose event. The regular gtk libraries (called from C) don't do this, nor do Xlib C programs; only pygtk.

I filed bug 172842 on pygtk; perhaps someone will come up with a workaround, though the couple of pygtk developers I found on #pygtk couldn't think of one (and said I shouldn't worry about it since most people don't use pointer focus ... sigh).

Tags: programming, python, gtk, pygtk, mapping
[ 16:26 Apr 06, 2005    More programming | permalink to this entry | comments ]

Python GTK Topographic Map Program

I couldn't stop myself -- I wrote up a little topo map viewer in PyGTK, so I can move around with arrow keys or by clicking near the edges. It makes it a lot easier to navigate the map directory if I don't know the exact starting coordinates.

It's called PyTopo, and it's in the same place as my earlier two topo scripts.

I think CoordsToFilename has some bugs; the data CD also has some holes, and some directories don't seem to exist in the expected place. I haven't figured that out yet.

Tags: programming, python, gtk, pygtk, mapping
[ 17:53 Mar 27, 2005    More programming | permalink to this entry | comments ]

Topographic Maps for Linux

I've long wished for something like those topographic map packages I keep seeing in stores. The USGS (US Geological Survey) sells digitized versions of their maps, but there's a hefty setup fee for setting up an order, so it's only reasonable when buying large collections all at once.

There are various Linux mapping applications which do things like download squillions of small map sections from online mapping sites, but they're all highly GPS oriented and I haven't had much luck getting them to work without one. I don't (yet?) have a GPS; but even if I had one, I usually want to make maps for places I've been or might go, not for where I am right now. (I don't generally carry a laptop along on hikes!)

The Topo! map/software packages sold in camping/hiking stores (sometimes under the aegis of National Geographic are very reasonably priced. But of course, the software is written for Windows (and maybe also Mac), not much help to Linux users, and the box gives no indication of the format of the data. Googling is no help; it seems no Linux user has ever tried buying one of these packages to see what's inside. The employees at my local outdoor equipment store (Mel Cotton's) were very nice without knowing the answer, and offered the sensible suggestion of calling the phone number on the box, which turns out to be a small local company, "Wildflower Productions", located in San Francisco.

Calling Wildflower, alas, results in an all too familiar runaround: a touchtone menu tree where no path results in the possibility of contact with a human. Sometimes I wonder why companies bother to list a phone number at all, when they obviously have no intention of letting anyone call in.

Concluding that the only way to find out was to buy one, I did so. A worthwhile experiment, as it turned out! The maps inside are simple GIF files, digitized from the USGS 7.5-minute series and, wonder of wonders, also from the discontinued but still useful 15-minute series. Each directory contains GIF files covering the area of one 7.5 minute map, in small .75-minute square pieces, including pieces of the 15-minute map covering the same area.

A few minutes of hacking with python and Image Magick resulted in a script to stitch together all images in one directory to make one full USGS 7.5 minute map; after a few hours of hacking, I can stitch a map of arbitrary size given start and end longitude and latitude. My initial scripts, such as they are.

Of course, I don't yet have nicities like a key, or an interactive scrolling window, or interpretation of the USGS digital elevation data. I expect I have more work to do. But for now, just being able to generate and print maps for a specific area is a huge boon, especially with all the mapping we're doing in Field Geology class. GIMP's "measure" tool will come in handy for measuring distances and angles!

Tags: programming, python, gtk, pygtk
[ 11:13 Mar 27, 2005    More programming | permalink to this entry | comments ]

Kitfox: Firefox for People Who Liked the Mozilla Browser

Debate rages on the mozilla-seamonkey list since the Mozilla Foundation announced that there would be no 1.8 release of the Mozilla browser (also called "the suite", or by its code name, "seamonkey"). Suite users are frustrated at lack of notice: anyone who was paying attention knew that seamonkey was going to be dropped eventually, but everyone expected at least a 1.8 final release. Mozilla.org is frustrated because they wish suite users would quit whining and switch to Firefox. Various people are slinging flames and insults, while a few try to mediate with logic and sense. There's a volunteer effort ramping up to continue support for the suite, but no plans for what to do about fixing all the regressions. Go read the list if you want all the gory details.

Anyway, the writing on the wall (and on the newsgroup) is clear: if you want a browser with continuing support from mozilla.org, Firefox is your only choice. Unfortunately for Linux users, firefox is designed by and for Windows users, copying Internet Explorer's user interface and dropping support for a number of nice features which the old mozilla browser offered.

I've decided that the best way to get a usable browser is to take firefox and put back the mozilla features that I miss. Mostly these are easy user interface tweaks. I pulled a tree last week and had most of the items that were blocking me addressed in a few hours. Building wasn't entirely straightforward: the build page doesn't make all the options clear, like the fact that xft and freetype are both enabled by default, so one of them has to be explicitly disabled. Updating the tree turns out to be a bit problematic: firefox' build dependencies turn out to be dicy, so sometimes changing a single .xul file causes the entire tree to rebuild, while other times an update builds a few files and the resulting build fails to run, and requires a clobber and a rebuild. Still, those problems are relatively minor.

So far, I have fixes for these bugs:

• bug 233853 : Ctrl-enter in the urlbar should open in a new tab, like ctrl-click does on a link. (Firefox uses ctrl-click but alt-enter.) I have no use for the IE ctrl-enter urlbar behavior (add .com or something to whatever is there, so if you're trying to search for linux browser you get "http://linux browser.com" instead of a new tab with a google search for linux browser). Good UI design dictates that the same modifier should be used for the same function, as it was in mozilla.
• bug 245015 : Shift-click should do Save As. Another thing I do many times a day, and Firefox offers no shortcut for it at all.
• Bugs 234110 and 66834: Disable the urlbar dropdown that flickers distractingly as I type, and steals system focus so that I can't do anything outside the firefox window until I dismiss the dropdown.
• Add a binding for ctrl-Q so I can quit the browser easily from the keyboard. (I can't find the relevant bug number.)

Next up: try to figure out why firefox takes so much longer than mozilla to start up. Fortunately, once it's up, it seems just as fast at browsing, but startup takes forever, and firefox doesn't even offer a splash option to tell me that something is happening.

Here is my patch, in case anyone else is bothered by these issues.

Perhaps this could be built as an extension. Some day I'll look into that. Certainly the current set of patches could be implemented as a script which exploded, edited, and re-packed the .jar archives in a firefox binary build, since the patch touches no C++ code as yet.

I'm calling my firefox-derived browser "Kitfox".

Tags: programming
[ 11:03 Mar 16, 2005    More programming | permalink to this entry | comments ]

New Crikey!

Crikey 0.5 is out, with some changes contributed by Efraim Feinstein -- it can read from stdin now, and has a debug flag. Reading from stdin means you can generate multi-line text now.

It's so cool when people send patches to my programs! Especially when they're nice clean code implementing useful features. Thanks, Efraim!

Tags: programming
[ 21:39 Nov 09, 2004    More programming | permalink to this entry | comments ]

Pho 0.9.5-pre3: evil Metacity window sizing

It turns out that the problem with pho windows not resizing in metacity is this: when metacity sees a window that's slightly larger (in either dimension) than the screen size, it unilaterally makes that window maximized, and thereafter refuses any request from the app to resize the window smaller.

Mandatory maximize might actually be useful in some circumstances (anyone who's ever tried to run on an 800x600 laptop has doubtless seen dialogs which don't fit on the screen) but the subsequent refusal to resize makes little sense, and causes bustage in programs which work fine under other window managers.

A workaround is for pho to unmaximize before any window resize. This would be a bummer with an app where the user might click the maximize button manually; with pho, that's unlikely (I hope) because anyone who wants to run maximized is better off running in fullscreen/presentation mode (which now finally sets its background to black, hooray).

Get'cher Pho 0.9.5-pre3 here.

Tags: programming
[ 18:21 Oct 10, 2004    More programming | permalink to this entry | comments ]

New showpix script now uses PHP

I'd been meaning for ages to write a PHP version of my showpix.cgi Perl script, to show images without needing a separate .html file generated for each image. I finally did it this morning, and it was much easier than I expected, and seems to run a lot faster than the perl CGI (not surprising, since PHP is cached in our web server and perl isn't; so this should be more scaleable and less load on the server).

The hardest part was writing the Python script to generate a new showpix.php for a directory of images, and that only because of all the escaping of quotes that needed to be done when telling python to print a line that tells php to print a line to serve up over http ...

Anyway, I've converted the Flume Trail images to use the new PHP stuff, and I've updated the page for the Imagebatch scripts to include PHP ability.

Tags: programming
[ 14:40 Oct 01, 2004    More programming | permalink to this entry | comments ]

Pho 0.9.5-pre2

I finally gathered together some of the pending fixes for Pho, so it keeps better track of where it is in the linked list when deleting or jumping around. No progress on the focus handling, though. I sure would like to solve that before 0.9.5 final.

Tags: programming
[ 21:42 Sep 30, 2004    More programming | permalink to this entry | comments ]

ADA doesn't apply to the web?!

The 11th Circuit Court of Appeals ruled last Friday that the Americans with Disabilities Act does not apply to the web. In other words, web designers are not required to make their pages accessible to readers with disabilities.

That seems very odd. It's required of software -- several lawsuits have been filed (and settled) related to inaccessible software. Accessibility is required universally for web sites in the EU, and I thought it was required of US government sites as well. It's a shame the court felt that it wasn't important for web sites in general.

Accessibility helps everybody, not just people with disabilities (even setting aside the fact that most of us will eventually experience some impairment, if we live long enough). Accessible web sites are usually easier to read and navigate, and translate more easily to offline readers (such as Sitescooper) and PDA readers (such as Opera and Plucker).

The decision was "largely on procedural grounds" and the court suggested that the decision could be revisited in a future case. I hope that case comes soon, before US web designers conclude there's no point in designing for accessibility.

W3C Web Content Accessibility Guidelines 2.0.

Tags: programming
[ 12:05 Sep 28, 2004    More programming | permalink to this entry | comments ]

XShapeCombineMask generates an Expose loop

I tried moonroot on blackbird today, under icewm for the first time. It went into an infinite expose/redraw loop. It turns out that XShapeCombineMask (the call that sets the shaped window's shape mask) generates an extra Expose, which of course happens asynchronously so disabling expose handling in the draw routine doesn't help.

What does help is maintaining a static variable to ensure that it only shapes the window the first time, and not on subsequent draws.

I also tweaked sonypid.c a bit -- 2.4.25 is generating two jogdial-release events whenever the machine resumes from bios suspend. But there's no jogdial-press event corresponding, so I fixed sonypid to ignore jogdial release unless there's already been a jogdial press (again, maintaining a static variable; I already had one so that it doesn't trigger a release after an UP+PRESSED or DOWN+PRESSED event, so I just had to tweak that code a little). That should eliminate that annoying paste that was happening every time I resumed from suspend.

Wish sonypi would quit changing, though I shouldn't complain since it's also good to see that it's still being worked on.

Tags: programming
[ 22:55 Jul 25, 2004    More programming | permalink to this entry | comments ]

Pho Makefile now builds under either gtk1 or gtk2

I fixed the pho Makefile to build under either gtk1 or gtk2, which required some grody bash code. I wish I knew of a better way to do that. Aren't massive API changes fun? Whee!

Patti told me about a park I didn't know about: the Ulistac Natural Area. I drove by it today after Toastmasters. It's a very small linear park along a levee, with one wooded area and a longer and skinnier open area. I didn't go in -- the combination of migraine and hot sun sapped my interest in a walk.

Tags: programming
[ 19:00 Jul 14, 2004    More programming | permalink to this entry | comments ]

Rootmoon

I finally whipped up an app I've been thinking about for a while: a little moon drawn with a shaped window, so you can put it on your root like that OS X moondock applet that Dave uses. Partly it was an excuse to play with shaped windows, which I hadn't used before, and partly it's that I hate seeing stuff that OS X can do that Linux can't.

I put it up (currently temp-named "moonroot" on my software page.

I also caught some nice shots of a hummer at the feeder (through the screen) and added them to my Hummingbird Photo page. A fairly productive day, really, including a nice hike.

Tags: programming
[ 19:00 Jul 11, 2004    More programming | permalink to this entry | comments ]

Pho and window managers

I worked some more on pho yesterday, trying to fix some of the problems Dave has been seeing in other window managers, particularly Metacity. I got rid of most of the problems, but metacity still asks him to place and size the initial window. I have no idea why; I'm specifying initial window size, and no other program asks him that. I also haven't solved the problem of the window getting focus after it resizes out from under the mouse cursor (in lots of window managers, such as openbox). Typing that, I just got a brainstorm: maybe it's a race condition, that the focus loss doesn't happen until after expose, but the "ask for focus" happens before the resize has propogated through the X server. Must try!

Tags: programming
[ 14:00 Jul 08, 2004    More programming | permalink to this entry | comments ]

Rewrote Pho

I finally got around to rewriting pho! The code is much cleaner now -- images are stored as structs in a linked list, no more motley collections of weird global arrays. Code is factored better, and it builds for gtk2. But the main motivation for rewriting it was to have it make a new window for each new or resized image, replacing the old window, to solve a bunch of window manager bugs I keep hitting:

• Some window managers (kwm, xfce4) don't correctly resize a small window to be bigger, or if they do, the contents of the bigger window only repaint over the area that was covered by the old size.
• Some window managers (xfce4, openbox) under pointer focus will lose window focus if a window is resized or moved to a place where it's no longer under the mouse, even if the window had focus before.

I handed the new pho to Dave for testing, and he hates the "new window each time" model; it takes too long in the window managers he runs. He says he wasn't that bothered by the repainting/resizing problems.

Fortunately, the rewrite factored the code so that it should be easy to provide both options (that was the plan anyway), isolated in the NewWindow routine. So I'll put back the "resize and reposition existing window" code, as a switchable option, and maybe try to grab focus to solve the pointer focus issues that have been plaguing me. I don't know what to do about the window manager resize/repaint issues; more research is required.

Tags: programming
[ 22:00 Jul 07, 2004    More programming | permalink to this entry | comments ]