Shallow Thoughts

Configuring git colors

I spent a morning wrestling with git after writing a minor GIMP fix that I wanted to check in. Deceptively simple ideas, like "Check the git log to see the expected format of check-in messages", turned out to be easier said than done.

Part of the problem was git's default colors: colors calculated to be invisible to anyone using a terminal with dark text on a light background. And that sent me down the perilous path of git configuration.

git-config does have a manual page. But it lacks detail: you can't get from there to knowing what to change so that the first line of commits in git log doesn't show up yellow.

But that's okay, thought I: all I need to do is list the default settings, then change anything that's a light color like yellow to a darker color. Easy, right?

Well, no. It turns out there's no way to get the default settings -- because they aren't part of git's config; they're hardwired into the C code.

But you can find most of them with a seach for GIT_COLOR in the source. The most useful lines are these the ones in diff.c, builtin-branch.c and wt-status.c.

gitconfig

The next step is to translate those C lines to git preferences, something you can put in a .gitconfig. Here's a list of all the colors mentioned in the man page, and their default values -- I used "normal" for grep and interactive where I wasn't sure of the defaults.

[color "diff"]
	plain = normal
	meta = bold
	frag = cyan
	old = red
	new = green
	commit = yellow
	whitespace = normal red
[color "branch"]
	current = green
	local = normal
	remote = red
	plain = normal
[color "status"]
	header = normal
	added = red
	updated = green
	changed = red
	untracked = red
	nobranch = red
[color "grep"]
	match = normal
[color "interactive"]
	prompt = normal
	header = normal
	help = normal
	error = normal

The syntax and colors are fairly clearly explained in the manual: allowable colors are normal, black, red, green, yellow, blue, magenta, cyan and white. After the foreground color, you can optionally list a background color. You can also list an attribute, chosen from bold, dim, ul, blink and reverse -- only one at a time, no combining of attributes.

So if you really wanted to, you could say something like

[color "status"]
	header = normal blink
	added = magenta yellow
	updated = green reverse
	changed = red bold
	untracked = blue white
	nobranch = red white bold

Minimal changes for light backgrounds

What's the minimum you need to get everything readable? On the light grey background I use, I needed to change the yellow, cyan and green entries:

[color "diff"]
	frag = cyan
	new = green
	commit = yellow
[color "branch"]
	current = green
[color "status"]
	updated = green

Disclaimer: I haven't tested all these settings -- because I haven't yet figured out where all of them apply. That's another area where the manual is a bit short on detail ...

Tags: git, programming, color
[ 22:26 Feb 02, 2010    More programming | permalink to this entry ]

On Linux Planet: a simple Poker game in Python-Qt

I've written in the past about Python GUI programming using the GTK and Tk toolkits, and several KDE fans felt that I was slighting the much nicer looking Qt.

So my latest article on Linux Planet, Make Pretty GUI Apps Fast with Python-Qt, shows how to develop a little poker game using the python-qt toolkit.

I didn't want to dwell on it in the article (and didn't have space anyway), but pyqt turned out to be a bit of a pain. There's no official documentation -- or at least nothing that's obviously official -- and a lot of the examples on google are out of date because of API changes. None of the tutorial examples explain much, and they never demonstrate the practical features I'd want to do in a real app. It was surprisingly hard to come up with an application idea that worked well, looked good and was still easy to explain.

And don't get me started on this whole "Slots and signals are revolutionarily different even though they look just like the callbacks every other toolkit has used for the last three decades" meme. I'm sure there is a subtle technical difference -- but if there's a difference that matters to the average UI programmer, their documentation sure doesn't make it clear.

All that aside, PyQt (and Qt in general) does produce very pretty apps and is worth trying for that reason.

The suit images in the article are adapted from some suits I found on Wikimedia Commons (the "Naipe" set). I wanted them to look more 3-dimensional, so I applied my blobipy GIMP script as well as scaling and resizing them. I really liked those shiny-looking Tango heart and spade emblems (also on the Wikimedia Commons page) but I couldn't find a diamond or club to match.

The poker program I wrote has menus and a second round of dealing, where you can mark off the cards you want to keep. I couldn't fit all that in a 700-word article, but the complete program is available here: qpoker.py or you can get it in a tarball along with the suit images at qpoker.tar.gz.

Tags: programming, writing, qt, python-qt
[ 09:53 Jan 28, 2010    More programming | permalink to this entry ]

Displaying images from Javascript file inputs

(despite Firefox's attempts to prevent that)

My Linux Planet article last week was on printing pretty calendars. But I hit one bug in Photo Calendar. It had a HTML file chooser for picking an image ... and when I chose an image and clicked Select to use it. it got the pathname wrong every time.

I poked into the code (Photo Calendar's code turned out to be exceptionally clean and well documented) and found that it was expecting to get the pathname from the file input element's value attribute. But input.File.value was just returning the filename, foo.jpg, instead of the full pathname, /home/user/Images/yosemite/foo.jpg. So when the app tried to make it into a file:/// URL, it ended up pointing to the wrong place.

It turned out the cause was a security change in Firefox 3. The issue: it's considered a security hole to expose full pathnames on your computer to Javascript code coming from someone else's server. The Javascript could give bad guys access to information about the directory structures on your disk. That's a perfectly reasonable concern, and it makes sense to consider it as a security hole.

The problem is that this happens even when you're running a local app on your local disk. Programs written in any other language and toolkit -- a Python program using pygtk, say, or a C++ Qt program -- have access to the directories on your disk, but you can't use Javascript inside Firefox to do the same thing. The only ways to make an exception seems to be an elaborate procedure requiring the user to change settings in about:config. Not too helpful.

Perhaps this is even reasonable, given how common cross-site scripting bugs have been in browsers lately -- maybe running a local script really is a security risk if you have other tabs active. But it leaves us with the problem of what to do about apps that need to do things like choose a local image file, then display it.

And it turns out there is: a data URL. Take the entire contents of the file (ouch) and create a URL out of those contents, then set the src attribute of the image to that.

Of course, that makes for a long, horrifying, unreadable URL -- but the user never has to see that part. I suspect it's also horribly memory intensive -- the image has to be loaded into memory anyway, to display it, but is Firefox also translating all of that to a URL-legal syntax? Obviously, any real app using this technique had better keep an eye on memory consumption. But meanwhile, it fixes Photo Calendar's file button.

Here's what the code looks like:

  img = document.getElementById("pic");
  fileinput = document.input.File;
  if (img && fileinput)
    img.src = fileinput.files[0].getAsDataURL();

Here's a working minimal demo of using getAsDataURL() with a file input.

Tags: javascript, web, programming
[ 13:57 Jan 17, 2010    More programming | permalink to this entry ]

Python-GTK regression: How to catch mouse button release

We just had the second earthquake in two days, and I was chatting with someone about past earthquakes and wanted to measure the distance to some local landmarks. So I fired up PyTopo as the easiest way to do that. Click on one point, click on a second point and it prints distance and bearing from the first point to the second.

Except it didn't. In fact, clicks weren't working at all. And although I have hacked a bit on parts of pytopo (the most recent project was trying to get scaling working properly in tiles imported from OpenStreetMap), the click handling isn't something I've touched in quite a while.

It turned out that there's a regression in PyGTK: mouse button release events now need you to set an event mask for button presses as well as button releases. You need both, for some reason. So you now need code that looks like this:

drawing_area.connect("button-release-event", button_event)
drawing_area.set_events(gtk.gdk.EXPOSURE_MASK |
                        # next line wasn't needed before:
                        gtk.gdk.BUTTON_PRESS_MASK |
                        gtk.gdk.BUTTON_RELEASE_MASK )

An easy fix ... once you find it.

I filed bug 606453 to see whether the regression was intentional.

I've checked in the fix to the PyTopo svn repository on Google Code. It's so nice having a public source code repository like that! I'm planning to move Pho to Google Code soon.

Tags: programming, python, pygtk, mapping
[ 13:20 Jan 08, 2010    More programming | permalink to this entry ]

Debug logging in Javascript under Firefox

While debugging Javascript, I've occasionally come across references to a useful function called console.log. Supposedly you can log errors with a line like:

  console.log(""key press, char code " + e.charCode);

Since the only native way of logging debug statements in Javascript is with a pop-up alert() box, having a less obtrusive way to print is something any JS programmer could use.

The catch? It didn't do anything -- except print console is not defined.

Today a friend was crowing about how wonderful Javascript debugging was in Google's Chrome browser -- because it has functions like console.log.

After some searching and poking around, we determined that Firefox also has console.log -- it's just well hidden and a bit hard to get going.

First, you need the Firebug extension. If you're developing Javascript, you probably already have it. If not, you need it.

Run Firebug and click to the Console tab. Now click on the tiny arrow that shows up at the right edge of that tab, as shown. Turns out there's a whole menu of options under there -- one of which is Enabled.

But wait, that's not all. In my case, the console was already Enabled according to the menu. To get the console working, I had to

1. Disable the console
2. Re-enable it
3. Shift-reload the page being debugged

My friend also said that if she didn't enable the console in Firebug, then her script died when she called console.log. That didn't happen for me -- all that happened was that I got error messages in the error console (the one accessed from Firefox's Tools menu, different from the Firebug console). But it's a good idea to check for its existence if you're going to use debugging statements in your code. Like this:

  if (typeof console != "undefined") {
    console.log( "key press, char code " + e.charCode
                + ", key code " + e.keyCode
                + ", " + e.ctrlKey + ", " + e.altKey
		+ ", " + e.metaKey );
  }

Here are some more things you can do with Firebug's console.

Tags: firefox, javascript, programming, tips
[ 15:41 Nov 28, 2009    More tech/web | permalink to this entry ]

Character Sets and Encodings in Linux, part 2

Continuing the discussion of those funny characters you sometimes see in email or on web pages, today's Linux Planet article discusses how to convert and handle encoding errors, using Python or the command-line tool recode:

Mastering Characters Sets in Linux (Weird Characters, part 2).

Tags: writing, linux, unicode, i18n, charsets, ascii, programming, python
[ 14:06 Nov 25, 2009    More writing | permalink to this entry ]

Building a Py-Webkit-GTK presentation tool

I almost always write my presentation slides using HTML. Usually I use Firefox to present them; it's the browser I normally run, so I know it's installd and the slides all work there. But there are several disadvantages to using Firefox:

• In fullscreen mode, it has a small "minimized urlbar" at the top of the screen that I've never figured out to banish -- not only is it visible to users, but it also messes up the geometry of the slides (they have to be 762 pixels high rather than 768);
• It's very heavyweight, bad when using a mini laptop or netbook;
• Any personal browsing preferences, like no-animation, flashblock or noscript, apply to slides too unless explicitly disabled, which I've forgotten to do more than once before a talk.

Last year, when I was researching lightweight browsers, one of the ones that impressed me most was something I didn't expect: the demo app that comes with pywebkitgtk (package python-webkit on Ubuntu). In just a few lines of Python, you can create your own browser with any UI you like, with a fully functional content area. Their current demo even has tabs.

So why not use pywebkitgtk to create a simple fullscreen webkit-based presentation tool?

It was even simpler than I expected. Here's the code:

#!/usr/bin/env python
# python-gtk-webkit presentation program.
# Copyright (C) 2009 by Akkana Peck.
# Share and enjoy under the GPL v2 or later.

import sys
import gobject
import gtk
import webkit

class WebBrowser(gtk.Window):
    def __init__(self, url):
        gtk.Window.__init__(self)
        self.fullscreen()

        self._browser= webkit.WebView()
        self.add(self._browser)
        self.connect('destroy', gtk.main_quit)

        self._browser.open(url)
        self.show_all()

if __name__ == "__main__":
    if len(sys.argv) <= 1 :
        print "Usage:", sys.argv[0], "url"
        sys.exit(0)

    gobject.threads_init()
    webbrowser = WebBrowser(sys.argv[1])
    gtk.main()

That's all! No navigation needed, since the slides include javascript navigation to skip to the next slide, previous, beginning and end. It does need some way to quit (for now I kill it with ctrl-C) but that should be easy to add.

Webkit and image buffering

It works great. The only problem is that webkit's image loading turns out to be fairly poor compared to Firefox's. In a presentation where most slides are full-page images, webkit clears the browser screen to white, then loads the image, creating a noticable flash each time. Having the images in cache, by stepping through the slide show then starting from the beginning again, doesn't help much (these are local images on disk anyway, not loaded from the net). Firefox loads the same images with no flash and no perceptible delay.

I'm not sure if there's a solution. I asked some webkit developers and the only suggestion I got was to rewrite the javascript in the slides to do image preloading. I'd rather not do that -- it would complicate the slide code quite a bit solely for a problem that exists only in one library.

There might be some clever way to hack double-buffering in the app code. Perhaps something like catching the 'load-started' signal, switching to another gtk widget that's a static copy of the current page (if there's a way to do that), then switching back on 'load-finished'.

But that will be a separate article if I figure it out. Ideas welcome!

Tags: programming, hack, python, web, speaking
[ 16:12 Nov 11, 2009    More programming | permalink to this entry ]

Friendlier error messages for shell newbies

Helping people get started with Linux shells, I've noticed they tend to make two common mistakes vastly more than any others:

1. Typing a file path without a slash, like etc/fstab
2. typing just a filename, without a command in front of it

The first boils down to a misunderstanding of how the Linux file system hierarchy works. (For a refresher, you might want to check out my Linux Planet article Navigating the Linux Filesystem.)

The second problem is due to forgetting the rules of shell grammar. Every shell sentence needs a verb, just like every sentence in English. In the shell, the command is the verb: what do you want to do? The arguments, if any, are the verb's direct object: What do you want to do it to?

(For grammar geeks, there's no noun phrase for a subject because shell commands are imperative. And yes, I ended a sentence with a preposition, so go ahead and feel superior if you believe that's incorrect.)

The thing is, both mistakes are easy to make, especially when you're new to the shell, perhaps coming from a "double-click on the file and let the computer decide what you should do with it" model. The shell model is a lot more flexible and (in my opinion) better -- you, not the computer, gets to decide what you should do with each file -- but it does take some getting used to.

But as a newbie, all you know is that you type a command and get some message like "Permission denied." Why was permission denied? How are you to figure out what the real problem was? And why can't the shell help you with that?

And a few days ago I realized ... it can! Bash, zsh and similar shells have a fairly flexible error handling mechanism. Ubuntu users have seen one part of this, where if you type a command you don't have installed, Ubuntu gives you a fancy error message suggesting what you might have meant and/or what package you might be missing:

$ catt /etc/fstab
No command 'catt' found, did you mean:
 Command 'cat' from package 'coreutils' (main)
 Command 'cant' from package 'swap-cwm' (universe)
catt: command not found

What if I tapped into that same mechanism and wrote a more general handler that could offer helpful suggestions when it looked like the user forgot the command or the leading slash?

It turns out that Ubuntu's error handler uses a ridiculously specific function called command_not_found_handle that can't be used for other errors. Some helpful folks I chatted with on #bash felt, as I did, that such a specific mechanism was silly. But they pointed me to a more general error trapping mechanism that turned out to work fine for my purposes.

It took some fussing and fighting with bash syntax, but I have a basic proof-of-concept. Of course it could be expanded to cover a lot more types of error cases -- and more types of files the user might want to open.

Here are some sample errors it catches:

$ schedule.html
bash: ./schedule.html: Permission denied

schedule.html is an HTML file. Did you want to run: firefox schedule.html

$ screenshot.jpg
bash: ./screenshot.jpg: Permission denied

screenshot.jpg is an image file. Did you want to run:
    pho screenshot.jpg
    gimp screenshot.jpg

$ .bashrc
bash: ./.bashrc: Permission denied

.bashrc is a text file. Did you want to run:
    less .bashrc
    vim .bashrc

$ ls etc/fstab
/bin/ls: cannot access etc/fstab: No such file or directory

Did you forget the leading slash?
etc/fstab doesn't exist, but /etc/fstab does.

You can find the code here: Friendly shell errors and of course I'm happy to take suggestions or contributions for how to make it friendlier to new shell users.

Tags: linux, shell, help, education, programming
[ 14:07 Nov 08, 2009    More linux | permalink to this entry ]

Gathering RSS files for a Palm PDA: FeedMe

For years I've been reading daily news feeds on a series of PalmOS PDAs, using a program called Sitescooper that finds new pages on my list of sites, downloads them, then runs Plucker to translate them into Plucker's open Palm-compatible ebook format.

Sitescooper has an elaborate series of rules for trying to get around the complicated formatting in modern HTML web pages. It has an elaborate cache system to figure out what it's seen before. When sites change their design (which most news sites seem to do roughly monthly), it means going in and figuring out the new format and writing a new Sitescooper site file. And it doesn't understand RSS, so you can't use the simplified RSS that most sites offer. Finally, it's no longer maintained; in fact, I was the last maintainer, after the original author lost interest.

Several weeks ago, bma tweeted about a Python RSS reader he'd hacked up using the feedparser package. His reader targeted email, not Palm, but finding out about feedparser was enough to get me started. So I wrote FeedMe (Carla Schroder came up with the all-important name).

I've been using it for a couple of weeks now and I'm very happy with the results. It's still quite rough, of course, but it's already producing better files than Sitescooper did, and it seems more maintainable. Time will tell.

Of course it needs to be made more flexible, adjusted so that it can produce formats besides Plucker, and so on. I'll get to it.

And the only site I miss now, because it doesn't offer an RSS feed, is Linux Planet. Maybe I'll find a solution for that eventually.

Tags: RSS, python, programming, palm, plucker
[ 20:08 Oct 20, 2009    More programming | permalink to this entry ]

GTK dialogs in GIMP (and updated wallpaper script)

I've been getting tired of my various desktop backgrounds, and realized that I had a lot of trip photos, from fabulous places like Grosvenor Arch (at right), that I'd never added to my background collection.

There's nothing like lots of repetitions of the same task to bring out the shortcomings of a script, and the wallpaper script I threw together earlier this year was no exception. I found myself frequently irritated by not having enough information about what the script was doing or being able to change the filename. Then I could have backgrounds named grosvenor.jpg rather than img2691.jpg.

Alas, I can't use the normal GIMP Save-as dialog, since GIMP doesn't make that dialog available to plug-ins. (That's a deliberate choice, though I've never been clear on the reason behind it.) If I wanted to give that control to the user, I'd have to make my own dialogs.

It's no problem to make a GTK dialog from Python. Just create a gtk.Dialog, add a gtk.Entry to it, call dialog.run(), then check the return value and get the entry's text to see if it changed. No problem, right?

Ha! If you think that, you don't work with computers. The dialog popped up fine, it read the text entry fine ... but it wouldn't go away afterward. So after the user clicked OK, the plug-in tried to save and GIMP popped up the JPEG save dialog (the one that has a quality slider and other controls, but no indication of filename) under my text entry dialog, which remained there.

All attempts at calling dialog.hide() and dialog.destroy() and similar mathods were of no avail. A helpful person on #pygtk worked with me but ended up as baffled as I was. What was up?

The code seemed so simple -- something like this:

    response = dialog.run()
    if response == gtk.RESPONSE_OK :
        pathname = pathentry.get_text()
        dialog.hide()
        dialog.destroy()
        pdb.gimp_file_save(newimg, newimg.active_layer, pathname, pathname,
                           run_mode=0)

In the end, GIMP guru Sven pointed me to the answer. The problem was that my dialog wasn't part of the GTK main loop. In retrospect, this makes sense: the plug-in is an entirely different process, so I shouldn't be surprised that it would have its own main loop. So when I hide() and destroy(), those events don't happen right away because there's no loop in the plug-in process that would see them.

The plug-in passes control back to GIMP to do the gimp_file_save(). GIMP's main loop doesn't have access to the hide and destroy signals I just sent. So the gimp_file_save runs, popping up its own dialog (under mine, because the JPEG save dialog is transient to the original image window while my python dialog isn't). That finishes, returns control to the plug-in, the plug-in exits and at that point GTK cleans up and finally destroys the dialog.

The solution is to loop over GTK events in the plug-in before calling gimp_file_save, like this:

    response = dialog.run()
    if response == gtk.RESPONSE_OK :
        pathname = pathentry.get_text()
        dialog.hide()
        dialog.destroy()
        while gtk.events_pending() :
            gtk.main_iteration()
        pdb.gimp_file_save(newimg, newimg.active_layer, pathname, pathname,
                           run_mode=0)

That loop gives the Python process a chance to clean up the dialog before passing control to GIMP and its main loop. GTK in the subprocess is happy, the user is happy, and I'm happy because now I have a much more efficient way of making lots of desktop backgrounds for lots of different machines.

The updated script, along with a lot more information on how to use it and how to set up tool presets for it.

Tags: programming, gimp
[ 22:21 Sep 15, 2009    More gimp | permalink to this entry ]

Using apt-file to track down build errors

Someone was asking for help building XEphem on the XEphem mailing list. It was a simple case of a missing include file, where the only trick is to find out what package you need to install to get that file. (This is complicated on Ubuntu, which the poster was using, by the way they fragment the X developement headers into a maze of a xillion tiny packages.)

The solution -- apt-file -- is so simple and easy to use, and yet a lot of people don't know about it. So here's how it works.

The poster reported getting these compiler errors:

ar rc libz.a adler32.o compress.o crc32.o uncompr.o deflate.o trees.o zutil.o inflate.o inftrees.o inffast.o
ranlib libz.a
make[1]: Leaving directory `/home/gregs/xephem-3.7.4/libz'
gcc -I../../libastro -I../../libip -I../../liblilxml -I../../libjpegd -I../../libpng -I../../libz -g -O2 -Wall -I../../libXm/linux86 -I/usr/X11R6/include   -c -o aavso.o aavso.c
In file included from aavso.c:12:
../../libXm/linux86/Xm/Xm.h:56:27: error: X11/Intrinsic.h: No such file or directory
../../libXm/linux86/Xm/Xm.h:57:23: error: X11/Shell.h: No such file or directory
../../libXm/linux86/Xm/Xm.h:58:23: error: X11/Xatom.h: No such file or directory
../../libXm/linux86/Xm/Xm.h:59:34: error: X11/extensions/Print.h: No such file or directory
In file included from ../../libXm/linux86/Xm/Xm.h:60,
                 from aavso.c:12:
../../libXm/linux86/Xm/XmStrDefs.h:1373: error: expected `=', `,', `;', `asm' or `__attribute__' before `char'
In file included from ../../libXm/linux86/Xm/Xm.h:60,
                 from aavso.c:12:
../../libXm/linux86/Xm/XmStrDefs.h:5439:28: error: X11/StringDefs.h: No such file or directory
In file included from ../../libXm/linux86/Xm/Xm.h:61,
                 from aavso.c:12:
../../libXm/linux86/Xm/VirtKeys.h:108: error: expected `)' before `*' token
In file included from ../../libXm/linux86/Xm/Display.h:49,
                 from ../../libXm/linux86/Xm/DragC.h:48,
                 from ../../libXm/linux86/Xm/Transfer.h:44,
                 from ../../libXm/linux86/Xm/Xm.h:62,
                 from aavso.c:12:
../../libXm/linux86/Xm/DropSMgr.h:88: error: expected specifier-qualifier-list before `XEvent'
../../libXm/linux86/Xm/DropSMgr.h:100: error: expected specifier-qualifier-list before `XEvent'

How do you go about figuring this out?

When interpreting compiler errors, usually what matters is the *first* error. So try to find that. In the transcript above, the first line saying "error:" is this one:

../../libXm/linux86/Xm/Xm.h:56:27: error: X11/Intrinsic.h: No such file or directory

So the first problem is that the compiler is trying to find a file called Intrinsic.h that isn't installed.

On Debian-based systems, there's a great program you can use to find files available for install: apt-file. It's not installed by default, so install it, then update it, like this (the update will take a long time):

$ sudo apt-get install apt-file
$ sudo apt-file update

Once it's updated, you can now find out what package would install a file like this:

$  apt-file search Intrinsic.h
libxt-dev: /usr/include/X11/Intrinsic.h
tendra: /usr/lib/TenDRA/lib/include/x5/t.api/X11/Intrinsic.h

In this case two two packages could install a file by that name. You can usually figure out from looking which one is the "real" one (usually the one with the shorter name, or the one where the package name sounds related to what you're trying to do). If you're stil not sure, try something like apt-cache show libxt-dev tendra to find out more about the packages involved.

In this case, it's pretty clear that tendra is a red herring, and the problem is likely that the libxt-dev package is missing. So apt-get install libxt-dev and try the build again.

Repeat the process until you have everything you need for the build.

Remember apt-file if you're not already using it. It's tremendously useful in tracking down build dependencies.

Tags: open source, linux, programming, debian, ubuntu
[ 10:25 Sep 06, 2009    More linux | permalink to this entry ]

Reading Palm Datebook files on Linux

Over the years, I've kept a few sets of records in the Datebook app on my PalmOS PDA -- health records and such. I've been experimenting with a few python plotting packages (pycha, CairoPlot and a few others) and I wanted to try plotting one of my Datebook databases.

Not so fast. It seems that it's been a year or more since I last crunched any of this data -- and in the time since then, pilot-link has bumped its version numbers and is now shipping libpisock.so.9 instead of .8.

So what? Well, the problem is that Linux hasn't offered any way to read Palm Datebook files for years. The pilot-link package offered on most distros used to include a program called pilot-datebook, but it was deleted from the source several years ago. Apparently it was hard to maintain.

Back when it first disappeared, I built the previous version of the source, stuck the pilot-datebook binary in ~/bin/linux and have been using it ever since. Which worked fine -- until libpisock.so.8 was no longer there. (Linking .9 to .8 didn't work either.) This is all the more ironic because I don't need pilot-datebook to talk to the PDA with libpisock -- all I want to do is parse the format of a file I've already uploaded.

Off to hunt for an old version of the source. I started at pilot-link.org, but gave up after a while -- they don't seem to have source there except for the latest couple of versions, nor do they have any documentation. Ironically, in their FAQ the very first question is "How can I read the databook entries from a Palm backup?" but the FAQ page is broken and the "answer" is actually another unrelated FAQ question.

Anyway, no help there. I tried googling for old tarballs but there doesn't seem to be anything like archive.org for source code. All I found was the original pilot-datebook page, with a tarball that you insert into a copy of pilot-link 0.9.5 then modify the Makefile. Might work but that's really old.

So I fell back on old distributions. I guessed that Ubuntu Dapper was old enough that it might still have pilot-datebook. So I went to the Dapper pilot-link source and downloaded the source tarball (curiously, they don't offer src debs -- you have to download the tarball and patches separately).

Of course, it doesn't build on Ubuntu Jaunty. It had various entertaining errors ranging from wanting a mysterious tcl.m4 file not present in the code ... to not being able to find <iostream.h< because all the C++ stdlib files have recently been renamed to remove the .h ... to a change in the open() system call where I needed to add permissions argument for O_CREAT.

But I did get it working! So now I have a pilot-datebook program that builds and runs on Ubuntu Jaunty, and parses my DatebookDB.pdb file.

Since I bet I'm not the only one in the world who occasionally wants to read a Palm Datebook file, I've put my working version of the source here: pilot-link_0.11.8.jaunty.tar.gz.

After the usual configure and make, if all you want is pilot-datebook, cd src/pilot-datebook then copy both pilot-datebook and the directory .libs to wherever you want to install them.

And yeah, it would be better to write a standalone program that just parsed the format. But it's hard to justify that for what's essentially a dead platform. The real solution is to quit using a Palm for this, import the data into some common format and keep it on my Linux workstation from now on.

Tags: palm, linux, programming, patch
[ 11:39 Aug 31, 2009    More linux | permalink to this entry ]

Linux Bloat 102

Part 2 of my Linux bloat article looks at information you can get from the kernel via some useful files in /proc, at three scripts that display that info, and also at how to use exmap, an app and kernel module that shows you a lot more about what resources your apps are using. How Do You Really Measure Linux Bloat?

Tags: writing, programming, linux, performance, bloat
[ 19:52 Aug 27, 2009    More writing | permalink to this entry ]

Crikey 0.8.3: bug fixes and several new syntaxes

Sometimes I love open source. A user contacted me about my program Crikey!, which lets you generate key events to do things like assign a key that will type in a string you don't want to type in by hand. He had some thorny problems where crikey was failing, and a few requests, like sending Alt and other modifier keys.

We corresponded a bit, figured out exactly how things should work and some test cases, went through a couple iterations of changes where I got lots of detailed and thoughtful feedback and more test cases, and now Crikey can do an assortment of new useful stuff.

New features: crikey now handles number codes like \27, modifier keys like \A for alt, does a better job with symbols like \(Return\), and handles a couple of new special characters like \e for escape. It also works better at sending window manager commands, like "\A\t" to change the active window.

I've added some better documentation on all the syntaxes it understands, both on the web page and in the -h and -l (longhelp) command-line arguments, and made a release: crikey 0.8.3.

Plus: a list of great regression tests that I can use when testing future updates (in the file TESTING in the tarball).

Tags: programming, crikey, X11, linux
[ 16:38 Aug 19, 2009    More programming | permalink to this entry ]

Linux Bloat 101

Continuing my Linux Planet series on Linux performance monitoring, the latest article looks at bloat and how you can measure it: Finding and Trimming Linux Bloat.

This one just covers the basics. The followup article, in two weeks, will dive into more detail on how to analyze what resources programs are really using.

Tags: writing, programming, linux, performance, bloat
[ 10:27 Aug 13, 2009    More writing | permalink to this entry ]

On Teaching Programming (GetSET 2009)

I survived another GetSET Javascript-in-a-day workshop for GetSET 2009.

It went okay, but not as well as I'd hoped. This year's class was more distractable than classes of past years -- and, judging by their career goals, less interested in computers or engineering, unfortunate in a program run by the Society of Women Engineers.

In the morning, we had a hard time getting them to focus long enough to learn basics like what a variable was. After a caucus at lunchtime, we decided to skip the next exercise (looping over an array of colors) and spend some time drilling on the basics, and keep at it 'til they got it. It took a while but we eventually got through. We needed more examples in the morning, more interaction, some visceral way of explaining programming basics so they really get it.

They do better working as a group on a concrete problem, like the final whiteboard exercise, "How do we figure out whether the click was on the flower?". That always ends up being a highlight of the class, even though it involves (gasp) doing math. This year was no exception, but it did take a while to get through. Using variables lost them completely ("is the mouse's X coordinate bigger than or less than the flower's X?") but when we used actual numnbers and ran through several examples, things eventually clicked. "The flower starts at (2, 5) and is 200 pixels wide. If the mouse click is at (34, 45), who thinks it's inside the flower? Raise your hands. Who thinks it's not? Now what if I click at (300, 24)?" A couple of them got it right away, but it took a long time to bring the whole class along.

I'm not still sure how to use that method for more basic concepts like "what is a variable?". Perhaps some sort of role-playing? Watching William Phelps guide the girls through planet motions in our astronomy workshop Wednesday, each girl playing the role of a solar system object, inspired me. I'd used role-playing like that with little kids, but William says it works even with adults to get concepts across, and after seeing him with the high schoolers I believe it. But how to adapt that to programming concepts? A recent Slate article on teaching programming had some interesting ideas I want to try.

Printed handouts for GetSET may be a waste of time. Nobody was even bothering to look at them, despite the fact that they had complete instructions for everything we were doing. Do schools not give students printed assignments or homework any more? Last year, they used the printed exercises but not the quick reference guides; this year they wouldn't even read the exercises. On the other hand, it might be worth it for the handful in each class who really love programming. I always hope some of them take the handouts home and try some of the extras on their own.

Finally, the class would be so much easier if we could teach it on a less pointy-clicky OS! Or at least on machines where IE isn't the default browser. The first 3-4 exercises go painfully slowly, guiding a roomful of girls through many GUI navigation steps:

• Open the GetSET folder on your desktop
• Find the file named whatever.html
• Right-click on it, find Open With, and choose Wordpad
• Now find the window where you were just looking at the GetSET files (because everything on Windows tends to open with huge windows, it's now covered by Wordpad)
• Drag whatever.html into Firefox

Then the helpers have to go around the room ensuring that the girls have the correct file loaded in both Wordpad and Firefox. This took way too long with only four people to check the whole class, especially since we had to do it for every exercise. Invariably some girls will doubleclick instead of right-clicking or dragging, and will end up in whatever HTML editor Microsoft is pushing this year, or with an IE window instead of Firefox (and then the Error Console won't be there when she looks for it later).

Suggestions like "Keep that window open, you'll need it throughout the class" or "Try making that window smaller, so you can see both windows at once" don't help. The girls are too used to the standard Windows model of one screen-filling window at a time. Keeping two apps visible at once is too foreign. A few them are good at using the taskbar to switch among apps, but for the rest, loading new files is awkward and error prone.

In postmortems two years ago we talked about having them work on one file throughout the whole workshop. That would solve the problem, but I'm still working on how to do it without a lot of "Now comment out the code you just wrote, so you won't get the prompt every time, then scroll down to the next block of code and uncomment it."

I couldn't help thinking how on Linux, we could just tell them to type leafpad whatever.html; firefox whatever.html and be done. Or even give them an alias that would do it. Hmm ... I wonder if I could make a Windows .bat file that would open the same file in Wordpad and Firefox both? Must try that.

Tags: education, programming
[ 19:12 Aug 07, 2009    More education | permalink to this entry ]

Twit: Now with pattern searches

During OSCON a couple of weeks ago, I kept wishing I could do Twitter searches for a pattern like #oscon in a cleaner way than keeping a tab open in Firefox where I periodically hit Refresh.

Python-twitter doesn't support searches, alas, though it is part of the Twitter API. There's an experimental branch of python-twitter with searching, but I couldn't get it to work. But it turns out Gwibber is also written in Python, and I was able to lift some JSON code from Gwibber to implement a search. (Gwibber itself, alas, doesn't work for me: it bombs out looking for the Gnome keyring. Too bad, looks like it might be a decent client.)

I hacked up a "search for OSCON" program and used it a little during the week of the conference, then got home and absorbed in catching up and preparing for next week's GetSET summer camp, where I'm running an astronomy workshop and a Javascript workshop for high school girls. That's been keeping me frazzled, but I found a little time last night to clean up the search code and release Twit 0.3 with search and a few other new command-line arguments.

No big deal, but it was nice to take a hacking break from all this workshop coordinating. I'm definitely happier program than I am organizing events, that's for sure.

Tags: twitter, python, programming
[ 17:23 Aug 03, 2009    More programming | permalink to this entry ]

Newbie Greasemonkey script writing

I was reading a terrific article on the New York Times about Watching Whales Watching Us. At least, I was trying to read it -- but the NYT website forces font faces and sizes that, on my system, end up giving me a tiny font that's too small to read. Of course I can increase font size with Ctrl-+ -- but it gets old having to do that every time I load a NYT page.

The first step was to get Greasemonkey working on Firefox 3.5. "Update scripts" doesn't find a new script, and if you go to Greasemonkey's home page, the last entry is from many months ago and announces Firefox 3.1 support. But curiously, if you go to the Greasemonkey page on the regular Mozilla add-ons site, it does support 3.5.

I've had Greasemonkey for quite some time, but every time I try to get started writing a script I have trouble getting started. There are dozens of Greasemonkey tutorials on the web, but most of them are oriented toward installing scripts and don't address "What do you type into the fields of the Greasemonkey New User Script dialog?"

Fortunately, I did find one that explained it: The beginner's guide to Greasemonkey scripting. I gave my script a name (NYT font) and a namespace (my own domain), added http://*nytimes.com/* for Includes, and nothing for Excludes.

Click OK, and Greasemonkey offers a "choose editor" dialog. I chose emacs, which mostly worked though the emacs window unaccountably came up with a split window that I had to dismiss with C-x 1.

Now what to type in the editor? Firebug came to the rescue here.

I went back to the NYT page with the too-small fonts and clicked on Firebug. The body style showed that they're setting

font-family: Georgia, serif
font-size: 84.5%

84.5%? Where does that come from? What happens if I change that to 100%? Fortunately, I can test that right there in the Firebug window. 100% made the fonts fairly huge, but 90% was about right.

I went back to greasemonkey's editor window and added:

document.body.style.fontSize = "90%";

Saved the file, and that was all I needed! Once I hit Reload on the NYT page I got a much more readable font size.

Tags: web, browsers, firefox, greasemonkey, programming
[ 11:30 Jul 12, 2009    More tech/web | permalink to this entry ]

Twittering -- and writing Twitter clients

I finally dragged myself into 2009 and tried Twitter.

I'd been skeptical, but it's actually fairly interesting and not that much of a time sink. While it's true that some people tweet about every detail of their lives -- "I'm waiting for a bus" / "Oh, hooray, the bus is finally here" / "I got a good seat in the second row of the bus" / "The bus just passed Second St. and two kids got on" / "Here's a blurry photo from my phone of the Broadway Av. sign as we pass it" -- it's easy enough to identify those people and un-follow them.

And there are tons of people tweeting about interesting stuff. It's like a news ticker, but customizable -- news on the latest protests in Iran, the latest progress on freeing the Mars Spirit Rover, the latest interesting publication on dinosaur fossils, and what's going on at that interesting conference halfway around the world.

The trick is to figure out how you want the information delivered. I didn't want to have to leave a tab open in Firefox all the time. There was an xchat plug-in that sounded perfect -- I have an xchat window up most of the time I'm online -- but it turned out it works by picking one of the servers you're connected to, making a private channel and posting things there. That seemed abusive to the server -- what if everyone on Freenode did that?

So I wanted a separate client. Something lightweight and simple. Unfortunately, all the Twitter clients available for Linux either require that I install a lot of infrastructure first (either Adobe Air or Mono), or they just plain didn't work (a Twitter client where you can't click on links? Come on!)

But then I tried out the Python-Twitter bindings, and they were so easy to use I decided to write them up for my next Linux Planet article, which came out today: Write Your Own Linux Twitter Client In Less Time Than It Takes To Find One!.

The article shows how to use the bindings to write a bare-bones client. But of course, I've been hacking on the client all along, so the one I'm actually using has a lot more features like *ahem* letting you click on links. And letting you block threads, though I haven't actually tested that since I haven't seen any threads I wanted to block since my first day.

You can download the current version of Twit, and anyone who's interested can follow me on Twitter. I don't promise to be interesting -- that's up to you to decide -- but I do promise not to tweet about every block of my bus ride.

Tags: writing, programming, python, twitter
[ 15:09 Jul 09, 2009    More writing | permalink to this entry ]

A Beginner's Guide to Free Software Programming Languages

Linux Planet asked me for an intro article for prospective programmers, explaining the pros and cons of various programming languages. Here it is: A Beginner's Guide to Free Software Programming Languages

Tags: writing, programming
[ 12:00 Jun 28, 2009    More writing | permalink to this entry ]

Pytopo 0.8 released

On my last Mojave trip, I spent a lot of the evenings hacking on PyTopo.

I was going to try to stick to OpenStreetMap and other existing mapping applications like TangoGPS, a neat little smartphone app for downloading OpenStreetMap tiles that also runs on the desktop -- but really, there still isn't any mapping app that works well enough for exploring maps when you have no net connection.

In particular, uploading my GPS track logs after a day of mapping, I discovered that Tango really wasn't a good way of exploring them, and I already know Merkaartor, nice as it is for entering new OSM data, isn't very good at working offline. There I was, with PyTopo and a boring hotel room; I couldn't stop myself from tweaking a bit.

Adding tracklogs was gratifyingly easy. But other aspects of the code bother me, and when I started looking at what I might need to do to display those Tango/OSM tiles ... well, I've known for a while that some day I'd need to refactor PyTopo's code, and now was the time.

Surprisingly, I completed most of the refactoring on the trip. But even after the refactoring, displaying those OSM tiles turned out to be a lot harder than I'd hoped, because I couldn't find any reliable way of mapping a tile name to the coordinates of that tile. I haven't found any documentation on that anywhere, and Tango and several other programs all do it differently and get slightly different coordinates. That one problem was to occupy my spare time for weeks after I got home, and I still don't have it solved.

But meanwhile, the rest of the refactoring was done, nice features like track logs were working, and I've had to move on to other projects. I am going to finish the OSM tile MapCollection class, but why hold up a release with a lot of useful changes just for that?

So here's PyTopo 0.8, and the couple of known problems with the new features will have to wait for 0.9.

Tags: programming, python, gtk, pygtk, mapping
[ 19:49 Jun 20, 2009    More programming | permalink to this entry ]

Python: show all methods in a given object or module

A silly little thing, but something that Python books mostly don't mention and I can never find via Google:

How do you find all the methods in a given class, object or module?

Ideally the documentation would tell you. Wouldn't that be nice? But in the real world, you can't count on that, and examining all of an object's available methods can often give you a good guess at how to do whatever you're trying to do.

Python objects keep their symbol table in a dictionary called __dict__ (that's two underscores on either end of the word). So just look at object.__dict__. If you just want the names of the functions, use object.__dict__.keys().

Thanks to JanC for suggesting dir(object) and help(object), which can be more helpful -- not all objects have a __dict__.

Tags: programming, python, tips
[ 11:44 Jun 19, 2009    More programming | permalink to this entry ]

Programming With PyGTK, part 3: Key events and object oriented Python

Part 3 of "Graphical Python Programming With PyGTK" uses object-oriented Python to clean up the code from Part 2, and also adds handling of key events to get rid of that silly Quit button. PythonGTK Programming part 3: Screensaver, Objects, and User Input

Tags: writing, python, programming
[ 11:18 Jun 14, 2009    More writing | permalink to this entry ]

A GPX file manager

Someone on the OSM newbies list asked how he could strip waypoints out of a GPX track file. Seems he has track logs of an interesting and mostly-unmapped place that he wants to add to openstreetmap, but there are some waypoints that shouldn't be included, and he wanted a good way of separating them out before uploading.

Most of the replies involved "just edit the XML." Sure, GPX files are pretty simple and readable XML -- but a user shouldn't ever have to do that! Gpsman and gpsbabel were also mentioned, but they're not terribly easy to use either.

That reminded me that I had another XML-parsing task I'd been wanting to write in Python: a way to split track files from my Garmin GPS.

Sometimes, after a day of mapping, I end up with several track segments in the same track log file. Maybe I mapped several different trails; maybe I didn't get a chance to upload one day's mapping before going out the next day. Invariably some of the segments are of zero length (I don't know why the Garmin does that, but it always does). Applications like merkaartor don't like this one bit, so I usually end up editing the XML file and splitting it into segments by hand. I'm comfortable with XML -- but it's still silly.

I already have some basic XML parsing as part of PyTopo and Ellie, so I know the parsing very easy to do. So, spurred on by the posting on OSM-newbies, I wrote a little GPX parser/splitter called gpxmgr. gpxmgr -l file.gpx can show you how many track logs are in the file; gpxmgr -w file.gpx can write new files for each non-zero track log. Add -p if you want to be prompted for each filename (otherwise it'll use the name of the track log, which might be something like "ACTIVE\ LOG\ #2").

How, you may wonder, does that help the original poster's need to separate out waypoints from track files? It doesn't. See, my GPS won't save tracklogs and waypoints in the same file, even if you want them that way; you have to use two separate gpsbabel commands to upload a track file and a waypoint file. So I don't actually know what a tracklog-plus-waypoint file looks like. If anyone wants to use gpxmgr to manage waypoints as well as tracks, send me a sample GPX file that combines them both.

Tags: mapping, programming, python
[ 19:43 Jun 01, 2009    More mapping | permalink to this entry ]

JS Jup: now, with variable animation speed

I wrote last week about the sorts of programmer compulsions that lead to silly apps like my animated Javascript Jupiter. I got it working well enough and stopped, knowing there were more features that would be easy to add but trying to ignore them.

My mom, immediately upon seeing it, unerringly zeroed in on the biggest missing feature I'd been trying to ignore. "Can you make it go faster or slower?"

I put it off for a while, but of course I had to do it -- so now there are Faster and Slower buttons. It still goes by hour jumps, so the fastest you can go is an hour per millisecond. Fun to watch. Or you can slow it down to 1 hour per 3600000 milliseconds if you want to see it animate in real time. :-)

Tags: programming, astronomy, javascript
[ 10:42 May 31, 2009    More programming | permalink to this entry ]

Programming With PyGTK, part 2: pretty screensaver-type graphics

Part 2 of Graphical Python Programming With PyGTK gets into how to do some cool Qix screensaver-style graphics, in: Graphical Python Programming With PyGTK, part 2: Write Your Own Screensaver.

There's also a digg link.

Tags: writing, python, programming
[ 17:09 May 28, 2009    More writing | permalink to this entry ]

Javascript Jupiter

It's a sickness, I tell you.

It's not like I needed another Jupiter's moons application. I've already written more or less the same app for four platforms.

I don't use the Java web version, Juplet, very much any more, because I often have Java disabled or missing. And I don't use my Zaurus any more so Juplet for Zaurus isn't very relevant. But I can always call up my Xlib or PalmOS Jupiter's moons app if I need to check on those Galilean moons. They work fine. Another version would be really pointless. A waste of time.

So it should have been no big deal when, during the course of explaining to someone the difference between Java and Javascript, it suddenly occurred to me that it would be awfully easy to re-implement that Java Juplet web page using Javascript, HTML and CSS. I mean, a rational person would just say "oh, yeah, I suppose that's true" and go on with life.

But what I'm trying to say is that programming isn't a career path, or a hobby, or a field of academic study. It's a disease. It's a compulsion, where, sometimes, just realizing that something could be done renders you unable to think about anything else until you just ... try ... just a few minutes ... see how well it works ... oh, wow, that really looks a lot better than the Java version, wouldn't it look even nicer if you just added in this one other little tweak ... but wait, now it's so close to working, I bet it wouldn't be all that hard to take the Java class and turn it into ...

... and before you know it, it's tomorrow and you have something that's almost a working app, and it's just really a shame to get that far and not finish it at least to the point where you can share it.

But then, Javascript and web pages are so easy to work on that it really isn't that much extra work to add in some features that the old version didn't have, like an animate button ...

... and your Saturday morning is gone forever, and there's not much you can do about that, but at least you have a nice animated Jupiter's moons (and shadows) page when the sickness passes and you can finally think about other things.

Tags: programming, astronomy, javascript
[ 20:10 May 23, 2009    More programming | permalink to this entry ]

Graphical Python Programming With PyGTK

This week's Linux Planet article is another one on Python and graphical toolkits, but this time it's a little more advanced: Graphical Python Programming With PyGTK.

This one started out as a fun and whizzy screensaver sort of program that draws lots of pretty colors -- but I couldn't quite fit it all into one article, so that will have to wait for the sequel two weeks from now.

Tags: writing, python, programming
[ 18:53 May 14, 2009    More writing | permalink to this entry ]

GUI Programming in Python For Beginners

Latest on Linux Planet: another introductory programming article, this time on Python's tkinter library: GUI Programming in Python For Beginners. (As usual, there's a Digg link and also a Reddit one.)

Tags: writing, python, programming
[ 15:36 Mar 26, 2009    More writing | permalink to this entry ]

Ellie: Plot GPS elevation profiles

Ever since I got the GPS I've been wanting something that plots the elevation data it stores. There are lots of apps that will show me the track I followed in latitude and longitude, but I couldn't find anything that would plot elevations.

But GPX (the XML-based format commonly used to upload track logs) is very straightforward -- you can look at the file and read the elevations right out of it. I knew it wouldn't be hard to write a script to plot them in Python; it just needed a few quiet hours. Sounded like just the ticket for a rainy day stuck at home with a sore throat.

Sure enough, it was fairly easy. I used xml.dom.minidom to parse the file (I'd already had some experience with it in gimplabels for converting gLabels templates), and pylab from matplotlib for doing the plotting. Easy and nice looking.

I even threw in the nice "conditional main" code from Matt Harrison's SCALE7x Python talk, so it should be callable from other Python code.

Here's the page and a screenshot: Ellie: plot elevation from a GPS track.

Tags: programming, mapping, python
[ 16:57 Mar 03, 2009    More programming | permalink to this entry ]

langgrep: search only in scripts of a specified language

I was making a minor tweak to my garmin script that uses gpsbabel to read in tracklogs and waypoints from my GPS unit, and I needed to look up the syntax of how to do some little thing in sh script. (One of the hazards of switching languages a lot: you forget syntax details and have to look things up a lot, or at least I do.)

I have quite a collection of scripts in various languages in my ~/bin (plus, of course, all the scripts normally installed in /usr/bin on any Linux machine) so I knew I'd have lots of examples. But there are scripts of all languages sharing space in those directories; it's hard to find just sh examples. For about the two-hundredth time, I wished, "Wouldn't it be nice to have a command that can search for patterns only in files that are really sh scripts?"

And then, the inevitable followup ... "You know, that would be really easy to write."

So I did -- a little python hack called langgrep that takes a language, grep arguments and a file list, looks for a shebang line and only greps the files that have a shebang matching the specified language.

Of course, while writing langgrep I needed langgrep, to look up details of python syntax for things like string.find (I can never remember whether it's string.find(s, pat) or s.find(pat); the python libraries are usually nicely object-oriented but strings are an exception and it's the former, string.find). I experimented with various shell options -- this is Unix, so of course there are plenty of ways of doing this in the shell, without writing a script. For instance:

grep find `egrep -l '#\\!.*python' *`
grep find `file * | grep python | sed 's/:.*//'`
i in foo; file $i|grep python && grep find $i; done    # in sh/bash

These are all pretty straightforward, but when I try to make them into tcsh aliases things get a lot trickier. tcsh lets you make aliases that take arguments, so you can use !:1 to mean the first argument, !2-$ to mean all the arguments starting with the second one. That's all very well, but when you put them into a shell alias in a file like .cshrc that has to be parsed, characters like ! and $ can mean other things as well, so you have to escape them with \. So the second of those three lines above turns into something like

alias greplang "grep \!:2-$ `file * | grep \!:1 | sed 's/:.*//'`"

except that doesn't work either, so it probably needs more escaping somewhere. Anyway, I decided after a little alias hacking that figuring out the right collection of backslash escapes would probably take just as long as writing a python script to do the job, and writing the python script sounded more fun.

So here it is: my langgrep script. (Awful name, I know; better ideas welcome!) Use it like this (if python is the language you're looking for, find is the search pattern, and you want -w to find only "find" as a whole word):

langgrep python -w find ~/bin/*

Tags: programming, python, shell
[ 09:57 Feb 28, 2009    More programming | permalink to this entry ]

Use GIMP Tool Presets for setting multiple crop aspect ratios

Making desktop backgrounds in GIMP is a bit tedious if you have several machines with screens of different sizes. The workflow goes something like this:

First, choose Crop tool and turn on Fixed: Aspect Ratio. Then loop over images:

1. Load an image
2. Go to Tool Options
3. Type in the aspect ratio: 4:3, 8:5, 5:4, 1366:768 etc.
4. Go to the image and crop.
5. Image->Scale (I have this on Shift-S, can't remember whether that was a default binding or not).
6. Ctrl-K to delete the current width (Ctrl-U also works, but beeps; I'm not sure why)
7. Type in the desired width (1024 or 1680 or 1366 or whatever) (I always hit Tab here, though it's probably not necessary)
8. Click Scale or type Alt-S (unfortunately, Return doesn't work in this dialog).
9. Save As to the appropriate name and path for the current resolution
10. Undo (the Scale), Undo (the Crop)
11. Load a new image (continue loop)

But you can use Save Options (Tool Presets) to avoid step 3, typing in the aspect ratio. Here's how:

• Set up the aspect ratio you want in the Crop tool, with Fixed checked
• Click on Save Options (the floppy disk icon in the lower left of Tool Options)
• Choose a name (choose New Entry first if you've already saved options).

Repeat, for each aspect ratio you might want to use.

Now clicking on Restore Options gives you a menu of all your commonly used aspect ratios -- much faster than typing them in every time. Too bad there's no way to use this shortcut for the Scale step, or to do Crop and Scale in one operation.

Nice shortcut! But having done that, I realized I could shorten it even more: I could make a selection (not a crop) with one of my preset aspect ratios, then run a script that would figure out from the aspect ratio which absolute size I wanted, crop and scale, and either save it to the right place, or make a new image so I could save without needing to Redo or Save a Copy. That was an easy script to write, so here it is: wallpaper.py.

Tags: gimp, tips, programming, hack
[ 22:45 Feb 05, 2009    More gimp | permalink to this entry ]

Crikey 0.8

Mostly this week has been consumed with preparations for LCA ... but programming is a sickness. When you get email from someone suggesting something relatively simple and obviously useful, well ... it's simply impossible not to pull out that emacs window and start typing.

And so it was when I got a request for a backspace character in crikey. Of course backspace and delete seem like perfectly reasonable and useful characters to want; don't know why I didn't think of putting them in before. So I did.

But while I was in there, suddenly it occurred to me that it really wouldn't be much harder to let users specify any key by symbol. (Did I mention being a programmer is a sickness?) And then I realized that specifying control characters with a caret, like ^H, would also be quite useful. (Did I mention that ...)

So anyway, now there's a Crikey 0.8 and it's time to get back to packing and endless fiddling with my talk slides. Except, wait, I need to update my netscheme script to work right with the new laptop, and ...

Did I mention that programming is a sickness?

Tags: programming, crikey
[ 20:56 Jan 14, 2009    More programming | permalink to this entry ]

Debian/Ubuntu repositories for Pho

I've been wanting for a long time to make Debian and Ubuntu repositories so people can install pho with apt-get, but every time I try to look it up I get bogged down.

But I got mail from a pho user who really wanted that, and even suggested a howto. That howto didn't quite do it, but it got me moving to look for a better one, which I eventually found in the Debian Repository Howto.

It wasn't complete either, alas, so it took some trial-and-error before it actually worked. Here's what finally worked:

I created two web-accessible directories, called hardy and etch. I copied all the files created by dpgk-buildpkg on each distro -- .deb, .dsc, .tar.gz, and .changes (I don't think this last file is used by anything) -- into each directory (renaming them to add -etch and -hardy as appropriate). Then:

% cd hardy/
% dpkg-scanpackages . /dev/null | gzip > Packages.gz
% dpkg-scansources . /dev/null | gzip > Sources.gz
% cd ../etch/
% dpkg-scanpackages . /dev/null | gzip > Packages.gz
% dpkg-scansources . /dev/null | gzip > Sources.gz

It gives an error,
** Packages in archive but missing from override file: **
but seems to work anyway.

Now you can use one of the following /etc/apt/sources.list lines:
deb http://shallowsky.com/apt/hardy ./
deb http://shallowsky.com/apt/etch ./

After an apt-get update, it saw pho, but it warned me

WARNING: The following packages cannot be authenticated!
  pho
Install these packages without verification [y/N]?

There's some discussion in the SecureAPT page on the Debian wiki, but it's a bit involved and I'm not clear if it helps me if I'm not already part of the official Debian keychain.

This page on Release check of non Debian sources was a little more helpful, and told me how to create the Release and Release.gpg file -- but then I just get a different error,

 The following signatures couldn't be verified because the public key is not available: NO_PUBKEY

And worse, it's an error now, not just a warning, preventing any apt-get update.

Going back to the SecureApt page, under Setting up a secure apt repository they give the two steps the other page gave for creating Release and Release.gpg, with a third step: "Publish the key fingerprint, that way your users will know what key they need to import in order to authenticate the files in the archive."

So apparently if users don't take steps to import the key manually, they can't update at all. Whereas if I leave out the Release and Release.gpg files, all they have to do is type y when they see the warning. Sounds like it's better to leave off the key. I wish, though, that there was a middle ground, where I could offer the key for those who wanted it without making it harder for those who don't care.

Tags: programming, pho, debian, ubuntu, linux
[ 20:14 Jan 13, 2009    More linux | permalink to this entry ]

Cleaning up the edges of Moonroot's transparent images

I wrote moonroot more to figure out how to do it than to run it myself. But on the new monitor I have so much screen real estate that I've started using it -- but the quality of the images was such an embarrassment that I couldn't stand it. So I took a few minutes and cleaned up the images and made a moonroot 0.6 release.

Turned out there was a trick I'd missed when I originally made the images, years ago. XPM apparently only allows 1-bit transparency. When I was editing the RGB image and removing the outside edge of the circle, some of the pixels ended up semi-transparent, and when I saved the file as .xpm, they ended up looking very different (much darker) from what I had edited.

Here are two ways to solve that in GIMP:

1. Use the "Hard edge" option on the eraser tool (and a hard-edged brush, of course, not a fuzzy one).
2. Convert the image to indexed, in which case GIMP will only allow one bit's worth of transparency. (That doesn't help for full-color images, but for a greyscale image like the moon, there's no loss of color since even RGB images can only have 8 bits per channel.)

Either way, the way to edit a transparent image where you're trying to make the edges look clean is to add a solid-color background layer (I usually use white, but of course it depends on how you're going to use the image) underneath the layer you're trying to edit. (In the layers dialog, click the New button, chose White for the new layer, click the down-arrow button to move it below the original layer, then click on the original layer so your editing will all happen there.)

Once you're editing a circle with sharp edges, you'll probably need to adjust the colors for some of the edge pixels too. Unfortunately the Smudge tool doesn't seem to work on indexed images, so you'll probably spend a lot of time alternating between the Color Picker and the Pencil tool, picking pixel colors then dabbing them onto other pixels. Key bindings are the best way to do that: o activates the Color Picker, N the Pencil, P the Paintbrush. Even if you don't normally use those shortcuts it's worth learning them for the duration of this sort of operation.

Or use the Clone tool, where the only keyboard shortcut you have to remember is Ctrl to pick a new source pixel. (I didn't think of that until I was already finished, but it works fine.)

Tags: programming, astronomy, gimp
[ 14:48 Nov 16, 2008    More gimp | permalink to this entry ]

Pho 0.9.6 finally released!

Pho 0.9.6-pre3 has been working great for me for about a month, and I've been trying to find the time to do a release. I finally managed it this weekend, after making a final tweak to change the default PHO_REMOTE command from gimp-remote to gimp since gimp-remote is obsolete and is no longer built by default.

The big changes from 0.9.5 are Keywords mode, slideshow mode, the new PHO_REMOTE environment variable, swapping -f and -F, and a bunch of performance work and minor bug fixing.

I built deb packages for Ubuntu (Hardy, but they should work on Intrepid too) and Debian (Etch), as well as the usual source tarball, and they're available at the usual place:

http://shallowsky.com/software/pho.

Tags: programming, pho, image viewer
[ 17:11 Nov 09, 2008    More programming | permalink to this entry ]

A word count bookmarklet

This posting ended up being published as a Linux Planet Quick Tip. You can read about my nifty word counting bookmarklet there: Quick Firefox Tip: Word Count Bookmarklet.

Tags: firefox, mozilla, bookmarklets, writing, programming
[ 22:41 Nov 03, 2008    More tech/web | permalink to this entry ]

Requesting no window decorations (and moonroot 0.4)

Someone on #openbox this morning wanted help in bringing up a window without decorations -- no titlebar or window borders.

Afterward, Mikael commented that the app should really be coded not to have borders in the first place.

Me: You can do that?

Turns out it's not a standard ICCCM request, but one that mwm introduced, MWM_HINTS_DECORATIONS. Mikael pointed me to the urxvt source as an example of an app that uses it.

My own need was more modest: my little moonroot Xlib program that draws the moon at approximately its current phase. Since the code is a lot simpler than urxvt, perhaps the new version, moonroot 0.4, will be useful as an example for someone (it's also an example of how to use the X Shape extension for making non-rectangular windows).

Tags: programming, xlib, astronomy, window managers
[ 11:06 Oct 20, 2008    More programming | permalink to this entry ]

New Pho 0.9.6-pre3

I've released Pho 0.9.6-pre3. The only change is to fix a sporadic bug where pho would sometimes jump back to the first image after deleting the last one, rather than backing up to the next-to-last image. I was never able to reproduce the bug reliably, but I cleaned up the image list next/prev code quite a bit and haven't seen the bug since then. I'd appreciate having a few testers exercising this code as much as possible.

Otherwise pho is looking pretty solid for a 0.9.6 release.

Tags: programming, pho, image viewer
[ 09:57 Oct 02, 2008    More programming | permalink to this entry ]

Fast Pixel Ops in GIMP-Python

Last night Joao and I were on IRC helping someone who was learning to write gimp plug-ins. We got to talking about pixel operations and how to do them in Python. I offered my arclayer.py as an example of using pixel regions in gimp, but added that C is a lot faster for pixel operations. I wondered if reading directly from the tiles (then writing to a pixel region) might be faster.

But Joao knew a still faster way. As I understand it, one major reason Python is slow at pixel region operations compared to a C plug-in is that Python only writes to the region one pixel at a time, while C can write batches of pixels by row, column, etc. But it turns out you can grab a whole pixel region into a Python array, manipulate it as an array then write the whole array back to the region. He thought this would probably be quite a bit faster than writing to the pixel region for every pixel.

He showed me how to change the arclayer.py code to use arrays, and I tried it on a few test layers. Was it faster? I made a test I knew would take a long time in arclayer, a line of text about 1500 pixels wide. Tested it in the old arclayer; it took just over a minute to calculate the arc. Then I tried Joao's array version: timing with my wristwatch stopwatch, I call it about 1.7 seconds. Wow! That might be faster than the C version.

The updated, fast version (0.3) of arclayer.py is on my arclayer page.

If you just want the trick to using arrays, here it is:

from array import array

[ ... setting up ... ]
        # initialize the regions and get their contents into arrays:
        srcRgn = layer.get_pixel_rgn(0, 0, srcWidth, srcHeight,
                                     False, False)
        src_pixels = array("B", srcRgn[0:srcWidth, 0:srcHeight])

        dstRgn = destDrawable.get_pixel_rgn(0, 0, newWidth, newHeight,
                                            True, True)
        p_size = len(srcRgn[0,0])               
        dest_pixels = array("B", "\x00" * (newWidth * newHeight * p_size))

[ ... then inside the loop over x and y ... ]
                        src_pos = (x + srcWidth * y) * p_size
                        dest_pos = (newx + newWidth * newy) * p_size
                        
                        newval = src_pixels[src_pos: src_pos + p_size]
                        dest_pixels[dest_pos : dest_pos + p_size] = newval

[ ... when the loop is all finished ... ]
        # Copy the whole array back to the pixel region:
        dstRgn[0:newWidth, 0:newHeight] = dest_pixels.tostring() 

Good stuff!

Tags: gimp, python, programming, performance
[ 21:02 Aug 16, 2008    More gimp | permalink to this entry ]

GetSET: Teaching Javascript to high school girls

Every summer I volunteer as an instructor for a one-day Javascript programming class at the GetSET summer technology camp for high school girls. GetSET is a great program run by the Society of Women Engineers. it's intended for minority girls from relatively poor neighborhoods, and the camp is free to the girls (thanks to some great corporate sponsors). They're selected through a competitive interview process so they're all amazingly smart and motivated, and it's always rewarding being involved.

Teaching programming in one day to people with no programming background at all is challenging, of course. You can't get into any of the details you'd like to cover, like style, or debugging techniques. By the time you get through if-else, for and while loops, some basic display methods, the usual debugging issues like reading error messages, and typographical issues like "Yes, uppercase and lowercase really are different" and "No, sorry, that's a colon, you need a semicolon", it's a pretty full day and the students are saturated.

I got drafted as lead presenter several years ago, by default by virtue of being the only one of the workshop leaders who actually programs in Javascript. For several years I'd been asking for a chance to rewrite the course to try to make it more fun and visual (originally it used a lot of form validation exercises), and starting with last year's class I finally got the chance. I built up a series of graphics and game exercises (using some of Sara Falamaki's Hangman code, which seemed perfect since she wrote it when she was about the same age as the girls in the class) and it went pretty well. Of course, we had no idea how fast the girls would go or how much material we could get through, so I tried to keep it flexible and we adjusted as needed.

Last year went pretty well, and in the time since then we've exchanged a lot of email about how we could improve it. We re-ordered some of the exercises, shifted our emphasis in a few places, factored some of the re-used code (like windowWidth()) into a library file so the exercise files weren't so long, and moved more of the visual examples earlier.

I also eliminated a lot of the slides. One of the biggest surprises last year was the "board work". I had one exercise where the user clicks in the page, and the student has to write the code to figure out whether the click was over the image or not. I had been nervous about that exercise -- I considered it the hardest of the exercises. You have to take the X and Y coordinates of the mouse click, the X and Y coordinates of the image (the upper left corner of the <div> or <img> tag), and the size of the image (assumed to be 200x200), and turn that all into a couple of lines of working Javascript code. Not hard once you understand the concepts, but hard to explain, right?

I hadn't made a slide for that, so we went to the whiteboard to draw out the image, the location of the mouse click, the location of the image's upper left corner, and figure out the math ... and the students, who had mostly been sitting passively through the heavily slide-intensive earlier stuff, came alive. They understood the diagram, they were able to fill in the blanks and keep track of mouse click X versus image X, and they didn't even have much trouble turning that into code they typed into their exercise. Fantastic!

Remembering that, I tried to use a lot fewer slides this year. I felt like I still needed to have slides to explain the basic concepts that they actually needed to use for the exercises -- but if there was anything I thought they could figure out from context, or anything that was just background, I cut it. I tried for as few slides as possible between exercises, and more places where we could elicit answers from the students. I think we still have too many slides and not enough "board work" -- but we're definitely making progress, and this year went a lot better and kept them much better engaged. We're considering next year doing the first several exercises on the board first, then letting them type it in to their own copies to verify that it works.

We did find we needed to leave code examples visible: after showing slides saying something like "Ex 7: Write a loop that writes a line of text in each color", I had to back up to the previous slide where I'd showed what the code actually looked like. I had planned on their using my "Javascript Quick Reference" handout for reference and not needing that information on the slides; but in fact, I think they were confused about the quickref and most never even opened it. Either that information needs to be in the handout, or it needs to be displayed on the screen as they work, or I have to direct them to the quickref page explicitly ("Now turn to page 3 in ...") or put that information in the exercises.

The graphical flower exercises were a big hit this year (I showed them early and promised we'd get to them, and when we did, just before lunch, several girls cheered) and, like last year, some of the girls who finished them earlier decided on their own that they wanted to change them to use other images, which was also a big hit. Several other girls decided they wanted more than 10 flowers displayed, and others hit on the idea of changing the timeout to be a lot shorter, which made for some very fun displays. Surprisingly, hardly anyone got into infinite loops and had to kill the browser (always a potential problem with javascript, especially when using popups like alert() or prompt()).

I still have some issues I haven't solved, like what to do about semicolons and braces. Javascript is fairly agnostic about them. Should I tell the girls that they're required? (I did that this year, but it's confusing because then when you get to "if" statements you have to explain why that's different.) Not mention them at all? (I'm leaning toward that for next year.)

And it's always a problem figuring out what the fastest girls should do while waiting for the rest to finish. This year, in addition to trying to make each exercise shorter, we tried having the girls work on them in groups of two or three, so they could help each other. It didn't quite work out that way -- they all worked on their own copies of the exercises but they did seem to collaborate more, and I think that's the best balance. We also encourage the ones who finish first to help the girls around them, which mostly they do on their own anyway.

And we really do need to find a better editor we can use on the Windows lab machines instead of Wordpad. Wordpad's font is too small on the projection machine, and on the lab machines it's impossible for most of us to tell the difference between parentheses, brackets and braces, which leads to lots of time-wasting subtle bugs. Surely there's something available for Windows that's easy to use, freely distributable, makes it easy to change the font, and has parenthesis and brace matching (syntax highlighting would be nice too). Well, we have a year to look for one now.

All in all, we had a good day and most of the girls gave the class high marks. Even the ones who concluded "I learned I shouldn't be a programmer because it takes too much attention to detail" said they liked the class. And we're fine with that -- not everybody wants to be a programmer, and the point isn't to force them into any specific track. We're happy if we can give them an idea of what computer programming is really like ... then they'll decide for themselves what they want to be.

Tags: education, javascript, programming, speaking
[ 11:54 Aug 09, 2008    More education | permalink to this entry ]

Crikey in Python, and generating key events with XTest

A user on the One Laptop Per Child (OLPC, also known as the XO) platform wrote to ask me how to use crikey on that platform.

There are two stages to getting crikey running on a new platform:

1. Build it, and
2. Figure out how to make a key run a specific program.

The crikey page contains instructions I've collected for binding keys in various window managers, since that's usually the hard part. On normal Linux machines the first step is normally no problem. But apparently the OLPC comes with gcc but without make or the X header files. (Not too surprising: it's not a machine aimed at developers and I assume most people developing for the machine cross-compile from a more capable Linux box.)

We're still working on that (if my correspondant gets it working, I'll post the instructions), but while I was googling for information about the OLPC's X environment I stumbled upon a library I didn't know existed: python-xlib. It turns out it's possible to do most or all of what crikey does from Python. The OLPC is Python based; if I could write crikey in Python, it might solve the problem. So I whipped up a little key event generating script as a test.

Unfortunately, it didn't solve the OLPC problem (they don't include python-xlib on the machine either) but it was a fun exercises, and might be useful as an example of how to generate key events in python-xlib. It supports both event generating methods: the X Test extension and XSendEvent. Here's the script: /pykey-0.1.

But while I was debugging the X Test code, I had to solve a bug that I didn't remember ever solving in the C version of crikey. Sure enough, it needed the same fix I'd had to do in the python version. Two fixes, actually. First, when you send a fake key event through XTest, there's no way to specify a shift mask. So if you need a shifted character like A, you have to send KeyPress Shift, KeyPress a. But if that's all you send, XTest on some systems does exactly what the real key would do if held down and never released: it autorepeats. (But only for a little while, not forever. Go figure.)

So the real answer is to send KeyPress Shift, KeyPress a, KeyRelease a, KeyRelease Shift. Then everything works nicely. I've updated crikey accordingly and released version 0.7 (though since XTest isn't used by default, most users won't see any change from 0.6). In the XSendEvent case, crikey still doesn't send the KeyRelease event -- because some systems actually see it as another KeyPress. (Hey, what fun would computers be if they were consistent and always predictable, huh?)

Both C and Python versions are linked off the crikey page.

Tags: programming, crikey, X11, python
[ 14:50 May 25, 2008    More programming | permalink to this entry ]

Setting app name and class in Xlib

I was looking at Dave's little phase-of-the-moon Mac application, and got the urge to play with moonroot, the little xlib ditty I wrote several years ago to put a moon (showing the right phase) on the desktop.

I fired it up, and got the nice moon-shaped window ... but with a titlebar. I didn't want that! Figuring out how to get rid of the titlebar in openbox was easy, just

<application name="moonroot">
    <decor>no</decor>
    <desktop>all</desktop>
</application>

... but it didn't work! A poke with xwininfo showed the likely cause: instead of "moonroot", the window was listed as "Unnamed window". Whoops!

A little poking around revealed three different ways to set "name" for a window: XStoreName, XSetClassHint (which sets both class name and app name), and XSetWMName. Available online documentation on these functions was not very helpful in explaining the differences; fortunately someone hanging out on the openbox channel knew the difference (thanks, Crazy_Hopper). Thus:

• XSetWMName sets a property called XA_NAME which is primarily used to update the window's titlebar. Note that this may be more than the app name (for instance, Firefox puts the title of the current page in the titlebar). To use XSetWMName, you have to set up and populate an XTextProperty structure, which first requires that you set up a string list then run XStringListToTextProperty -- not difficult but it's several annoying steps.
• XStoreName is a shortcut to XSetWMName, a way to set the XA_NAME (titlebar name) in one step.
• XSetClassHint sets two properties at once: a name hint and a class hint. This is the name and class that the window manager uses for directives like suppressing the titlebar.

I didn't see much in the way of example code for what an app ought to do with these, so I'll post mine here:

    char* appname;
    XClassHint* classHint;
[ ... ]
    if (argv && argc > 1)
        appname = basename(argv[0]);
    else
        appname = "moonroot";

    /* set the titlebar name */
    XStoreName(dpy, win, appname);

    /* set the name and class hints for the window manager to use */
    classHint = XAllocClassHint();
    if (classHint) {
        classHint->res_name = appname;
        classHint->res_class = "MoonRoot";
    }
    XSetClassHint(dpy, win, classHint);
    XFree(classHint);

And if anyone is interested in my silly moon program, it's at moonroot-0.3.tar.gz. moonroot gives you a large moon, moonroot -s gives a smaller one. I'm not terribly happy with its accuracy and wasted too much time today fiddling with it and verifying that it's doing the right time conversions. All I can figure is that the approximation in Meeus' Astronomical Algorithms is way too approximate (it's sometimes off by more than a day) and I should just rewrite all my moon programs to calculate moon phase the hard (and slow) way.

Tags: programming, xlib, astronomy
[ 20:15 Mar 18, 2008    More programming | permalink to this entry ]

Script to add tags

Python is so cool. I love how I'll be working on a script and suddenly think "Oh, it should also do X, but I bet that'll be a lot more work", and then it occurs to me that I can do exactly that by adding about 2 more lines of python. And I add them and it works the first time.

Anyway, it turned out to be very easy to go through all existing blog articles and add tags for the current category hierarchy, being careful to preserve each file's last-modified date since that's what pyblosxom uses for the date of the entry. add-tags.py

Tags: blogging, programming
[ 18:37 Feb 29, 2008    More blogging | permalink to this entry ]

Got tags working with pybloxsom

Entries on this blog are arranged by category. But all too often I have something that really belongs equally well in two categories. Since pyblosxom's categories follow the hierarchy on disk, there's no way to have an entry in two categories. Enter tags.

Tags are a way of assigning any number of keywords to each blog entry. Search engines apparently pay attention to tags, and most tagged blogs also let you search by tag.

I wanted my tags to follow whatever canonical tag format the big blogging sites use, so search engines would index them. Unfortunately, this isn't well documented anywhere. Wikipedia has a tags entry that mentions a couple of common formats; the HTML format given in that entry (<a rel="tag" ...>) turns out to be the format used on most popular sites like livejournal and blogspot, so that's what I wanted to use. Later, someone pointed me to a much better tag explanation on technorati, which is useful whether or not you decide to register with technorati.

Next: how to implement searching? The simplest pyblosxom tags plug-in is called simply tags.py. All the others are much more complex and do tons of things I'm not interested in. But tags.py doesn't support static mode, and points to a modified tags.py that's supposedly modified to work with static blogs.

Alas, when I tried that version, it didn't work (and an inquiry on the pybloxsom list got a response from someone who agreed it didn't work). So I hacked around and eventually got it working. Here's a diff for what I changed or just the tags-static.py plug-in.

Additional steps I needed that weren't mentioned in tags.py:

• Add "#tags foo,bar" directives as the second line of an entry, right under the title; anywhere else in the file it will be ignored.
• You may ned to create the tag directories http://yoursite/tags/$tagname yourself (pyblosxom created the directories for me on the web server, but not on the machine where I first tested).
• In addition to the config file entries discussed below, if you use an extension other than .txt (or maybe even if you don't) you also need to set py[ 'taggable_files' ] = [ "ext" ]
• In your story.html template, include $tag_links wherever you want the tags line to go. But make "Tags: " or something similar be part of the pretext, so it won't be included on un-tagged entries.

I also wrote a little python index.cgi for my blog's /tags directory, so you can see the list of tags used so far. Strangely, tags.py didn't create any such index, and it was easier to make a cgi than to figure out how to do it from a blosxom plug-in.

And as long as I'm posting pyblosxom diffs, here's the little filename diff for 1.4.3 that I apply to pyblosxom whenever I update it, to let me use the .blx extension rather than .txt for my blog source files. (That way I can configure my editor to treat blog files as html, which they are -- they aren't plaintext.)

Anyway, it all seems to be working now, and in theory I can tag all future articles. I'll probably go back and gradually add tags to older articles, but that's a bigger project and there's no rush.

Tags: blogging, programming
[ 15:04 Feb 27, 2008    More blogging | permalink to this entry ]

PyTopo and PyGTK pixbuf memory leakage

On a recent Mojave desert trip, we tried to follow a minor dirt road that wasn't mapped correctly on any of the maps we had, and eventually had to retrace our steps. Back at the hotel, I fired up my trusty PyTopo on the East Mojave map set and tried to trace the road. But I found that as I scrolled along the road, things got slower and slower until it just wasn't usable any more.

PyTopo was taking up all of my poor laptop's memory. Why? Python is garbage collected -- you're not supposed to have to manage memory explicitly, like freeing pixbufs. I poked around in all the sample code and man pages I had available but couldn't find any pygtk examples that seemed to be doing any explicit freeing.

When we got back to civilization (read: internet access) I did some searching and found the key. It's even in the PyGTK Image FAQ, and there's also some discussion in a mailing list thread from 2003.

Turns out that although Python is supposed to handle its own garbage collection, the Python interpreter doesn't grok the size of a pixbuf object; in particular, it doesn't see the image bits as part of the object's size. So dereferencing lots of pixbuf objects doesn't trigger any "enough memory has been freed that it's time to run the garbage collector" actions.

The solution is easy enough: call gc.collect() explicitly after drawing a map (or any other time a bunch of pixbufs have been dereferenced).

So there's a new version of PyTopo, 0.6 that should run a lot better on small memory machines, plus a new collection format (yet another format from the packaged Topo! map sets) courtesy of Tom Trebisky.

Oh ... in case you're wondering, the ancient USGS maps from Topo! didn't show the road correctly either.

Tags: programming, python, gtk, pygtk
[ 21:21 Oct 12, 2007    More programming | permalink to this entry ]

Egg Timer in Python and TkInter

I left the water on too long in the garden again. I keep doing that: I'll set up something where I need to check back in five minutes or fifteen minutes, then I get involved in what I'm doing and 45 minutes later, the cornbread is burnt or the garden is flooded.

When I was growing up, my mom had a little mechanical egg timer. You twist the dial to 5 minutes or whatever, and it goes tick-tick-tick and then DING! I could probably find one of those to buy (they're probably all digital now and include clocks and USB plugs and bluetooth ports) but since the problem is always that I'm getting distracted by something on the computer, why not run an app there?

Of course, you can do this with shell commands. The simple solution is:

(sleep 300; zenity --info --text="Turn off the water!") &

But the zenity dialogs are small -- what if I don't notice it? -- and besides, I have to multiply by 60 to turn a minute delay into sleep seconds. I'm lazy -- I want the computer to do that for me!
Update: Ed Davies points out that "sleep 5m" also works.

A slightly more elaborate solution is at. Say something like: at now + 15 minutes and when it prompts for commands, type something like:

export DISPLAY=:0.0
zenity --info --text="Your cornbread is ready"

to pop up a window with a message. But that's too much typing and has the same problem of the small easily-ignored dialogs. I'd really rather have a great big red window that I can't possibly miss.

Surely, I thought, someone has already written a nice egg-timer application! I tried aptitude search timer and found several apps such as gtimer, which is much more complicated than I wanted (you can define named events and choose from a list of ... never mind, I stopped reading there). I tried googling, but didn't have much luck there either (lots of Windows and web apps, no Linux apps or cross-platform scripts).

Clearly just writing the damn thing was going to be easier than finding one. (Why is it that every time I want to do something simple on a computer, I have to write it? I feel so sorry for people who don't program.)

I wanted to do it in python, but what to use for the window that pops up? I've used python-gtk in the past, but I've been meaning to check out TkInter (the gui toolkit that's kinda-sorta part of Python) and this seemed like a nice opportunity since the goal was so simple.

The resulting script: eggtimer. Call it like this:

eggtimer 5 Turn off the water

and in five minutes, it will pop up a huge red window the size of the screen with your message in big letters. (Click it or hit a key to dismiss it.)

First Impressions of TkInter

It was good to have an excuse to try TkInter and compare it with python-gtk. TkInter has been recommended as something normally installed with Python, so the user doesn't have to install anything extra. This is apparently true on Windows (and maybe on Mac), but on Ubuntu it goes the other way: I already had pygtk, because GIMP uses it, but to use TkInter I had to install python-tk.

For developing I found TkInter irritating. Most of the irritation concerned the poor documentation: there are several tutorials demonstrating very basic uses, but not much detailed documentation for answering questions like "What class is the root Tk() window and what methods does it have?" (The best I found -- which never showed up in google, but was referenced from O'Reilly's Programming Python -- was here.) In contrast, python-gtk is very well documented.

Things I couldn't do (or, at least, couldn't figure out how to do, and googling found only postings from other people wanting to do the same thing):

• Button didn't respond to any of the obvious keys, like Return or Space, and in fact setting key handlers on the button didn't work -- I ended up setting a key handler on the root window.
• I couldn't find a way to set the root window size and background explicitly, so I had to set approximate window size by guessing at the size of the internal padding of the button.
• There's an alternate to the root Tk() window called Toplevel, which is documented and does allow setting window size. Unfortunately, it also pops up an empty dialog without being told to (presumably a bug).
• All of the tutorials I found for creating dialogs was wrong, and I finally gave up on dialogs and just used a regular window.
• I couldn't fork and return control to the shell, because TkInter windows don't work when called from a child process (for reasons no one seems to be able to explain), so you have to run it in the background with & if you want your shell prompt back.

I expect I'll be sticking with pygtk for future projects. It's just too hard figuring things out with no documentation. But it was fun having an excuse to try something new.

Tags: programming, python, tkinter
[ 13:35 Sep 04, 2007    More programming | permalink to this entry ]

Find the nearest matching color name

Someone showed up on #gimp today with a color specified as an HTML hex color specifier, and wanted to know how to find the nearest color name.

Easy, right? There have got to be a bazillion pages that do that, plus at least a couple of Linux apps.

But I googled for a while and couldn't find a single one. There are lots of pages that list all the RGB colors, or convert decimal red, green and blue into HTML #nnn hex codes, or offer aesthetic advice about pleasing combinations of colors for themes (including this lovely page on butterfly-inspired color themes, courtesy of Rik) but nothing I could find that gave color names. Apparently there used to be a Linux app that did that, a piece of Gnome 1 called GColorSel, but it's gone now.

I got to thinking (always dangerous!) ... /etc/X11/rgb.txt has a list of color names with their RGB color equivalents. It would be really easy to write something that just read down the list finding the ones closest to the specified color.

Uh-oh ... of course, once that thought occurred to me, I was doomed. Programmer's disease. I had to write it. So I did, and here it is: Find the Nearest Matching Color Name. It checks against both rgb.txt and the much smaller list of 17 CSS color names.

Tags: programming
[ 22:18 Jun 11, 2007    More programming | permalink to this entry ]

A Culture of Regressions (or, Why I no longer work on Mozilla)

A couple of friends periodically pester me to write about why I stopped contributing to Mozilla after so many years with the project. I've held back, feeling like it's airing dirty laundry in public.

But a discussion on mozilla.dev.planning over the last week, started by Nelson Bolyard, aired it for me: it was their culture of regressions.

I love Mozilla technology. I'm glad it exists, and I still use it for my everyday browsing. But trying to contribute to Mozilla just got too frustrating. I spent more time chasing down and trying to fix other people's breakages than I did working on anything I wanted to work on.

That might be okay, barely, when you're getting paid for it. But when you're volunteering your own time, who wants to spend it fixing code checked in by some other programmer who just can't be bothered to clean up his own mess?

It's the difference between spending a day cleaning your own house ... and spending every day cleaning other people's houses.

Nelson said it eloquently in this exchange:

(Robert Kaiser writes)
As we are open source, everyone can access and test that code, and find and file the regressions, so that they get fixed over time.

(Boris Zbarsky writes)
That last conclusion doesn't necessarily follow. To get them fixed you need someone fixing them.

(Nelson Bolyard writes)
We're very unlikely to get volunteers to spent large amounts of effort, rewriting formerly working code to get it to work again, after it was broken by someone else's checkin. This demotivates developers and drives them away. They think "why should I keep working on this when others can break my code and I must pay for their mistakes?" and "I worked hard to get that working, and now person X has broken it. Let HIM fix it."

This was exactly how I felt, and it's the reason I quit working on Mozilla.

A little later in the thread, Boris Zbarsky reports that the trunk has been so broken with regressions that it's been unusable for him for weeks or months. (When you have someone as committed and sharp as Boris unable to use your software, you know there's something wrong with your project's culture.) He writes: "For example, on my machine (Linux) about one in three SVG testcases in Bugzilla causes trunk Gecko to hang X ..."

Justin Dolske replies, "Oh, Linux," and asks if it's related to turning on Cairo. Boris replies affirmatively. Just another example where a change was checked in that caused serious regressions keeping at least one important contributor from using the browser on a regular basis; yet it's still there and hasn't been backed out. Of course, it's "only Linux".

David Baron appears to take Nelson's concerns seriously, and suggests criteria for closing the tree and making everyone stop work to track down regressions. As he correctly comments, closing the tree is very serious and inefficient, and should be avoided in all but the most serious cases.

But Nelson repeats the real question:

(Nelson Bolyard writes)
Under what circumstances does a Sheriff back out a patch due to functional regressions? From what you wrote above, I gather it's "never". :(

Alas, the thread peters out after that; there's no reply to Nelson's question.

The problem with Mozilla isn't that there are regressions. Mistakes happen. The problem is that regressions never get fixed, because the project's culture encourages regressions. The prevailing attitude is that it's okay to check in changes that break other people's features, as long as your new feature is cool enough or the right people want it. If you break something, well, hey, someone will figure out a fix eventually. Or not. Either way, it's not your problem.

Working on new features is fun, and so is getting the credit for being the one to check them in. Fixing bugs, writing API documentation, extensive testing -- these things aren't fun, they're hard work, and there isn't much glory in them either (you don't get much appreciation or credit for it). So why do them if you don't have to? Let someone else worry about it, as long as the project lets you get away with it!

A project with a culture of responsibility would say that the person who broke something should fix it, and that broken stuff should stay out of the tree. If programmers don't do that themselves just because it's the right thing to do, the project could enforce it: just insist that regression-causing changes that can't be fixed right away be backed out. Fix the regressions out of the tree where they aren't causing problems for other people. Get help from people to test it and to integrate it with those other modules you forgot about the first time around.

Yes, even if it's a change that's needed -- even if it's something a lot of people want. If it's a good change, there will always be time to check it in later.

When it's really working.

Tags: programming
[ 10:07 May 29, 2007    More programming | permalink to this entry ]

A Kitfox Extension

For a bit over a year I've been running a patched version of Firefox, which I call Kitfox, as my main browser. I patch it because there are a few really important features that the old Mozilla suite had which Firefox removed; for a long time this kept me from using Firefox (and I'm not the only one who feels that way), but when the Mozilla Foundation stopped supporting the suite and made Firefox the only supported option, I knew my only choice was to make Firefox do what I needed. The patches were pretty simple, but they meant that I've been building my own Firefox all this time.

Since all my changes were in JavaScript code, not C++, I knew this was probably all achievable with a Firefox extension. But never around to it; building the Mozilla source isn't that big a deal to me. I did it as part of my job for quite a few years, and my desktop machine is fast enough that it doesn't take that long to update and rebuild, then copy the result to my laptop.

But when I installed the latest Debian, "Etch", on the laptop, things got more complicated. It turns out Etch is about a year behind in its libraries. Programs built on any other system won't run on Etch. So I'd either have to build Mozilla on my laptop (a daunting prospect, with builds probably in the 4-hour range) or keep another system around for the purpose of building software for Etch. Not worth it. It was time to learn to build an extension.

There are an amazing number of good tutorials on the web for writing Firefox extensions (I won't even bother to link to any; just google firefox extension and make your own choices). They're all organized as step by step examples with sample code. That's great (my favorite type of tutorial) but it left my real question unanswered: what can you do in an extension? The tutorial examples all do simple things like add a new menu or toolbar button. None of them override existing Javascript, as I needed to do.

Canonical URL to the rescue. It's an extension that overrides one of the very behaviors I wanted to override: that of adding "www." to the beginning and ".com" or ".org" to the end of whatever's in the URLbar when you ctrl-click. (The Mozilla suite behaved much more usefully: ctrl-click opened the URL in a new tab, just like ctrl-clicking on a link. You never need to add www. and .com or .org explicitly because the URL loading code will do that for you if the initial name doesn't resolve by itself.) Canonical URL showed me that all you need to do is make an overlay containing your new version of the JavaScript method you want to override. Easy!

So now I have a tiny Kitfox extension that I can use on the laptop or anywhere else. Whee!

Since extensions are kind of a pain to unpack, I also made a source tarball which includes a simple Makefile: kitfox-0.1.tar.gz.

Tags: tech, web, mozilla, firefox, programming
[ 10:59 May 27, 2007    More tech/web | permalink to this entry ]

Pho 0.9.5 Released

Pho 0.9.5-pre5 has been working nicely since I released it two weeks ago. And meanwhile, I've already started working on the next version. So I've released it as 0.9.5 with no changes (except for version string and some updates to the documentation and debian config files).

I made a .deb on Ubuntu Edgy, but haven't actually tested it yet (anyone who sees problems, please let me know) and I'll try to make a straight Debian package on Sarge sometime soon.

So what's this stuff I've been working on for the next version? Image categorization. I shoot so many photos, and categorizing them by keyword can be a lot of work. Although Pho's "Notes 0 through 9" are helpful for a small number of notes, it's tough keeping track of which note corresponds to which keyword when I'm categorizing a directory full of photos from a trip. The next Pho release (which will have a much shorter release cycle than 0.9.5 did, honest!) will have an optional Keywords dialog where you can type in keywords and associate them with photos. I know there are apps such as f-spot, gthumb and Picasa, but they all seem much more heavyweight than what I need, and Pho only needs a tiny bit of work to get there.

While I'm working on dialogs, I'm also cleaning up modality: Pho dialogs will now stay visible so they can't get lost behind the image, and the question dialog ("Really delete?" or "Do you want to quit?" will be modal.

But that's all coming in the next version. For now, 0.9.5 is the stable version: get it from the Pho page.

Tags: programming
[ 15:56 Mar 22, 2007    More programming | permalink to this entry ]

Pho 0.9.5-pre5

Pho's been static for a long time -- it's been working well enough that I keep forgetting that there were a couple of bugs that need fixing for a 0.9.5 release.

I had some time tonight, so I dug in and fixed the bugs I remembered: some issues with zooming in and out, a bug with aspect ratio when switching out of fullscreen mode, and the fact that Note 0 didn't work.

While I was at it, I added an environment variable, PHO_ARGS, where you can preset your default values. I find that I always want -p (presentation mode), so now I can specify that with PHO_ARGS=p, and use pho -P when I want window borders. I also updated the man page.

I'll test this for a little while and if nobody finds any serious bugs, maybe I can finally release 0.9.5.

Get Pho here.

Tags: programming
[ 23:06 Mar 05, 2007    More programming | permalink to this entry ]

Random Wallpaper

I was talking about desktop backgrounds -- wallpaper -- with some friends the other day, and it occurred to me that it might be fun to have my system choose a random backdrop for me each morning.

Finding backgrounds is no problem: I have plenty of images stored in ~/Backgrounds -- mostly photos I've taken over the years, with a smattering of downloads from sites like the APOD. So all I needed was a way to select one file at random from the directory.

This is Unix, so there's definitely a commandline way to do it, right? Well, surprisingly, I couldn't find an easy way that didn't involve any scripting. Some shells have a random number generator built in ($RANDOM in bash) but you still have to do some math on the result.

Of course, I could have googled, since I'm sure other people have written random-wallpaper scripts ... but what's the fun in that? If it has to be a script, I might as well write my own.

Rather than write a random wallpaper script, I wanted something that could be more generally useful: pick one random line from standard input and print it. Then I could pass it the output of ls -1 $HOME/Backgrounds, and at the same time I'd have a script that I could also use for other purposes, such as choosing a random quotation, or choosing a "flash card" question when studying for an exam.

The obvious approach is to read all of standard input into an array, count the lines, then pick a random number between one and $num_lines and print that array element. It took no time to whip that up in Python and it worked fine. But it's not very efficient -- what if you're choosing a line from a 10Mb file?

Then Sara Falamaki (thanks, Sara!) pointed me to a page with a neat Perl algorithm. It's Perl so it's not easy to read, but the algorithm is cute. You read through the input line by line, keeping track of the line number. For each line, the chance that this line should be the one printed at the end is the reciprocal of the line number: in other words, there's one chance out of $line_number that this line is the one to print.

So if there's only one line, of course you print that line; when you get to the second line, there's one chance out of two that you should switch; on the third, one chance out of three, and so on.

A neat idea, and it doesn't require storing the whole file in memory. In retrospect, I should have thought of it myself: this is basically the same algorithm I used for averaging images in GIMP for my silly Chix Stack Mars project, and I later described the method in the image stacking section of my GIMP book. To average images by stacking them, you give the bottom layer 100% opacity, the second layer 50% opacity, the third 33% opacity, and so on up the stack. Each layer makes an equal contribution to the final result, so what you see is the average of all layers.

The randomline script, which you can inspect here, worked fine, so I hooked it up to accomplish the original problem: setting a randomly chosen desktop background each day. Since I use a lightweight window manager (fvwm) rather than gnome or kde, and I start X manually rather than using gdm, I put this in my .xinitrc:

(xsetbg -fullscreen -border black `find $HOME/Backgrounds -name "*.*" | randomline`) &

Update: I've switched to using hsetroot, which is a little more robust than xsetbg. My new command is:

hsetroot -center `find -L $HOME/Backgrounds -name "*.*" | randomline`

So, an overlong article about a relatively trivial but nontheless nifty algorithm. And now I have a new desktop background each day. Today it's something prosaic: mud cracks from Death Valley. Who knows what I'll see tomorrow?

Tags: programming, pipelines, shell
[ 13:02 Feb 28, 2007    More programming | permalink to this entry ]

PyTopo 0.5

Belated release announcement: 0.5b2 of my little map viewer PyTopo has been working well, so I released 0.5 last week with only a few minor changes from the beta. I'm sure I'll immediately find six major bugs -- but hey, that's what point releases are for. I only did betas this time because of the changed configuration file format.

I also made a start on a documentation page for the .pytopo file (though it doesn't really have much that wasn't already written in comments inside the script).

Tags: programming, python, gtk, pygtk, mapping
[ 21:10 Aug 25, 2006    More programming | permalink to this entry ]

Javascript Warnings Everywhere

I'm working on some little Javascript demos (for a workshop at this summer's Get SET girls' technology camp) so I've had the Javascript Console up for most of my browsing over the last few days. I also have Mozilla's strict Javascript checking on (user_pref("javascript.options.strict", true); in prefs.js or user.js) since I don't want to show the girls code that generates warnings. (Strict mode also reports errors in CSS.)

It's been eye opening how many sites give warnings. You know that nice clean ultra-simple Google search page? One CSS error and one JS warning. But that's peanuts to the pages of errors I see on most sites, and they're not all missing "var" declarations. I have to hit the "Clear" button frequently if I want to be able to see the errors on the pages I'm working on.

And my own sites? Yes, I admit it, I've seen some errors in my own pages too. Though it makes me feel better that there aren't very many of them (mostly CSS problems, not JS). I'm going to keep the JS Console visible more often so I'll see these errors and correct them.

Tags: programming
[ 23:06 Jul 31, 2006    More programming | permalink to this entry ]

Cleaner, More Flexible Python Map Viewing

A few months ago, someone contacted me who was trying to use my PyTopo map display script for a different set of map data, the Topo! National Parks series. We exchanged some email about the format the maps used.

I'd been wanting to make PyTopo more general anyway, and already had some hacky code in my local version to let it use a local geologic map that I'd chopped into segments. So, faced with an Actual User (always a good incentive!), I took the opportunity to clean up the code, use some of Python's support for classes, and introduce several classes of map data.

I called it 0.5 beta 1 since it wasn't well tested. But in the last few days, I had occasion to do some map exploring, cleaned up a few remaining bugs, and implemented a feature which I hadn't gotten around to implementing in the new framework (saving maps to a file).

I think it's ready to use now. I'm going to do some more testing: after visiting the USGS Open House today and watching Jim Lienkaemper's narrated Virtual Tour of the Hayward Fault, I'm all fired up about trying again to find more online geologic map data. But meanwhile, PyTopo is feature complete and has the known bugs fixed. The latest version is on the PyTopo page.

Tags: programming, python, gtk, pygtk, mapping
[ 17:25 Jun 03, 2006    More programming | permalink to this entry ]

Aid for Java Victims

Over dinner, I glanced at the cover of the latest Dr. Dobb's (a new article on Ruby on Rails), then switched to BBC World News. The first Beeb headline was Aid flow begins for Java victims.

I guess I was a little distracted from dinner preparations ... my first thought was "Are they going to give them all copies of Ruby and Rails?"

Then, of course, I remembered the earthquake. Oh, right, those Java victims!

(Not to make light of the situation there, which sounds grim. And just as I was writing this, I got email from the USGS Earthquake Notification Service reporting another aftershock in Indonesia, this one magnitude 5.6. I hope it doesn't make matters worse.)

Tags: programming
[ 21:05 May 29, 2006    More programming | permalink to this entry ]

MySQL Losing its Socket!

This morning I was all ready to continue working on an ongoing web project when I discovered that mysql wasn't running.

That's funny, it was running fine yesterday! I tried /etc/init.d/mysql start, but it failed. The only error message was, "Please take a look at the syslog."

So I hied myself over to /var/log, to discover that mysql.log and mysql.err were both there, but empty.

Some poking around /etc/mysql/my.cnf revealed that logging is commented out by default, because: "# Be aware that this log type is a performance killer."

I uncommented logging and tried again, but /var/log/mysql.err remained blank, and all that was in mysql.log was three lines related basically giving its runtime arguments and the name of the socket file.

Back to the drawing board. I was well aware that I had changed the mysql settings yesterday. See, mysqld on Ubuntu likes to create its socket as /var/run/mysqld/mysqld.sock, but other apps, like Ruby, all expect to find it in /tmp/mysql.sock. It's easy enough to change Ruby's expectations. But then I found out that although the cmdline client mysql also expects the socket in /var/run/mysqld, it depends on something called mysqladmin that wants the socket in /tmp. (I may have those two reversed. No matter: the point is that you can't use the client to talk to the database because it and the program it depends on disagree about the name of the socket. This is probably a Dapper bug.)

Okay, so I had to pick one. I decided that /tmp/mysql.sock was easier to remember and more standard with non-Debian setups. I knew where to change it in the server (/etc/mysql/my.cnf is there and well commented) but the mysql client doesn't use that, and it took some googling and help from clueful friends to find out that what it wanted was a new file called /etc/my.cnf (how's that for a nice clear configuration file name?) containing one line:

socket          = /tmp/mysql.sock

That done, mysql started and ran and everything worked. Woo!

Except that it didn't the following morning after a reboot, and didn't give any error messages as to why.

Off I went on a merry chase across init files: /etc/init.d/mysql calls /etc/mysql/debian-start (which made me realize that debian has added yet another config file, debian.cnf, which has yet another copy of the line specifying the socket filename) which calls /usr/share/mysql/debian-start.inc.sh as well as calling various other programs. But those were all red herrings: the trick to debugging the problem was to run mysqld directly (not via /etc/init.d/mysql start: it actually does print error messages, but they were being hidden by using the init.d script.

The real problem turned out to be that I had changed the location of the socket file, but not the pid file, in /etc/mysql/my.cnf, which was also located by default in /var/run/mysqld. Apparently that directory is created dynamically at each boot, and it isn't created unless it's needed for the socket file (whether the pid file needs it doesn't matter). So since I'd moved the socket file to /tmp, /var/run/mysqld wasn't created, mysqld couldn't create its pid file and it bailed. Solution: edit my.cnf to use /tmp for the pid file.

Tags: programming
[ 12:29 Mar 17, 2006    More programming | permalink to this entry ]

Ruby on Rails on Ubuntu

Back when I laboriously installed Ruby and Rails on Ubuntu "Hoary Hedgehog" (which involved basically ignoring all the Ubuntu packages and building everything, including Ruby itself, from source), I was cheered by the notes in Ubuntu's forums and bugzilla indicating that as of the next release ("Breezy Badger") all the right versions would be there, and all this source compilation would no longer be necessary.

I didn't get around to trying it until today. Breezy and its successor "Dapper Drake" do indeed have a rails package as well as a Ruby package, and I happily installed them. All looked great -- until I actually tried to use them on a real-world application. It turns out that the Ruby and Rails packages don't include gems, Ruby's package manager (similar to the CPAN system familiar to Perl programmers). And gems is required for doing anything useful in Rails.

Drat! After several false starts, I eventually found the instructions on this page. Except that installs way more than seems necessary for what I need to do, and if you copy/paste lines from that page you may end up with a bunch of packages you don't want, like an out of date version of mysql.

So here are simplified instructions for using Ruby on Rails on Ubuntu Breezy or Dapper.

As yourself:

wget http://rubyforge.org/frs/download.php/5207/rubygems-0.8.11.tgz
tar zxvf rubygems-0.8.11.tgz

As root:

cd rubygems-0.8.11
ruby setup.rb
gem install rubygems-update
gem install rails

Say yes to all dependency questions during the gem install of rails. Add your web server and database of choice (you probably already have them installed, anyway) and you should be good to go.

You may note that the page I referenced tells you to install two versions of rails: the Ubuntu package plus the one from gems. At least on Dapper, you don't need both. Installing rails pulls in the packages:
irb irb1.8 libpgsql-ruby1.8 libreadline-ruby1.8 libredcloth-ruby1.8 libruby1.8 rake rdoc rdoc1.8 ruby ruby1.8
I haven't experimented with which of these packages are and are not needed. If you run into problems, some set of packages from this list may solve them.

Update: it seems that none of these are required. Many of them are dummy packages anyway, which contain no files related to the actual package name. For instance, the rake package contains, not rake, but a single bash completion file related to rake. So you should be fine ignoring all of them, installing just Ruby and nothing else.

(I filed bug 34840 requesting that Ubuntu ship a usable version of Rails, since it didn't seem to be filed already.)

Tags: programming
[ 21:56 Mar 13, 2006    More programming | permalink to this entry ]

Set C Styles Automatically in Emacs

I just figured out a nifty emacs trick: how to set C styles automatically based on the location of the file I'm editing. That way, emacs will autoindent with the proper style whether I'm editing my own code, GIMP code, Linux kernel code, etc. assuming that I keep the source for these projects in predictable places.

First, define a derived mode for each style you might want to use. Each minor mode inherits from c-mode, but adds a c-set-style call. (Obviously, this assumes that the styles you want to use are already defined.)

(define-derived-mode gnu-c-mode c-mode "GNU C mode"
  (c-set-style "gnu"))
(define-derived-mode linux-c-mode c-mode "GNU C mode"
  (c-set-style "linux"))

Then add entries to your auto-mode-alist for each directory you want to treat specially:

(setq auto-mode-alist
[ . . . ]
      (cons '("gimp.*/" . gnu-c-mode)
      (cons '("linux-.*/" . linux-c-mode)
            auto-mode-alist) ) )

Your normal default style will still work for any files not matching the patterns specified in auto-mode-alist.

It might be possible to skip the step of defining derived modes by switching on the file's pathname in the C mode hook, but I've never figured out how to get the pathname of the file being loaded in any reliable way.

Tags: programming
[ 12:42 Sep 30, 2005    More programming | permalink to this entry ]

New Pho

Pho 0.9.5-pre4 seems to be working pretty well and fixes a couple of bugs in pre3, so I posted a tarball. I really need to quit this pre- stuff and just release 0.9.5. Soon, really!

Tags: programming
[ 11:51 Jul 26, 2005    More programming | permalink to this entry ]

A Fast Volume Control App

I updated my Debian sid system yesterday, and discovered today that gnome-volume-control has changed their UI yet again. Now the window comes up with two tabs, Playback and Capture; the default tab, Playback, has only one slider in it, PCM, and all the important sliders, like Volume, are under Capture. (I'm told this is some interaction with how ALSA sees my sound chip.)

That's just silly. I've never liked the app anyway -- it takes forever to come up, so I end up missing too much of any clip that starts out quiet. All I need is a simple, fast window with a single slider controlling master volume. But nothing like that seems to exist, except panel applets that are tied to the panels of particular window managers.

So I wrote one, in PyGTK. vol is a simple script which shows a slider, and calls aumix under the hood to get and set the volume. It's horizontal by default; vol -h gives a vertical slider.

Aside: it's somewhat amazing that Python has no direct way to read an integer out of a string containing more than just that integer: for example, to read 70 out of "70,". I had to write a function to handle that. It's such a terrific no-nonsense language most of the time, yet so bad at a few things. (And when I asked about a general solution in the python channel at [large IRC network], I got a bunch of replies like "use int(str[0:2])" and "use int(str[0:-1])". Shock and bafflement ensued when I pointed out that 5, 100, and -27 are all integers too and wouldn't be handled by those approaches.)

Tags: programming, python, gtk, pygtk
[ 14:54 Jun 21, 2005    More programming | permalink to this entry ]

PyTopo 0.3

I needed to print some maps for one of my geology class field trips, so I added a "save current map" key to PyTopo (which saves to .gif, and then I print it with gimp-print). It calls montage from Image Magick.

Get yer PyTopo 0.3 here.

Tags: programming, python, imagemagick, mapping
[ 16:56 Apr 13, 2005    More programming | permalink to this entry ]

Python Expose vs. Focus

A few days ago, I mentioned my woes regarding Python sending spurious expose events every time the drawing area gains or loses focus.

Since then, I've spoken with several gtk people, and investigated several workarounds, which I'm writing up here for the benefit of anyone else trying to solve this problem.

First, "it's a feature". What's happening is that the default focus in and out handlers for the drawing area (or perhaps its parent class) assume that any widget which gains keyboard focus needs to redraw its entire window (presumably because it's locate-highlighting and therefore changing color everywhere?) to indicate the focus change. Rather than let the widget decide that on its own, the focus handler forces the issue via this expose event. This may be a bad decision, and it doesn't agree with the gtk or pygtk documentation for what an expose event means, but it's been that way for long enough that I'm told it's unlikely to be changed now (people may be depending on the current behavior).

Especially if there are workarounds -- and there are.

I wrote that this happened only in pygtk and not C gtk, but I was wrong. The spurious expose events are only passed if the CAN_FOCUS flag is set. My C gtk test snippet did not need CAN_FOCUS, because the program from which it was taken, pho, already implements the simplest workaround: put the key-press handler on the window, rather than the drawing area. Window apparently does not have the focus/expose misbehavior.

I worry about this approach, though, because if there are any other UI elements in the window which need to respond to key events, they will never get the chance. I'd rather keep the events on the drawing area.

And that becomes possible by overriding the drawing area's default focus in/out handlers. Simply write a no-op handler which returns TRUE, and set it as the handler for both focus-in and focus-out. This is the solution I've taken (and I may change pho to do the same thing, though it's unlikely ever to be a problem in pho).

In C, there's a third workaround: query the default focus handlers, and disconnect() them. That is a little more efficient (you aren't calling your nop routines all the time) but it doesn't seem to be possible from pygtk: pygtk offers disconnect(), but there's no way to locate the default handlers in order to disconnect them.

But there's a fourth workaround which might work even in pygtk: derive a class from drawing area, and set the focus in and out handlers to null. I haven't actually tried this yet, but it may be the best approach for an app big enough that it needs its own UI classes.

One other thing: it was suggested that I should try using AccelGroups for my key bindings, instead of a key-press handler, and then I could even make the bindings user-configurable. Sounded great! AccelGroups turn out to be very easy to use, and a nice feature. But they also turn out to have undocumented limitations on what can and can't be an accelerator. In particular, the arrow keys can't be accelerators; which makes AccelGroup accelerators less than useful for a widget or app that needs to handle user-initiated scrolling or movement. Too bad!

Tags: programming, python, gtk, pygtk
[ 20:52 Apr 09, 2005    More programming | permalink to this entry ]

PyTopo is usable; pygtk is inefficient

While on vacation, I couldn't resist tweaking pytopo so that I could use it to explore some of the areas we were visiting.

It seems fairly usable now. You can scroll around, zoom in and out to change between the two different map series, and get the coordinates of a particular location by clicking. I celebrated by making a page for it, with a silly tux-peering-over-map icon.

One annoyance: it repaints every time it gets a focus in or out, which means, for people like me who use mouse focus, that it repaints twice for each time the mouse moves over the window. This isn't visible, but it would drag the CPU down a bit on a slow machine (which matters since mapping programs are particularly useful on laptops and handhelds).

It turns out this is a pygtk problem: any pygtk drawing area window gets spurious Expose events every time the focus changes (whether or not you've asked to track focus events), and it reports that the whole window needs to be repainted, and doesn't seem to be distinguishable in any way from a real Expose event. The regular gtk libraries (called from C) don't do this, nor do Xlib C programs; only pygtk.

I filed bug 172842 on pygtk; perhaps someone will come up with a workaround, though the couple of pygtk developers I found on #pygtk couldn't think of one (and said I shouldn't worry about it since most people don't use pointer focus ... sigh).

Tags: programming, python, gtk, pygtk, mapping
[ 16:26 Apr 06, 2005    More programming | permalink to this entry ]

Python GTK Topographic Map Program

I couldn't stop myself -- I wrote up a little topo map viewer in PyGTK, so I can move around with arrow keys or by clicking near the edges. It makes it a lot easier to navigate the map directory if I don't know the exact starting coordinates.

It's called PyTopo, and it's in the same place as my earlier two topo scripts.

I think CoordsToFilename has some bugs; the data CD also has some holes, and some directories don't seem to exist in the expected place. I haven't figured that out yet.

Tags: programming, python, gtk, pygtk, mapping
[ 17:53 Mar 27, 2005    More programming | permalink to this entry ]

Topographic Maps for Linux

I've long wished for something like those topographic map packages I keep seeing in stores. The USGS (US Geological Survey) sells digitized versions of their maps, but there's a hefty setup fee for setting up an order, so it's only reasonable when buying large collections all at once.

There are various Linux mapping applications which do things like download squillions of small map sections from online mapping sites, but they're all highly GPS oriented and I haven't had much luck getting them to work without one. I don't (yet?) have a GPS; but even if I had one, I usually want to make maps for places I've been or might go, not for where I am right now. (I don't generally carry a laptop along on hikes!)

The Topo! map/software packages sold in camping/hiking stores (sometimes under the aegis of National Geographic are very reasonably priced. But of course, the software is written for Windows (and maybe also Mac), not much help to Linux users, and the box gives no indication of the format of the data. Googling is no help; it seems no Linux user has ever tried buying one of these packages to see what's inside. The employees at my local outdoor equipment store (Mel Cotton's) were very nice without knowing the answer, and offered the sensible suggestion of calling the phone number on the box, which turns out to be a small local company, "Wildflower Productions", located in San Francisco.

Calling Wildflower, alas, results in an all too familiar runaround: a touchtone menu tree where no path results in the possibility of contact with a human. Sometimes I wonder why companies bother to list a phone number at all, when they obviously have no intention of letting anyone call in.

Concluding that the only way to find out was to buy one, I did so. A worthwhile experiment, as it turned out! The maps inside are simple GIF files, digitized from the USGS 7.5-minute series and, wonder of wonders, also from the discontinued but still useful 15-minute series. Each directory contains GIF files covering the area of one 7.5 minute map, in small .75-minute square pieces, including pieces of the 15-minute map covering the same area.

A few minutes of hacking with python and Image Magick resulted in a script to stitch together all images in one directory to make one full USGS 7.5 minute map; after a few hours of hacking, I can stitch a map of arbitrary size given start and end longitude and latitude. My initial scripts, such as they are.

Of course, I don't yet have nicities like a key, or an interactive scrolling window, or interpretation of the USGS digital elevation data. I expect I have more work to do. But for now, just being able to generate and print maps for a specific area is a huge boon, especially with all the mapping we're doing in Field Geology class. GIMP's "measure" tool will come in handy for measuring distances and angles!

Tags: programming, python, gtk, pygtk
[ 11:13 Mar 27, 2005    More programming | permalink to this entry ]

Kitfox: Firefox for People Who Liked the Mozilla Browser

Debate rages on the mozilla-seamonkey list since the Mozilla Foundation announced that there would be no 1.8 release of the Mozilla browser (also called "the suite", or by its code name, "seamonkey"). Suite users are frustrated at lack of notice: anyone who was paying attention knew that seamonkey was going to be dropped eventually, but everyone expected at least a 1.8 final release. Mozilla.org is frustrated because they wish suite users would quit whining and switch to Firefox. Various people are slinging flames and insults, while a few try to mediate with logic and sense. There's a volunteer effort ramping up to continue support for the suite, but no plans for what to do about fixing all the regressions. Go read the list if you want all the gory details.

Anyway, the writing on the wall (and on the newsgroup) is clear: if you want a browser with continuing support from mozilla.org, Firefox is your only choice. Unfortunately for Linux users, firefox is designed by and for Windows users, copying Internet Explorer's user interface and dropping support for a number of nice features which the old mozilla browser offered.

I've decided that the best way to get a usable browser is to take firefox and put back the mozilla features that I miss. Mostly these are easy user interface tweaks. I pulled a tree last week and had most of the items that were blocking me addressed in a few hours. Building wasn't entirely straightforward: the build page doesn't make all the options clear, like the fact that xft and freetype are both enabled by default, so one of them has to be explicitly disabled. Updating the tree turns out to be a bit problematic: firefox' build dependencies turn out to be dicy, so sometimes changing a single .xul file causes the entire tree to rebuild, while other times an update builds a few files and the resulting build fails to run, and requires a clobber and a rebuild. Still, those problems are relatively minor.

So far, I have fixes for these bugs:

• bug 233853 : Ctrl-enter in the urlbar should open in a new tab, like ctrl-click does on a link. (Firefox uses ctrl-click but alt-enter.) I have no use for the IE ctrl-enter urlbar behavior (add .com or something to whatever is there, so if you're trying to search for linux browser you get "http://linux browser.com" instead of a new tab with a google search for linux browser). Good UI design dictates that the same modifier should be used for the same function, as it was in mozilla.
• bug 245015 : Shift-click should do Save As. Another thing I do many times a day, and Firefox offers no shortcut for it at all.
• Bugs 234110 and 66834: Disable the urlbar dropdown that flickers distractingly as I type, and steals system focus so that I can't do anything outside the firefox window until I dismiss the dropdown.
• Add a binding for ctrl-Q so I can quit the browser easily from the keyboard. (I can't find the relevant bug number.)

Next up: try to figure out why firefox takes so much longer than mozilla to start up. Fortunately, once it's up, it seems just as fast at browsing, but startup takes forever, and firefox doesn't even offer a splash option to tell me that something is happening.

Here is my patch, in case anyone else is bothered by these issues.

Perhaps this could be built as an extension. Some day I'll look into that. Certainly the current set of patches could be implemented as a script which exploded, edited, and re-packed the .jar archives in a firefox binary build, since the patch touches no C++ code as yet.

I'm calling my firefox-derived browser "Kitfox".

Tags: programming
[ 11:03 Mar 16, 2005    More programming | permalink to this entry ]

New Crikey!

Crikey 0.5 is out, with some changes contributed by Efraim Feinstein -- it can read from stdin now, and has a debug flag. Reading from stdin means you can generate multi-line text now.

It's so cool when people send patches to my programs! Especially when they're nice clean code implementing useful features. Thanks, Efraim!

Tags: programming
[ 21:39 Nov 09, 2004    More programming | permalink to this entry ]

Pho 0.9.5-pre3: evil Metacity window sizing

It turns out that the problem with pho windows not resizing in metacity is this: when metacity sees a window that's slightly larger (in either dimension) than the screen size, it unilaterally makes that window maximized, and thereafter refuses any request from the app to resize the window smaller.

Mandatory maximize might actually be useful in some circumstances (anyone who's ever tried to run on an 800x600 laptop has doubtless seen dialogs which don't fit on the screen) but the subsequent refusal to resize makes little sense, and causes bustage in programs which work fine under other window managers.

A workaround is for pho to unmaximize before any window resize. This would be a bummer with an app where the user might click the maximize button manually; with pho, that's unlikely (I hope) because anyone who wants to run maximized is better off running in fullscreen/presentation mode (which now finally sets its background to black, hooray).

Get'cher Pho 0.9.5-pre3 here.

Tags: programming
[ 18:21 Oct 10, 2004    More programming | permalink to this entry ]

New showpix script now uses PHP

I'd been meaning for ages to write a PHP version of my showpix.cgi Perl script, to show images without needing a separate .html file generated for each image. I finally did it this morning, and it was much easier than I expected, and seems to run a lot faster than the perl CGI (not surprising, since PHP is cached in our web server and perl isn't; so this should be more scaleable and less load on the server).

The hardest part was writing the Python script to generate a new showpix.php for a directory of images, and that only because of all the escaping of quotes that needed to be done when telling python to print a line that tells php to print a line to serve up over http ...

Anyway, I've converted the Flume Trail images to use the new PHP stuff, and I've updated the page for the Imagebatch scripts to include PHP ability.

Tags: programming
[ 14:40 Oct 01, 2004    More programming | permalink to this entry ]

Pho 0.9.5-pre2

I finally gathered together some of the pending fixes for Pho, so it keeps better track of where it is in the linked list when deleting or jumping around. No progress on the focus handling, though. I sure would like to solve that before 0.9.5 final.

Tags: programming
[ 21:42 Sep 30, 2004    More programming | permalink to this entry ]

ADA doesn't apply to the web?!

The 11th Circuit Court of Appeals ruled last Friday that the Americans with Disabilities Act does not apply to the web. In other words, web designers are not required to make their pages accessible to readers with disabilities.

That seems very odd. It's required of software -- several lawsuits have been filed (and settled) related to inaccessible software. Accessibility is required universally for web sites in the EU, and I thought it was required of US government sites as well. It's a shame the court felt that it wasn't important for web sites in general.

Accessibility helps everybody, not just people with disabilities (even setting aside the fact that most of us will eventually experience some impairment, if we live long enough). Accessible web sites are usually easier to read and navigate, and translate more easily to offline readers (such as Sitescooper) and PDA readers (such as Opera and Plucker).

The decision was "largely on procedural grounds" and the court suggested that the decision could be revisited in a future case. I hope that case comes soon, before US web designers conclude there's no point in designing for accessibility.

W3C Web Content Accessibility Guidelines 2.0.

Tags: programming
[ 12:05 Sep 28, 2004    More programming | permalink to this entry ]

XShapeCombineMask generates an Expose loop

I tried moonroot on blackbird today, under icewm for the first time. It went into an infinite expose/redraw loop. It turns out that XShapeCombineMask (the call that sets the shaped window's shape mask) generates an extra Expose, which of course happens asynchronously so disabling expose handling in the draw routine doesn't help.

What does help is maintaining a static variable to ensure that it only shapes the window the first time, and not on subsequent draws.

I also tweaked sonypid.c a bit -- 2.4.25 is generating two jogdial-release events whenever the machine resumes from bios suspend. But there's no jogdial-press event corresponding, so I fixed sonypid to ignore jogdial release unless there's already been a jogdial press (again, maintaining a static variable; I already had one so that it doesn't trigger a release after an UP+PRESSED or DOWN+PRESSED event, so I just had to tweak that code a little). That should eliminate that annoying paste that was happening every time I resumed from suspend.

Wish sonypi would quit changing, though I shouldn't complain since it's also good to see that it's still being worked on.

Tags: programming
[ 22:55 Jul 25, 2004    More programming | permalink to this entry ]

Pho Makefile now builds under either gtk1 or gtk2

I fixed the pho Makefile to build under either gtk1 or gtk2, which required some grody bash code. I wish I knew of a better way to do that. Aren't massive API changes fun? Whee!

Patti told me about a park I didn't know about: the Ulistac Natural Area. I drove by it today after Toastmasters. It's a very small linear park along a levee, with one wooded area and a longer and skinnier open area. I didn't go in -- the combination of migraine and hot sun sapped my interest in a walk.

Tags: programming
[ 19:00 Jul 14, 2004    More programming | permalink to this entry ]

Rootmoon

I finally whipped up an app I've been thinking about for a while: a little moon drawn with a shaped window, so you can put it on your root like that OS X moondock applet that Dave uses. Partly it was an excuse to play with shaped windows, which I hadn't used before, and partly it's that I hate seeing stuff that OS X can do that Linux can't.

I put it up (currently temp-named "moonroot" on my software page.

I also caught some nice shots of a hummer at the feeder (through the screen) and added them to my Hummingbird Photo page. A fairly productive day, really, including a nice hike.

Tags: programming
[ 19:00 Jul 11, 2004    More programming | permalink to this entry ]

Pho and window managers

I worked some more on pho yesterday, trying to fix some of the problems Dave has been seeing in other window managers, particularly Metacity. I got rid of most of the problems, but metacity still asks him to place and size the initial window. I have no idea why; I'm specifying initial window size, and no other program asks him that. I also haven't solved the problem of the window getting focus after it resizes out from under the mouse cursor (in lots of window managers, such as openbox). Typing that, I just got a brainstorm: maybe it's a race condition, that the focus loss doesn't happen until after expose, but the "ask for focus" happens before the resize has propogated through the X server. Must try!

Tags: programming
[ 14:00 Jul 08, 2004    More programming | permalink to this entry ]

Rewrote Pho

I finally got around to rewriting pho! The code is much cleaner now -- images are stored as structs in a linked list, no more motley collections of weird global arrays. Code is factored better, and it builds for gtk2. But the main motivation for rewriting it was to have it make a new window for each new or resized image, replacing the old window, to solve a bunch of window manager bugs I keep hitting:

• Some window managers (kwm, xfce4) don't correctly resize a small window to be bigger, or if they do, the contents of the bigger window only repaint over the area that was covered by the old size.
• Some window managers (xfce4, openbox) under pointer focus will lose window focus if a window is resized or moved to a place where it's no longer under the mouse, even if the window had focus before.

I handed the new pho to Dave for testing, and he hates the "new window each time" model; it takes too long in the window managers he runs. He says he wasn't that bothered by the repainting/resizing problems.

Fortunately, the rewrite factored the code so that it should be easy to provide both options (that was the plan anyway), isolated in the NewWindow routine. So I'll put back the "resize and reposition existing window" code, as a switchable option, and maybe try to grab focus to solve the pointer focus issues that have been plaguing me. I don't know what to do about the window manager resize/repaint issues; more research is required.

Tags: programming
[ 22:00 Jul 07, 2004    More programming | permalink to this entry ]