Shallow Thoughts

Python-GTK regression: How to catch mouse button release

We just had the second earthquake in two days, and I was chatting with someone about past earthquakes and wanted to measure the distance to some local landmarks. So I fired up PyTopo as the easiest way to do that. Click on one point, click on a second point and it prints distance and bearing from the first point to the second.

Except it didn't. In fact, clicks weren't working at all. And although I have hacked a bit on parts of pytopo (the most recent project was trying to get scaling working properly in tiles imported from OpenStreetMap), the click handling isn't something I've touched in quite a while.

It turned out that there's a regression in PyGTK: mouse button release events now need you to set an event mask for button presses as well as button releases. You need both, for some reason. So you now need code that looks like this:

drawing_area.connect("button-release-event", button_event)
drawing_area.set_events(gtk.gdk.EXPOSURE_MASK |
                        # next line wasn't needed before:
                        gtk.gdk.BUTTON_PRESS_MASK |
                        gtk.gdk.BUTTON_RELEASE_MASK )

An easy fix ... once you find it.

I filed bug 606453 to see whether the regression was intentional.

I've checked in the fix to the PyTopo svn repository on Google Code. It's so nice having a public source code repository like that! I'm planning to move Pho to Google Code soon.

Tags: programming, python, pygtk, mapping
[ 13:20 Jan 08, 2010    More programming | permalink to this entry ]

Pytopo 0.8 released

On my last Mojave trip, I spent a lot of the evenings hacking on PyTopo.

I was going to try to stick to OpenStreetMap and other existing mapping applications like TangoGPS, a neat little smartphone app for downloading OpenStreetMap tiles that also runs on the desktop -- but really, there still isn't any mapping app that works well enough for exploring maps when you have no net connection.

In particular, uploading my GPS track logs after a day of mapping, I discovered that Tango really wasn't a good way of exploring them, and I already know Merkaartor, nice as it is for entering new OSM data, isn't very good at working offline. There I was, with PyTopo and a boring hotel room; I couldn't stop myself from tweaking a bit.

Adding tracklogs was gratifyingly easy. But other aspects of the code bother me, and when I started looking at what I might need to do to display those Tango/OSM tiles ... well, I've known for a while that some day I'd need to refactor PyTopo's code, and now was the time.

Surprisingly, I completed most of the refactoring on the trip. But even after the refactoring, displaying those OSM tiles turned out to be a lot harder than I'd hoped, because I couldn't find any reliable way of mapping a tile name to the coordinates of that tile. I haven't found any documentation on that anywhere, and Tango and several other programs all do it differently and get slightly different coordinates. That one problem was to occupy my spare time for weeks after I got home, and I still don't have it solved.

But meanwhile, the rest of the refactoring was done, nice features like track logs were working, and I've had to move on to other projects. I am going to finish the OSM tile MapCollection class, but why hold up a release with a lot of useful changes just for that?

So here's PyTopo 0.8, and the couple of known problems with the new features will have to wait for 0.9.

Tags: programming, python, gtk, pygtk, mapping
[ 19:49 Jun 20, 2009    More programming | permalink to this entry ]

Bing thinks we're WHERE?

Lots has been written about Bing, Microsoft's new search engine. It's better than Google, it's worse than Google, it'll never catch up to Google. Farhad Manjoo of Slate had perhaps the best reason to use Bing: "If you switch, Google's going to do some awesome things to try to win you back."

But what I want to know about Bing is this: Why does it think we're in Portugal when Dave runs it under Omniweb on Mac?

In every other browser it gives the screen you've probably seen, with side menus (and a horizontal scrollbar if your window isn't wide enough, ugh) and some sort of pretty picture as a background. In Omniweb, you get a cleaner layout with no sidebars or horizontal scrollbars, a different pretty picture -- often prettier than the one you get on all the other browsers, though both images change daily -- and a set of togglebuttons that don't show up in any of the other browsers, letting you restrict results to only English or only results from Portugal.

Why does it think we're in Portugal when Dave uses Omniweb?

Equally puzzling, why do only people in Portugal have the option of restricting the results to English only?

Tags: tech, browsers, mapping, geolocation
[ 09:37 Jun 10, 2009    More tech | permalink to this entry ]

A GPX file manager

Someone on the OSM newbies list asked how he could strip waypoints out of a GPX track file. Seems he has track logs of an interesting and mostly-unmapped place that he wants to add to openstreetmap, but there are some waypoints that shouldn't be included, and he wanted a good way of separating them out before uploading.

Most of the replies involved "just edit the XML." Sure, GPX files are pretty simple and readable XML -- but a user shouldn't ever have to do that! Gpsman and gpsbabel were also mentioned, but they're not terribly easy to use either.

That reminded me that I had another XML-parsing task I'd been wanting to write in Python: a way to split track files from my Garmin GPS.

Sometimes, after a day of mapping, I end up with several track segments in the same track log file. Maybe I mapped several different trails; maybe I didn't get a chance to upload one day's mapping before going out the next day. Invariably some of the segments are of zero length (I don't know why the Garmin does that, but it always does). Applications like merkaartor don't like this one bit, so I usually end up editing the XML file and splitting it into segments by hand. I'm comfortable with XML -- but it's still silly.

I already have some basic XML parsing as part of PyTopo and Ellie, so I know the parsing very easy to do. So, spurred on by the posting on OSM-newbies, I wrote a little GPX parser/splitter called gpxmgr. gpxmgr -l file.gpx can show you how many track logs are in the file; gpxmgr -w file.gpx can write new files for each non-zero track log. Add -p if you want to be prompted for each filename (otherwise it'll use the name of the track log, which might be something like "ACTIVE\ LOG\ #2").

How, you may wonder, does that help the original poster's need to separate out waypoints from track files? It doesn't. See, my GPS won't save tracklogs and waypoints in the same file, even if you want them that way; you have to use two separate gpsbabel commands to upload a track file and a waypoint file. So I don't actually know what a tracklog-plus-waypoint file looks like. If anyone wants to use gpxmgr to manage waypoints as well as tracks, send me a sample GPX file that combines them both.

Tags: mapping, programming, python
[ 19:43 Jun 01, 2009    More mapping | permalink to this entry ]

Ellie: Plot GPS elevation profiles

Ever since I got the GPS I've been wanting something that plots the elevation data it stores. There are lots of apps that will show me the track I followed in latitude and longitude, but I couldn't find anything that would plot elevations.

But GPX (the XML-based format commonly used to upload track logs) is very straightforward -- you can look at the file and read the elevations right out of it. I knew it wouldn't be hard to write a script to plot them in Python; it just needed a few quiet hours. Sounded like just the ticket for a rainy day stuck at home with a sore throat.

Sure enough, it was fairly easy. I used xml.dom.minidom to parse the file (I'd already had some experience with it in gimplabels for converting gLabels templates), and pylab from matplotlib for doing the plotting. Easy and nice looking.

I even threw in the nice "conditional main" code from Matt Harrison's SCALE7x Python talk, so it should be callable from other Python code.

Here's the page and a screenshot: Ellie: plot elevation from a GPS track.

Tags: programming, mapping, python
[ 16:57 Mar 03, 2009    More programming | permalink to this entry ]

LCA 2009 Tuesday

I missed a lot of the miniconf talks on Tuesday because I wanted to make some last-minute changes to my talk. But I do want to comment on one: Simon Greener's talk on "A Review of Australian Geodata Providers." Of course, I'm not in Australia, but it was quite interesting to hear how similar Australia's problematic geodata siguation is to the situation in the US. His presentation was entertaining, animated and I learned some interesting facts about GPS and geodata in general.

And Dave and I got another good astronomy opportunity with the dark skies at Peppermint Bay at the Speakers' Dinner. Despite occasional intrusive clouds we managed to get a great view of the Large Magellanic Cloud and a decent view of the small one, as well as eta Carinae and the star clouds between Crux and Carina. Pity I'd forgotten to bring my thumpin' travel optics that I'd been using the previous evening: a 6x20 monocular.

Tags: lca2009, linux.conf.au, astronomy, mapping
[ 16:15 Jan 20, 2009    More conferences/lca2009 | permalink to this entry ]

Garmin Vista Cx on Ubuntu "Hardy"

I got myself a GPS unit for Christmas.

I've been resisting the GPS siren song for years -- mostly because I knew it would be a huge time sink involving months of futzing with drivers and software trying to get it to do something useful.

But my experience at an OpenStreetMap mapping party got me fired up about it, and I ordered a Garmin Vista Cx.

Shopping for a handheld GPS is confusing. I was fairly convinced I wanted a Garmin, just because it's the brand used by most people in the open source mapping community so I knew they were likely to work. I wanted one with a barometric altimeter, because I wanted that data from my hikes and bike rides (and besides, it's fun to know how much you've climbed on an outing; I used to have a bike computer with an altimeter and it was a surprisingly good motivator for working harder and getting in better shape).

But Garmin has a bazillion models and I never found any comparison page explaining the differences among the various hiking eTrex models. Eventually I worked it out:

Garmin eTrex models, decoded

C

Color display. This generally also implies USB connectivity instead of serial, just because the color models are newer.

H

High precision (a more sensitive satellite receiver).

x

Takes micro-SD cards. This may not be important for storing tracks and waypoints (you can store quite a long track with the built-in memory) but they mean that you can load extra base maps, like topographic data or other useful features.

Vista, Summit

These models have barometric altimeters and magnetic compasses. (I never did figure out the difference between a Vista and a Summit, except that in the color models (C), Vistas take micro-SD cards (x) while Summits don't, so there's a Summit C and HC while Vistas come in Cx and HCx. I don't know what the difference is between a monochrome Summit and Vista.)

Legend, Venture

These have no altimeter or compass. A Venture is a Legend that comes without the bundled extras like SD card, USB cable and base maps, so it's cheaper.

For me, the price/performance curve pointed to the Vista Cx.

Loading maps

Loading base maps was simplicity itself, and I found lots of howtos on how to use downloadable maps. Just mount the micro-SD card on any computer, make a directory called Garmin, and name the file gmapsupp.img. I used the CloudMade map for California, and it worked great. There are lots of howtos on generating your own maps, too, and I'm looking forward to making some with topographic data (which the CloudMade maps don't have). The most promising howtos I've found so far are the OSM Map On Garmin page on the OSM wiki and the much more difficult, but gorgeous, Hiking Biking Mapswiki page.

Uploading tracks and waypoints

But the real goal was to be able to take this toy out on a hike, then come back and upload the track and waypoint files.

I already knew, from the mapping party, that Garmins have an odd misfeature: you can connect them in usb-storage mode, where they look like an external disk and don't need any special software ... but then you can't upload any waypoints. (In fact, when I tried it with my Vista Cx I didn't even see the track file.) To upload tracks and waypoints, you need to use something that speaks Garmin protocol: namely, the excellent GPSBabel.

So far so good. How do you call GPSbabel? Luckily for me, just before my GPS arrived, Iván Sánchez Ortega posted a useful little gpsbabel script to the OSM newbies list and I thought I was all set.

But once I actually had the Vista in hand, complete with track and waypoints from a walk around the block, it turned out it wasn't quite that simple -- because Ubuntu didn't create the /dev/ttyUSB0 that Iván's script used. A web search found tons of people having that problem on Ubuntu and talking about various workarounds, involving making sure the garmin_usb driver is blacklisted in /etc/modprobe.d/blacklist (it was already), adding a /etc/udev/rules.d/45-garmin.rules file that changes permissions and ownership of ... um, I guess of the file that isn't being created? That didn't make much sense. Anyway, none of it helped.

But finally I found the fix: keep the garmin_usb driver blacklisted use "usb:" as the device to pass to GPSBabel rather than "/dev/ttyUSB0". So the commands are:

gpsbabel -t -i garmin -f usb: -o gpx -F tracks.gpx
gpsbabel -i garmin -f usb: -o gpx -F waypoints.gpx

Like so many other things, it's easy once you know the secret! Viewing tracklogs works great in Merkaartor, though I haven't yet found an app that does anything useful with the elevation data. I may have to write one.

Update: After I wrote this but before I was able to post it, a discussion on the OSM Newbies list with someone who was having similar troubles resulted in this useful wiki page: Garmin on GNU/Linux. It may also be worth checking the Discussion tab on that wiki page for further information.

Second Update: Also, I did have to create a udev file, /etc/udev/rules.d/51-garmin.rules, to set the permissions so that I could access the device without being root. It contains the line:

ATTRS{idVendor}=="091e", ATTRS{idProduct}=="0003", MODE="0660", GROUP="plugdev"

I did that early on then forgot about that step because I thought that the /dev/ttyUSB0 vs. usb: issue was the only one causing the problem.

Tags: gps, mapping, linux, ubuntu, udev
[ 15:31 Jan 04, 2009    More mapping | permalink to this entry ]

OpenStreetMap mapping parties

Latest obsession: mapping with OpenStreetMap.

Last month, OpenStreetMap and its benefactor company CloudMade held a "mapping party" in Palo Alto. I love maps and mapping (I wrote my own little topographic map viewer when I couldn't find one ready-made) and I've been wanting to know more about the state of open source mapping. A mapping party sounded perfect.

The party was a loosely organized affair. We met at a coffeehouse and discussed basics of mapping and openstreetmap. The hosts tried to show us newbies how OSM works, but that was complicated by the coffeehouse's wireless net being down. No big deal -- turns out the point of a mapping party is to hand out GPSes to anyone who doesn't already have one and send us out to do some mapping.

I attached myself to a couple of CloudMade folks who had some experience already and we headed north on a pedestrian path. We spent a couple of hours walking urban trails and marking waypoints. Then we all converged on a tea shop (whose wireless worked a little better than the one at the coffeehouse, but still not very reliably) for lunch and transfer of track and waypoint files.

This part didn't work all that well. It turned out the units we were using (Garmin Legend HCx) can transfer files in two modes, USB mass storage (the easy way, just move files as if from an external disk) or USB Garmin protocol (the hard way: you have to use software like gpsbabel, or the Garmin software if you're on Windows). And in mass storage mode, you get a file but the waypoints aren't there.

The folks running the event all had Macs, and there were several Linux users there as well, but no Windows laptops. By the time the Macs both had gpsbabel downloaded over the tea shop's flaky net, it was past time for me to leave, so I never did get to see our waypoint files. Still, I could see it was possible (and one of the Linux attendees assured me that he had no trouble with any of the software; in fact, he found it easier than what the Mac people at the party were going through).

But I was still pretty jazzed about how easy OpenStreetMap is to use. You can contribute to the maps even without a GPS. Once you've registered on the site, you just click on the Edit tab on any map, and you see a flash application called "Potlatch" that lets you mark trails, roads or other features based on satellite images or the existing map. I was able to change a couple of mismarked roads near where I live, as well as adding a new trail and correcting the info on an existing one for one of the nearby parks.

If you prefer (as, I admit, I do) to work offline or don't like flash, you can use a Java app, JOSM, or a native app, merkaartor. Very cool! Merkaartor is my favorite so far (because it's faster and works better in standalone mode) though it's still fairly rough around the edges. They're all described on the OSM Map Editing page.

Of course, all this left me lusting after a GPS. But that's another story, to be told separately.

Tags: gps, mapping, open source
[ 12:00 Jan 03, 2009    More mapping | permalink to this entry ]

PyTopo 0.5

Belated release announcement: 0.5b2 of my little map viewer PyTopo has been working well, so I released 0.5 last week with only a few minor changes from the beta. I'm sure I'll immediately find six major bugs -- but hey, that's what point releases are for. I only did betas this time because of the changed configuration file format.

I also made a start on a documentation page for the .pytopo file (though it doesn't really have much that wasn't already written in comments inside the script).

Tags: programming, python, gtk, pygtk, mapping
[ 21:10 Aug 25, 2006    More programming | permalink to this entry ]

Cleaner, More Flexible Python Map Viewing

A few months ago, someone contacted me who was trying to use my PyTopo map display script for a different set of map data, the Topo! National Parks series. We exchanged some email about the format the maps used.

I'd been wanting to make PyTopo more general anyway, and already had some hacky code in my local version to let it use a local geologic map that I'd chopped into segments. So, faced with an Actual User (always a good incentive!), I took the opportunity to clean up the code, use some of Python's support for classes, and introduce several classes of map data.

I called it 0.5 beta 1 since it wasn't well tested. But in the last few days, I had occasion to do some map exploring, cleaned up a few remaining bugs, and implemented a feature which I hadn't gotten around to implementing in the new framework (saving maps to a file).

I think it's ready to use now. I'm going to do some more testing: after visiting the USGS Open House today and watching Jim Lienkaemper's narrated Virtual Tour of the Hayward Fault, I'm all fired up about trying again to find more online geologic map data. But meanwhile, PyTopo is feature complete and has the known bugs fixed. The latest version is on the PyTopo page.

Tags: programming, python, gtk, pygtk, mapping
[ 17:25 Jun 03, 2006    More programming | permalink to this entry ]

PyTopo 0.3

I needed to print some maps for one of my geology class field trips, so I added a "save current map" key to PyTopo (which saves to .gif, and then I print it with gimp-print). It calls montage from Image Magick.

Get yer PyTopo 0.3 here.

Tags: programming, python, imagemagick, mapping
[ 16:56 Apr 13, 2005    More programming | permalink to this entry ]

PyTopo is usable; pygtk is inefficient

While on vacation, I couldn't resist tweaking pytopo so that I could use it to explore some of the areas we were visiting.

It seems fairly usable now. You can scroll around, zoom in and out to change between the two different map series, and get the coordinates of a particular location by clicking. I celebrated by making a page for it, with a silly tux-peering-over-map icon.

One annoyance: it repaints every time it gets a focus in or out, which means, for people like me who use mouse focus, that it repaints twice for each time the mouse moves over the window. This isn't visible, but it would drag the CPU down a bit on a slow machine (which matters since mapping programs are particularly useful on laptops and handhelds).

It turns out this is a pygtk problem: any pygtk drawing area window gets spurious Expose events every time the focus changes (whether or not you've asked to track focus events), and it reports that the whole window needs to be repainted, and doesn't seem to be distinguishable in any way from a real Expose event. The regular gtk libraries (called from C) don't do this, nor do Xlib C programs; only pygtk.

I filed bug 172842 on pygtk; perhaps someone will come up with a workaround, though the couple of pygtk developers I found on #pygtk couldn't think of one (and said I shouldn't worry about it since most people don't use pointer focus ... sigh).

Tags: programming, python, gtk, pygtk, mapping
[ 16:26 Apr 06, 2005    More programming | permalink to this entry ]

Python GTK Topographic Map Program

I couldn't stop myself -- I wrote up a little topo map viewer in PyGTK, so I can move around with arrow keys or by clicking near the edges. It makes it a lot easier to navigate the map directory if I don't know the exact starting coordinates.

It's called PyTopo, and it's in the same place as my earlier two topo scripts.

I think CoordsToFilename has some bugs; the data CD also has some holes, and some directories don't seem to exist in the expected place. I haven't figured that out yet.

Tags: programming, python, gtk, pygtk, mapping
[ 17:53 Mar 27, 2005    More programming | permalink to this entry ]