Shallow Thoughts : tags : mapping

Akkana's Musings on Open Source Computing and Technology, Science, and Nature.

Thu, 06 Apr 2017

Clicking through a translucent window: using X11 input shapes

It happened again: someone sent me a JPEG file with an image of a topo map, with a hiking trail and interesting stopping points drawn on it. Better than nothing. But what I really want on a hike is GPX waypoints that I can load into OsmAnd, so I can see whether I'm still on the trail and how to get to each point from where I am now.

My PyTopo program lets you view the coordinates of any point, so you can make a waypoint from that. But for adding lots of waypoints, that's too much work, so I added an "Add Waypoint" context menu item -- that was easy, took maybe twenty minutes. PyTopo already had the ability to save its existing tracks and waypoints as a GPX file, so no problem there.

[transparent image viewer overlayed on top of topo map] But how do you locate the waypoints you want? You can do it the hard way: show the JPEG in one window, PyTopo in the other, and do the "let's see the road bends left then right, and the point is off to the northwest just above the right bend and about two and a half times as far away as the distance through both road bends". Ugh. It takes forever and it's terribly inaccurate.

More than once, I've wished for a way to put up a translucent image overlay that would let me click through it. So I could see the image, line it up with the map in PyTopo (resizing as needed), then click exactly where I wanted waypoints.

I needed two features beyond what normal image viewers offer: translucency, and the ability to pass mouse clicks through to the window underneath.

A translucent image viewer, in Python

The first part, translucency, turned out to be trivial. In a class inheriting from my Python ImageViewerWindow, I just needed to add this line to the constructor:


Plus one more step. The window was translucent now, but it didn't look translucent, because I'm running a simple window manager (Openbox) that doesn't have a compositor built in. Turns out you can run a compositor on top of Openbox. There are lots of compositors; the first one I found, which worked fine, was xcompmgr -c -t-6 -l-6 -o.1

The -c specifies client-side compositing. -t and -l specify top and left offsets for window shadows (negative so they go on the bottom right). -o.1 sets the opacity of window shadows. In the long run, -o0 is probably best (no shadows at all) since the shadow interferes a bit with seeing the window under the translucent one. But having a subtle .1 shadow was useful while I was debugging.

That's all I needed: voilà, translucent windows. Now on to the (much) harder part.

A click-through window, in C

X11 has something called the SHAPE extension, which I experimented with once before to make a silly program called moonroot. It's also used for the familiar "xeyes" program. It's used to make windows that aren't square, by passing a shape mask telling X what shape you want your window to be. In theory, I knew I could do something like make a mask where every other pixel was transparent, which would simulate a translucent image, and I'd at least be able to pass clicks through on half the pixels.

But fortunately, first I asked the estimable Openbox guru Mikael Magnusson, who tipped me off that the SHAPE extension also allows for an "input shape" that does exactly what I wanted: lets you catch events on only part of the window and pass them through on the rest, regardless of which parts of the window are visible.

Knowing that was great. Making it work was another matter. Input shapes turn out to be something hardly anyone uses, and there's very little documentation.

In both C and Python, I struggled with drawing onto a pixmap and using it to set the input shape. Finally I realized that there's a call to set the input shape from an X region. It's much easier to build a region out of rectangles than to draw onto a pixmap.

I got a C demo working first. The essence of it was this:

    if (!XShapeQueryExtension(dpy, &shape_event_base, &shape_error_base)) {
        printf("No SHAPE extension\n");

    /* Make a shaped window, a rectangle smaller than the total
     * size of the window. The rest will be transparent.
    region = CreateRegion(outerBound, outerBound,
                          XWinSize-outerBound*2, YWinSize-outerBound*2);
    XShapeCombineRegion(dpy, win, ShapeBounding, 0, 0, region, ShapeSet);

    /* Make a frame region.
     * So in the outer frame, we get input, but inside it, it passes through.
    region = CreateFrameRegion(innerBound);
    XShapeCombineRegion(dpy, win, ShapeInput, 0, 0, region, ShapeSet);

CreateRegion sets up rectangle boundaries, then creates a region from those boundaries:

Region CreateRegion(int x, int y, int w, int h) {
    Region region = XCreateRegion();
    XRectangle rectangle;
    rectangle.x = x;
    rectangle.y = y;
    rectangle.width = w;
    rectangle.height = h;
    XUnionRectWithRegion(&rectangle, region, region);

    return region;

CreateFrameRegion() is similar but a little longer. Rather than post it all here, I've created a GIST: transregion.c, demonstrating X11 shaped input.

Next problem: once I had shaped input working, I could no longer move or resize the window, because the window manager passed events through the window's titlebar and decorations as well as through the rest of the window. That's why you'll see that CreateFrameRegion call in the gist: -- I had a theory that if I omitted the outer part of the window from the input shape, and handled input normally around the outside, maybe that would extend to the window manager decorations. But the problem turned out to be a minor Openbox bug, which Mikael quickly tracked down (in openbox/frame.c, in the XShapeCombineRectangles call on line 321, change ShapeBounding to kind). Openbox developers are the greatest!

Input Shapes in Python

Okay, now I had a proof of concept: X input shapes definitely can work, at least in C. How about in Python?

There's a set of python-xlib bindings, and they even supports the SHAPE extension, but they have no documentation and didn't seem to include input shapes. I filed a GitHub issue and traded a few notes with the maintainer of the project. It turned out the newest version of python-xlib had been completely rewritten, and supposedly does support input shapes. But the API is completely different from the C API, and after wasting about half a day tweaking the demo program trying to reverse engineer it, I gave up.

Fortunately, it turns out there's a much easier way. Python-gtk has shape support, even including input shapes. And if you use regions instead of pixmaps, it's this simple:

    if self.is_composited():
        region = gtk.gdk.region_rectangle(gtk.gdk.Rectangle(0, 0, 1, 1))
        self.window.input_shape_combine_region(region, 0, 0)

My came out nice and simple, inheriting from and adding only translucency and the input shape.

If you want to define an input shape based on pixmaps instead of regions, it's a bit harder and you need to use the Cairo drawing API. I never got as far as working code, but I believe it should go something like this:

    # Warning: untested code!
    bitmap = gtk.gdk.Pixmap(None, self.width, self.height, 1)
    cr = bitmap.cairo_create()
    # Draw a white circle in a black rect:
    cr.rectangle(0, 0, self.width, self.height)

    # draw white filled circle
    cr.arc(self.width / 2, self.height / 2, self.width / 4,
           0, 2 * math.pi);

    self.window.input_shape_combine_mask(bitmap, 0, 0)

The translucent image viewer worked just as I'd hoped. I was able to take a JPG of a trailmap, overlay it on top of a PyTopo window, scale the JPG using the normal Openbox window manager handles, then right-click on top of trail markers to set waypoints. When I was done, a "Save as GPX" in PyTopo and I had a file ready to take with me on my phone.

Tags: , , ,
[ 17:08 Apr 06, 2017    More programming | permalink to this entry | comments ]

Fri, 26 Aug 2016

More map file conversions: ESRI Shapefiles and GeoJSON

I recently wrote about Translating track files between mapping formats like GPX, KML, KMZ and UTM But there's one common mapping format that keeps coming up that's hard to handle using free software, and tricky to translate to other formats: ESRI shapefiles.

ArcGIS shapefiles are crazy. Typically they come as an archive that includes many different files, with the same base name but different extensions: filename.sbn, filename.shx, filename.cpg, filename.sbx, filename.dbf, filename.shp, filename.prj, and so forth. Which of these are important and which aren't?

To be honest, I don't know. I found this description in my searches: "A shape file map consists of the geometry (.shp), the spatial index (.shx), the attribute table (.dbf) and the projection metadata file (.prj)." Poking around, I found that most of the interesting metadata (trail name, description, type, access restrictions and so on) was in the .dbf file.

You can convert the whole mess into other formats using the ogr2ogr program. On Debian it's part of the gdal-bin package. Pass it the .shp filename, and it will look in the same directory for files with the same basename and other shapefile-related extensions. For instance, to convert to KML:

 ogr2ogr -f KML output.kml input.shp

Unfortunately, most of the metadata -- comments on trail conditions and access restrictions that were in the .dbf file -- didn't make it into the KML.

GPX was even worse. ogr2ogr knows how to convert directly to GPX, but that printed a lot of errors like "Field of name 'foo' is not supported in GPX schema. Use GPX_USE_EXTENSIONS creation option to allow use of the <extensions> element." So I tried ogr2ogr -f "GPX" -dsco GPX_USE_EXTENSIONS=YES output.gpx input.shp but that just led to more errors. It did produce a GPX file, but it had almost no useful data in it, far less than the KML did. I got a better GPX file by using ogr2ogr to convert to KML, then using gpsbabel to convert that KML to GPX.

Use GeoJSON instead to preserve the metadata

But there is a better way: GeoJSON.

ogr2ogr -f "GeoJSON" -t_srs crs:84 output.geojson input.shp

That preserved most, maybe all, of the metadata the .dbf file and gave me a nicely formatted file. The only problem was that I didn't have any programs that could read GeoJSON ...

[PyTopo showing metadata from GeoJSON converted from a shapefile]

But JSON is a nice straightforward format, easy to read and easy to parse, and it took surprisingly little work to add GeoJSON parsing to PyTopo. Now, at least, I have a way to view the maps converted from shapefiles, click on a trail and see the metadata from the original shapefile.

See also:

Tags: , , ,
[ 12:11 Aug 26, 2016    More mapping | permalink to this entry | comments ]

Wed, 17 Aug 2016

Making New Map Tracks with Google Earth

A few days ago I wrote about track files in maps, specifically Translating track files between mapping formats. I promised to follow up with information on how to create new tracks.

For instance, I have some scans of old maps from the 60s and 70s showing the trails in the local neighborhood. There's no newer version. (In many cases, the trails have disappeared from lack of use -- no one knows where they're supposed to be even though they're legally trails where you're allowed to walk.) I wanted a way to turn trails from the old map into GPX tracks.

My first thought was to trace the old PDF map. A lot of web searching found a grand total of one page that talks about that: How to convert image of map into vector format?. It involves using GIMP to make an image containing just black lines on a white background, saving as uncompressed TIFF, then using a series of commands in GRASS. I made a start on that, but it was looking like it might be a big job that way. Since a lot of the old trails are still visible as faint traces in satellite photos, I decided to investigate tracing satellite photos in a map editor first, before trying the GRASS method.

But finding a working open source map editor turns out to be basically impossible. (Opportunity alert: it actually wouldn't be that hard to add that to PyTopo. Some day I'll try that, but now I was trying to solve a problem and hoping not to get sidetracked.)

The only open source map editor I've found is called Viking, and it's terrible. The user interface is complicated and poorly documented, and I could input only two or three trail segments before it crashed and I had to restart. Saving often, I did build up part of the trail network that way, but it was so slow and tedious restoring between crashes that I gave up.

OpenStreetMap has several editors available, and some of them are quite good, but they're (quite understandably) oriented toward defining roads that you're going to upload to the OpenStreetMap world map. I do that for real trails that I've walked myself, but it doesn't seem appropriate for historical paths between houses, some of which are now fenced off and few of which I've actually tried walking yet.

Editing a track in Google Earth

In the end, the only reasonable map editor I found was Google Earth -- free as in beer, not speech. It's actually quite a good track editor once I figured out how to use it -- the documentation is sketchy and no one who writes about it tells you the important parts, which were, for me:

Click on "My Places" in the sidebar before starting, assuming you'll want to keep these tracks around.

Right-click on My Places and choose Add->Folder if you're going to be creating more than one path. That way you can have a single KML file (Google Earth creates KML/KMZ, not GPX) with all your tracks together.

Move and zoom the map to where you can see the starting point for your path.

Click the "Add Path" button in the toolbar. This brings up a dialog where you can name the path and choose a color that will stand out against the map. Do not hit Return after typing the name -- that will immediately dismiss the dialog and take you out of path editing mode, leaving you with an empty named object in your sidebar. If you forget, like I kept doing, you'll have to right-click it and choose Properties to get back into editing mode.

Iconify, shade or do whatever your window manager allows to get that large, intrusive dialog out of the way of the map you're trying to edit. Shade worked well for me in Openbox.

Click on the starting point for your path. If you forgot to move the map so that this point is visible, you're out of luck: there's no way I've found to move the map at this point. (You might expect something like dragging with the middle mouse button, but you'd be wrong.) Do not in any circumstances be tempted to drag with the left button to move the map: this will draw lots of path points.

If you added points you don't want -- for instance, if you dragged on the map trying to move it -- Ctrl-Z doesn't undo, and there's no Undo in the menus, but Delete removes previous points. Whew.

Once you've started adding points, you can move the map using the arrow keys on your keyboard. And you can always zoom with the mousewheel.

When you finish one path, click OK in its properties dialog to end it.

Save periodically: click on the folder you created in My Places and choose Save Place As... Google Earth is a lot less crashy than Viking, but I have seen crashes.

When you're done for the day, be sure to File->Save->Save My Places. Google Earth apparently doesn't do this automatically; I was forever being confused why it didn't remember things I had done, and why every time I started it it would give me syntax errors on My Places saying it was about to correct the problem, then the next time I'd get the exact same error. Save My Places finally fixed that, so I guess it's something we're expected to do now and then in Google Earth.

Once I'd learned those tricks, the map-making went fairly quickly. I had intended only to trace a few trails then stop for the night, but when I realized I was more than halfway through I decided to push through, and ended up with a nice set of KML tracks which I converted to GPX and loaded onto my phone. Now I'm ready to explore.

[ 17:26 Aug 17, 2016    More mapping | permalink to this entry | comments ]

Sun, 14 Aug 2016

Translating track files between mapping formats

I use map tracks quite a bit. On my Android phone, I use OsmAnd, an excellent open-source mapping tool that can download map data generated from free OpenStreetMap, then display the maps offline, so I can use them in places where there's no cellphone signal (like nearly any hiking trail). At my computer, I never found a decent open-source mapping program, so I wrote my own, PyTopo, which downloads tiles from OpenStreetMap.

In OsmAnd, I record tracks from all my hikes, upload the GPX files, and view them in PyTopo. But it's nice to go the other way, too, and take tracks or waypoints from other people or from the web and view them in my own mapping programs, or use them to find them when hiking.

Translating between KML, KMZ and GPX

Both OsmAnd and PyTopo can show Garmin track files in the GPX format. PyTopo can also show KML and KMZ files, Google's more complicated mapping format, but OsmAnd can't. A lot of track files are distributed in Google formats, and I find I have to translate them fairly often -- for instance, lists of trails or lists of waypoints on a new hike I plan to do may be distributed as KML or KMZ.

The command-line gpsbabel program does a fine job translating KML to GPX. But I find its syntax hard to remember, so I wrote a shell alias:

kml2gpx () {
        gpsbabel -i kml -f $1 -o gpx -F $1:t:r.gpx
so I can just type kml2gpx file.kml and it will create a file.gpx for me.

More often, people distribute KMZ files, because they're smaller. They're just gzipped KML files, so the shell alias is only a little bit longer:

kmz2gpx () {
        gunzip -c $1 > $kmlfile
        gpsbabel -i kml -f $kmlfile -o gpx -F $kmlfile:t:r.gpx

Of course, if you ever have a need to go from GPX to KML, you can reverse the gpsbabel arguments appropriately; and if you need KMZ, run gzip afterward.

UTM coordinates

A couple of people I know use a different format, called UTM, which stands for Universal Transverse Mercator, for waypoints, and there are some secret lists of interesting local features passed around in that format.

It's a strange system. Instead of using latitude and longitude like most world mapping coordinate systems, UTM breaks the world into 60 longitudinal zones. UTM coordinates don't usually specify their zone (at least, none of the ones I've been given ever have), so if someone gives you a UTM coordinate, you need to know what zone you're in before you can translate it to a latitude and longitude. Then a pair of UTM coordinates specifies easting, and northing which tells you where you are inside the zone. Wikipedia has a map of UTM zones.

Note that UTM isn't a file format: it's just a way of specifying two (really three, if you count the zone) coordinates. So if you're given a list of UTM coordinate pairs, gpsbabel doesn't have a ready-made way to translate them into a GPX file. Fortunately, it allows a "universal CSV" (comma separated values) format, where the first line specifies which field goes where. So you can define a UTM UniCSV format that looks like this:

Trailhead,13,0395145,3966291,Trailhead on Buckman Rd
Sierra Club TH,13,0396210,3966597,Alternate trailhead in the arroyo
then translate it like this:
gpsbabel -i unicsv -f filename.csv -o gpx -F filename.gpx
I (and all the UTM coordinates I've had to deal with) are in zone 13, so that's what I used for that example and I hardwired that into my alias, but if you're near a zone boundary, you'll need to figure out which zone to use for each coordinate.

I also know someone who tends to send me single UTM coordinate pairs, because that's what she has her Garmin configured to show her. For instance, "We'll be using the trailhead at 0395145 3966291". This happened often enough, and I got tired of looking up the UTM UniCSV format every time, that I made another shell function just for that.

utm2gpx () {
        unicsv=`mktemp /tmp/point-XXXXX.csv` 
        echo "name,utm_z,utm_e,utm_n,comment" >> $unicsv
        printf "Point,13,%s,%s,point" $1 $2 >> $unicsv
        gpsbabel -i unicsv -f $unicsv -o gpx -F $gpxfile
        echo Created $gpxfile
So I can say utm2gpx 0395145 3966291, pasting the two coordinates from her email, and get a nice GPX file that I can push to my phone.

What if all you have is a printed map, or a scan of an old map from the pre-digital days? That's part 2, which I'll post in a few days.

[ 10:29 Aug 14, 2016    More mapping | permalink to this entry | comments ]

Thu, 09 Jul 2015

Taming annoyances in the new Google Maps

For a year or so, I've been appending "output=classic" to any Google Maps URL. But Google disabled Classic mode last month. (There have been a few other ways to get classic Google maps back, but Google is gradually disabling them one by one.)

I have basically three problems with the new maps:

  1. If you search for something, the screen is taken up by a huge box showing you what you searched for; if you click the "x" to dismiss the huge box so you can see the map underneath, the box disappears but so does the pin showing your search target.
  2. A big swath at the bottom of the screen is taken up by a filmstrip of photos from the location, and it's an extra click to dismiss that.
  3. Moving or zooming the map is very, very slow: it relies on OpenGL support in the browser, which doesn't work well on Linux in general, or on a lot of graphics cards on any platform.

Now that I don't have the "classic" option any more, I've had to find ways around the problems -- either that, or switch to Bing maps. Here's how to make the maps usable in Firefox.

First, for the slowness: the cure is to disable webgl in Firefox. Go to about:config and search for webgl. Then doubleclick on the line for webgl.disabled to make it true.

For the other two, you can add userContent lines to tell Firefox to hide those boxes.

Locate your Firefox profile. Inside it, edit chrome/userContent.css (create that file if it doesn't already exist), and add the following two lines:

div#cards { display: none !important; }
div#viewcard { display: none !important; }

Voilà! The boxes that used to hide the map are now invisible. Of course, that also means you can't use anything inside them; but I never found them useful for anything anyway.

Tags: , , ,
[ 10:54 Jul 09, 2015    More tech/web | permalink to this entry | comments ]

Tue, 20 Aug 2013

Using Google Maps with Python to turn a list of addresses into waypoints

A few days ago I tlaked about how I use making waypoint files for a list of house addresses is OsmAnd. For waypoint files, you need latitude/longitude coordinates, and I was getting those from a web page that used the online Google Maps API to convert an address into latitude and longitude coordinates.

It was pretty cool at first, but pasting every address into the latitude/longitude web page and then pasting the resulting coordinates into the address file, got old, fast. That's exactly the sort of repetitive task that computers are supposed to handle for us.

The lat/lon page used Javascript and the Google Maps API. and I already had a Google Maps API key (they have all sorts of fun APIs for map geeks) ... but I really wanted something that could run locally, reading and converting a local file.

And then I discovered the Python googlemaps package. Exactly what I needed! It's in the Python Package Index, so I installed it with pip install googlemaps. That enabled me to change my waymaker Python script: if the first line of a description wasn't a latitude and longitude, instead it looked for something that might be an address.

Addresses in my data files might be one line or might be two, but since they're all US addresses, I know they'll end with a two-capital-letter state abbreviation and a 5-digit zip code: 2948 W Main St. Anytown, NM 12345. You can find that with a regular expression:

    match ='.*[A-Z]{2}\s+\d{5}$', line)

But first I needed to check whether the first line of the entry was already latitude/longitude coordinates, since I'd already converted some of my files. That uses another regular expression. Python doesn't seem to have a built-in way to search for generic numeric expressions (containing digits, decimal points or +/- symbols) so I made one, since I had to use it twice if I was searching for two numbers with whitespace between them.

    numeric = '[\+\-\d\.]'
    match ='^(%s+)\s+(%s+)$' % (numeric, numeric),
(For anyone who wants to quibble, I know the regular expression isn't perfect. For instance, it would match expressions like 23+48..6.1-64.5. Not likely to be a problem in these files, so I didn't tune it further.)

If the script doesn't find coordinates but does find something that looks like an address, it feeds the address into Google Maps and gets the resulting coordinates. That code looks like this:

from googlemaps import GoogleMaps

    lat, lon = gmaps.address_to_latlng(addr)
except googlemaps.GoogleMapsError, e:
    print "Oh, no! Couldn't geocode", addr
    print e

Overall, a nice simple solution made possible with python-googlemaps. The full script is on github: waymaker.

Tags: , , , , , ,
[ 12:24 Aug 20, 2013    More mapping | permalink to this entry | comments ]

Fri, 16 Aug 2013

Offline mapping with lists of waypoints

Dave and I have been doing some exploratory househunting trips, and one of the challenges is how to maintain a list of houses and navigate from location to location. It's basically like geocaching, navigating from one known location to the next.

Sure, there are smartphone apps to do things like "show houses for sale near here" against a Google Maps background. But we didn't want everything, just the few gems we'd picked out ahead of time. And some of the places we're looking are fairly remote -- you can't always count on a consistent signal everywhere as you drive around, let alone a connection fast enough to download map tiles.

Fortunately, I use a wonderful open-source Android program called OsmAnd. It's the best, bar none, at offline mapping: download data files prepared from OpenStreetMap vector data, and you're good to go, even into remote areas with no network connectivity. It's saved our butts more than once exploring remote dirt tracks in the Mojave. And since the maps come from OpenStreetMap, if you find anything wrong with the map, you can fix it.

So the map part is taken care of. What about that list of houses?

Making waypoint files

On the other hand, one of OsmAnd's many cool features is that it can show track logs. I can upload a GPX file from my Garmin, or record a track within OsmAnd, and display the track on OsmAnd's map.

GPX track files can include waypoints. What if I made a GPX file consisting only of waypoints and descriptions for each house?

My husband was already making text files of potentially interesting houses:

404 E David Dr 
Flagstaff, AZ 86001
3 Bed 2 Bath
1,673 Sq Ft
0.23 acres

2948 W Wilson Dr 
Flagstaff, AZ 86001
3 Bed 2 Bath
1,908 Sq Ft
8,000 Sq Ft Lot 

... (and so on)
So I just needed to turn those into GPX.

GPX is a fairly straightforward XML format -- I've parsed GPX files for pytopo and for ellie, and generating them from Python should be easier than parsing. But first I needed latitude and longitude coordinates. A quick web search solved that: an excellent page called Find latitude and longitude with Google Maps. You paste the address in and it shows you the location on a map along with latitude and longitude. Thanks to Bernard Vatant at Mondeca!

For each house, I copied the coordinates directly from the page and pasted them into the file. (Though that got old after about the fifth house; I'll write about automating that step in a separate article.)

Then I wrote a script called waymaker that parses a file of coordinates and descriptions and makes waypoint files. Run it like this: waymaker infile.txt outfile.gpx and it will create (or overwrite) a gpx file consisting of those waypoints.

Getting it into OsmAnd

I plugged my Android device into my computer's USB port, mounted it as usb-storage and copied all the GPX files into osmand/tracks (I had to create the tracks subdirectory myself, since I hadn't recorded any tracks. After restarting OsmAnd, it was able to see all the waypoint files.

OsmAnd has a couple of other ways of showing points besides track files. "Favorites" lets you mark a point on the map and save it to various Favorites categories. But although there's a file named favorites.gpx, changes you make to it never show up in the program. Apparently they're cached somewhere else. "POI" (short for Points of Interest) can be uploaded, but only as a .obf OsmAnd file or a .sqlitedb database, and there isn't much documentation on how to create either one. GPX tracks seemed like the easiest solution, and I've been happy with them so far.

Update: I asked on the osmand mailing list; it turns out that on the Favorites screen (Define View, then Favorites) there's a Refresh button that makes osmand re-read favorites.gpx. Works great. It uses pretty much the same format as track files -- I took <wpt></wpt> sequences I'd generated with waymaker and added them to my existing favorites.gpx file, adding appropriate categories. It's nice to have two different ways to display and categorize waypoints within the app.

Using waypoints in OsmAnd

How do you view these waypoints once they're loaded? When you're in OsmAnd's map view, tap the menu button and choose Define View, then GPX track... You'll see a list of all your GPX files; choose the one you want.

You'll be taken back to the map view, at a location and zoom level that shows all your waypoints. Don't panic if you don't see them immediately; sometimes I needed to scroll and zoom around a little before OsmAnd noticed there were waypoints and started drawing them.

Then you can navigate in the usual way. When you get to a waypoint, tap on it to see the description brieftly -- I was happy to find that multiple line descriptions work just fine. Or long-press on it to pop up a persistent description window that will stay up until you dismiss it.

It worked beautifully for our trip, both for houses and for other things like motels and points of interest along the way.

Tags: , , ,
[ 15:58 Aug 16, 2013    More mapping | permalink to this entry | comments ]

Wed, 16 Nov 2011

New trails, and new PyTopo 1.1 release

A new trail opened up above Alum Rock park! Actually a whole new open space preserve, called Sierra Vista -- with an extensive set of trails that go all sorts of interesting places.

Dave and I visit Alum Rock frequently -- we were married there -- so having so much new trail mileage is exciting. We tried to explore it on foot, but quickly realized the mileage was more suited to mountain bikes. Even with bikes, we'll be exploring this area for a while (mostly due to not having biked in far too long, so it'll take us a while to work up to that much riding ... a combination of health problems and family issues have conspired to keep us off the bikes).

Of course, part of the fun of discovering a new trail system is poring over maps trying to figure out where the trails will take us, then taking GPS track logs to study later to see where we actually went.

And as usual when uploading GPS track logs and viewing them in pytopo, I found some things that weren't working quite the way I wanted, so the session ended up being less about studying maps and more about hacking Python.

In the end, I fixed quite a few little bugs, improved some features, and got saved sites with saved zoom levels working far better.

Now, PyTopo 1.0 happened quite a while ago -- but there were two of us hacking madly on it at the time, and pinning down the exact time when it should be called 1.0 wasn't easy. In fact, we never actually did it. I know that sounds silly -- of all releases to not get around to, finally reaching 1.0? Nevertheless, that's what happened.

I thought about cheating and calling this one 1.0, but we've had 1.0 beta RPMs floating around for so long (and for a much earlier release) that that didn't seem right.

So I've called the new release PyTopo 1.1. It seems to be working pretty solidly. It's certainly been very helpful to me in exploring the new trails. It's great for cross-checking with Google Earth: the OpenCycleMap database has much better trail data than Google does, and pytopo has easy track log loading and will work offline, while Google has the 3-D projection aerial imagery that shows where trails and roads were historically (which may or may not correspond to where they decide to put the new trails). It's great to have both.

Anyway, here's the new PyTopo.

Tags: , ,
[ 20:59 Nov 16, 2011    More mapping | permalink to this entry | comments ]

Sun, 24 Apr 2011

WhereCamp 2011

I spent Friday and Saturday at the WhereCamp unconference on mapping, geolocation and related topics.

This was my second year at WhereCamp. It's always a bit humbling. I feel like I'm pretty geeky, and I've written a couple of Python mapping apps and I know spherical geometry and stuff ... but when I get in a room with the folks at WhereCamp I realize I don't know anything at all. And it's all so interesting I want to learn all of it! It's a terrific and energetic unconference. I

I won't try to write up a full report, but here are some highlights.

Several Grassroots Mapping people were there again this year. Jeffrey Warren led people in constructing balloons from tape and mylar space blankets in the morning, and they shot some aerial photos. Then in a late-afternoon session he discussed how to stitch the aerial photos together using Cargen Knitter.

But he also had other projects to discuss: the Passenger Pigeon project to give cameras to people who will be flying over environmental that need to be monitored -- like New York's Gowanus Canal superfund site, next to La Guardia airport. And the new Public Laboratory for Open Technology and Science has a new project making vegetation maps by taking aerial photos with two cameras simultaneously, one normal, one modified for infra-red photography.

How do you make an IR camera? First you have to remove the IR-blocking filter that all digital cameras come with (CCD sensors are very sensitive to IR light). Then you need to add a filter that blocks out most of the visible light. How? Well, it turns out that exposed photographic film (remember film?) makes a good IR-only filter. So you go to a camera store, buy a roll of film, rip it out of the reel while ignoring the screams of the people in the store, then hand it back to them and ask to have it developed. Cheap and easy.

Even cooler, you can use a similar technique to make a spectrometer from a camera, a cardboard box and a broken CD. Jeffrey showed spectra for several common objects, including bacon (actually pancetta, it turns out).
JW: See the dip in the UV? Pork fat is very absorbent in the UV. That's why some people use pork products as sunscreen.
Audience member: Who are these people?
JW: Well, I read about them on the internet.
I ask you, how can you beat a talk like that?

Two Google representatives gave an interesting demo of some of the new Google APIs related to maps and data visualization, in particular Fusion Tables. Motion charts sounded especially interesting but they didn't have a demo handy; there may be one appearing soon in the Fusion Charts gallery. They also showed the new enterprise-oriented Google Earth Builder, and custom street views for Google Maps.

There were a lot of informal discussion sessions, people brainstorming and sharing ideas. Some of the most interesting ones I went to included

Lightning talks included demonstrations and discussions of global Twitter activity as the Japanese quake and tsunami news unfolded, the new CD from OSGeo, the upcoming PII conference -- that's privacy identity innovation -- in Santa Clara.

There were quite a few outdoor game sessions Friday. I didn't take part myself since they all relied on having an iPhone or Android phone: my Archos 5 isn't reliable enough at picking up distant wi-fi signals to work as an always-connected device, and the Stanford wi-fi net was very flaky even with my laptop, with lots of dropped connections.

Even the OpenStreetMap mapping party was set up to require smartphones, in contrast with past mapping parties that used Garmin GPS units. Maybe this is ultimately a good thing: every mapping party I've been to fizzled out after everyone got back and tried to upload their data and discovered that nobody had GPSBabel installed, nor the drivers for reading data off a Garmin. I suspect most mapping party data ended up getting tossed out. If everybody's uploading their data in realtime with smartphones, you avoid all that and get a lot more data. But it does limit your contributors a bit.

There were a couple of lowlights. Parking was very tight, and somewhat expensive on Friday, and there wasn't any info on the site except a cheerfully misleading "There's plenty of parking!" And the lunch schedule on Saturday as a bit of a mess -- no one was sure when the lunch break was (it wasn't on the schedule), so afternoon schedule had to be re-done a couple times while everybody worked it out. Still, those are pretty trivial complaints -- sheesh, it's a free, volunteer conference! and they even provided free meals, and t-shirts too!

Really, WhereCamp is an astoundingly fun gathering. I always leave full of inspiration and ideas, and appreciation for the amazing people and projects presented there. A big thanks to the organizers and sponsors. I can't wait 'til next year -- and I hope I'll have something worth presenting then!

Tags: , ,
[ 23:40 Apr 24, 2011    More mapping | permalink to this entry | comments ]

Wed, 03 Nov 2010

Garmin GPX timestamp bizarreness

My last entry mentioned some work I'd done to one of my mapping programs, Ellie, to gather statistics from the track logs I get from my Garmin GPS.

In the course of working on Ellie, I discovered something phenomenally silly about the GPX files from my Garmin Vista CX, as uploaded with gpsbabel.

Track log points, quite reasonably, have time stamps in "Zulu time" (essentially the same as GMT, give or take some fraction of a second). They look like this:

<trkpt lat="35.289519913" lon="-115.227057561">

But the waypoints you set for specific points of interest, even if they're in the same GPX file, have timestamps that have no time zone at all. They look like this:

<wpt lat="35.334813371" lon="-115.178730609">
  <cmt>14-OCT-10 11:18:51AM</cmt>
  <desc>14-OCT-10 11:18:51AM</desc>
  <sym>Flag, Blue</sym>

Notice the waypoint's time isn't actually in a time field -- it's duplicated in two fields, cmt (comment) and desc (description). So it's not really intended to be a time stamp -- but it sure would be handy if you could use it as one.

You might be able to correlate waypoints with track points by comparing coordinates ... unless you spent more than an hour hanging around a particular location, or came back several hours later (perhaps starting and ending your hike at the same place). In that case ... you'd better know what the local time zone was, including daylight savings time.

What a silly omission, considering that the GPS obviously already knows the Zulu time and could just as easily use that!

Tags: , ,
[ 22:09 Nov 03, 2010    More mapping | permalink to this entry | comments ]

Sat, 30 Oct 2010

New versions of mapping programs: Pytopo and Ellie

[pytopo logo] On our recent Mojave trip, as usual I spent some of the evenings reviewing maps and track logs from some of the neat places we explored.

There isn't really any existing open source program for offline mapping, something that works even when you don't have a network. So long ago, I wrote Pytopo, a little program that can take map tiles from a Windows program called Topo! (or tiles you generate yourself somehow) and let you navigate around in that map.

But in the last few years, a wonderful new source of map tiles has become available: OpenStreetMap. On my last desert trip, I whipped up some code to show OSM tiles, but a lot of the code was hacky and empirical because I couldn't find any documentation for details like the tile naming scheme.

Well, that's changed. Upon returning to civilization I discovered there's now a wonderful page explaining the Slippy map tilenames very clearly, with sample code and everything. And that was the missing piece -- from there, all the things I'd been missing in pytopo came together, and now it's a useful self-contained mapping script that can download its own tiles, and cache them so that when you lose net access, your maps don't disappear along with everything else.

Pytopo can show GPS track logs and waypoints, so you can see where you went as well as where you might want to go, and whether that road off to the right actually would have connected with where you thought you were heading.

It's all updated in svn and on the Pytopo page.


[Ellie icon]

Most of the pytopo work came after returning from the desert, when I was able to google and find that OSM tile naming page. But while still out there and with no access to the web, I wanted to review the track logs from some of our hikes and see how much climbing we'd done. I have a simple package for plotting elevation from track logs, called Ellie. But when I ran it, I discovered that I'd never gotten around to installing the pylab Python plotting package (say that three times fast!) on this laptop.

No hope of installing the package without a net ... so instead, I tweaked Ellie so that so that without pylab you can still print out statistics like total climb. While I was at it I added total distance, time spent moving and time spent stopped. Not a big deal, but it gave me the numbers I wanted. It's available as ellie 0.3.

Tags: , ,
[ 19:24 Oct 30, 2010    More mapping | permalink to this entry | comments ]

Wed, 21 Jul 2010

Writing scripts for your Canon camera with CHDK

On Linux Planet yesterday: an article on how to write scripts for chdk, the Canon Hack Development Kit -- Part 3 in my series on CHDK.

Time-Lapse Photography with your Inexpensive Canon Camera (CHDK p. 3)

I found that CHDK scripting wasn't quite as good as I'd hoped -- some of the functions, especially the aperture and shutter setting, were quite flaky on my A540 so it really didn't work to write a bracketing script. But it's fantastic for simple tasks like time-lapse photography, or taking a series of shots like the Grass Roots Mapping folk do.

If you're at OSCON and you like scripting and photos, check out my session on Thursday afternoon at 4:30: Writing GIMP Plug-ins and Scripts, in which I'll walk through several GIMP scripts in Python and Script-Fu and show some little-known tricks you can do with Python plug-ins.

Tags: , , , , , ,
[ 10:31 Jul 21, 2010    More photo | permalink to this entry | comments ]

Mon, 05 Apr 2010

Where 2.0 2010

Last week I had the opportunity to go to the Where 2.0 conference (thanks, Linux Pro Magazine!) Then, on the weekend, the free WhereCamp followed it up.

I'd been to WhereCamp last year. It was wonderful, geeky, highly technical and greatly inspiring. I thought I was the only person interested in mapping, especially in Python, and after the first couple of sessions I was blown away with how little I knew and what a thriving and expert community there was. I was looking forward to the full experience this year -- I figured Where 2.0 must be similar but even better.

Actually they're completely different events. Where 2.0 was dominated by location-aware startups: people with iPhone games (Foursquare and others in a similar mold), shopping apps (find the closest pizza place to your location!) and so on. The talks were mostly 15 minutes long, so while there were lots of people there with fascinating apps or great stories to tell, there was no time to get detail on anything. I think the real point of Where 2.0 is to get a sketch of who's doing what so you can go collar them in the "hallway track" later and make business deals.

Here are some highlights from Where 2.0. I'll write up WhereCamp separately.

Ignite Where

The Ignite session Tuesday night was great fun, as Ignite sessions almost always are.

The Ignite session was broken in the middle by a half-hour interlude where a bunch of startups gave one-minute presentations on their products, then the audience voted on the best, then an award was given which had already been decided and had nothing to do with the audience vote (we didn't even get to find out which company the audience chose). Big yawner: one minute isn't long enough for anyone to show off a product meaningfully, and I wasn't the only one there who brought reading material to keep them occupied until the second round of Ignite talks started up again.

Best Ignite talks (Ignite Where 2.0 videos here):

Wednesday talks

Patrick Meier gave a longer version of his Ignite talk on Mobilizing Ushahidi-Haiti, full of interesting stories of how OpenStreetMap and other technologies like Twitter came together to help in the Haiti rescue effort.

Clouds, Crowds, and Shrouds: How One Government Agency Seeks to Change the Way It Spatially Enables Its Information, by Terrance Busch of the US Defense Intelligence Agency, was an interesting look into the challenges of setting up a serious mapping effort, then integrating later with commercial and crowdsourced efforts.

In Complexities in Bringing Home Environmental Awareness, Kim Balassiano of the US EPA showed the EPA MyEnvironment page, where you can find information about local environmental issues like toxic waste cleanups. They want users to enter good news too, like composting workshops or community gardens, but so far the data on the map is mostly bad. Still a useful site.

Thursday talks

There were a couple of interesting keynotes on Thursday morning, but work kept me at home. I thought I could catch them on the live video stream, but unfortunately the stream that had worked fine on Wednesday wasn't working on Thursday, so I missed the Mark L. DeMulder's talk on the USGS's National Map efforts. Fortunately, they were at WhereCamp where they gave much more detail. Likewise, I missed the big ESRI announcement that everyone was talking about all afternoon -- they released some web thing, but as far as I can tell they're still totally Windows-centric and thus irrelevant to a Linux and open source user. But I want to go back and view the video anyway.

There was another talk on Thursday which I won't name, but it had a few lessons for speakers:

Base Map 2.0 was a panel-slash-debate between Steve Coast (OpenStreetMap), Timothy Trainor (U.S. Census Bureau), Peter ter Haar (Ordnance Survey), Di-Ann Eisnor (Platial), and moderated by Ian White of Urban Mapping. It was fabulous. I've never seen such a lively panel: White kept things moving, told jokes, asked provocative and sometimes inflammatory questions and was by far the best panel moderator I've seen. The panelists kept up with him and gave cogent, interesting and illuminating answers. Two big issues were the just-announced release of Ordnance Survey data, and licensing issues causing mismatches between OSM, OS and Census datasets.

Community-based Grassroots Mapping with Balloons and Kites in Lima, Peru by Jeffrey Warren was another fabulous talk. He builds balloons out of garbage bags, soda bottles and a digital camera, goes to poor communities in places like Lima and teaches the community (including the kids) how to map their own communities. This is more than an academic exercise for them, since maps can help them prove title to their land. Check it out at and build your own aerial mapping balloon! (He was at WhereCamp, too, where we got to see the equipment up close.)

Visualizing Spatio-temporal War Casualty Data in Google Earth by Sean Askay of Google was just as good. He's built a KML file called Map the Fallen showing US and allied casualties from Iraq: the soldiers' hometowns, place of death, age, gender, and lots of other details about them with links to tribute pages, plus temporal information showing how casualties changed as the war progressed. It's an amazing piece of work, and sobering ... and I was most annoyed to find out that it needs a version of Google Earth that doesn't run on Linux, so I can't run it for myself. Boo!

Overall, a very fun conference, though it left me hungry for detail. Happily, after a day off there was WhereCamp to fill that void.

Tags: ,
[ 22:34 Apr 05, 2010    More conferences | permalink to this entry | comments ]

Fri, 08 Jan 2010

Python-GTK regression: How to catch mouse button release

We just had the second earthquake in two days, and I was chatting with someone about past earthquakes and wanted to measure the distance to some local landmarks. So I fired up PyTopo as the easiest way to do that. Click on one point, click on a second point and it prints distance and bearing from the first point to the second.

Except it didn't. In fact, clicks weren't working at all. And although I have hacked a bit on parts of pytopo (the most recent project was trying to get scaling working properly in tiles imported from OpenStreetMap), the click handling isn't something I've touched in quite a while.

It turned out that there's a regression in PyGTK: mouse button release events now need you to set an event mask for button presses as well as button releases. You need both, for some reason. So you now need code that looks like this:

drawing_area.connect("button-release-event", button_event)
drawing_area.set_events(gtk.gdk.EXPOSURE_MASK |
                        # next line wasn't needed before:
                        gtk.gdk.BUTTON_PRESS_MASK |
                        gtk.gdk.BUTTON_RELEASE_MASK )

An easy fix ... once you find it.

I filed bug 606453 to see whether the regression was intentional.

I've checked in the fix to the PyTopo svn repository on Google Code. It's so nice having a public source code repository like that! I'm planning to move Pho to Google Code soon.

Tags: , , ,
[ 14:20 Jan 08, 2010    More programming | permalink to this entry | comments ]

Sat, 20 Jun 2009

Pytopo 0.8 released

On my last Mojave trip, I spent a lot of the evenings hacking on PyTopo.

I was going to try to stick to OpenStreetMap and other existing mapping applications like TangoGPS, a neat little smartphone app for downloading OpenStreetMap tiles that also runs on the desktop -- but really, there still isn't any mapping app that works well enough for exploring maps when you have no net connection.

In particular, uploading my GPS track logs after a day of mapping, I discovered that Tango really wasn't a good way of exploring them, and I already know Merkaartor, nice as it is for entering new OSM data, isn't very good at working offline. There I was, with PyTopo and a boring hotel room; I couldn't stop myself from tweaking a bit.

Adding tracklogs was gratifyingly easy. But other aspects of the code bother me, and when I started looking at what I might need to do to display those Tango/OSM tiles ... well, I've known for a while that some day I'd need to refactor PyTopo's code, and now was the time.

Surprisingly, I completed most of the refactoring on the trip. But even after the refactoring, displaying those OSM tiles turned out to be a lot harder than I'd hoped, because I couldn't find any reliable way of mapping a tile name to the coordinates of that tile. I haven't found any documentation on that anywhere, and Tango and several other programs all do it differently and get slightly different coordinates. That one problem was to occupy my spare time for weeks after I got home, and I still don't have it solved.

But meanwhile, the rest of the refactoring was done, nice features like track logs were working, and I've had to move on to other projects. I am going to finish the OSM tile MapCollection class, but why hold up a release with a lot of useful changes just for that?

So here's PyTopo 0.8, and the couple of known problems with the new features will have to wait for 0.9.

Tags: , , , ,
[ 20:49 Jun 20, 2009    More programming | permalink to this entry | comments ]

Wed, 10 Jun 2009

Bing thinks we're WHERE?

Lots has been written about Bing, Microsoft's new search engine. It's better than Google, it's worse than Google, it'll never catch up to Google. Farhad Manjoo of Slate had perhaps the best reason to use Bing: "If you switch, Google's going to do some awesome things to try to win you back."

[Bing in Omniweb thinks we're in Portugal] But what I want to know about Bing is this: Why does it think we're in Portugal when Dave runs it under Omniweb on Mac?

In every other browser it gives the screen you've probably seen, with side menus (and a horizontal scrollbar if your window isn't wide enough, ugh) and some sort of pretty picture as a background. In Omniweb, you get a cleaner layout with no sidebars or horizontal scrollbars, a different pretty picture -- often prettier than the one you get on all the other browsers, though both images change daily -- and a set of togglebuttons that don't show up in any of the other browsers, letting you restrict results to only English or only results from Portugal.

Why does it think we're in Portugal when Dave uses Omniweb?

Equally puzzling, why do only people in Portugal have the option of restricting the results to English only?

Tags: , , ,
[ 10:37 Jun 10, 2009    More tech | permalink to this entry | comments ]

Mon, 01 Jun 2009

A GPX file manager

Someone on the OSM newbies list asked how he could strip waypoints out of a GPX track file. Seems he has track logs of an interesting and mostly-unmapped place that he wants to add to openstreetmap, but there are some waypoints that shouldn't be included, and he wanted a good way of separating them out before uploading.

Most of the replies involved "just edit the XML." Sure, GPX files are pretty simple and readable XML -- but a user shouldn't ever have to do that! Gpsman and gpsbabel were also mentioned, but they're not terribly easy to use either.

That reminded me that I had another XML-parsing task I'd been wanting to write in Python: a way to split track files from my Garmin GPS.

Sometimes, after a day of mapping, I end up with several track segments in the same track log file. Maybe I mapped several different trails; maybe I didn't get a chance to upload one day's mapping before going out the next day. Invariably some of the segments are of zero length (I don't know why the Garmin does that, but it always does). Applications like merkaartor don't like this one bit, so I usually end up editing the XML file and splitting it into segments by hand. I'm comfortable with XML -- but it's still silly.

I already have some basic XML parsing as part of PyTopo and Ellie, so I know the parsing very easy to do. So, spurred on by the posting on OSM-newbies, I wrote a little GPX parser/splitter called gpxmgr. gpxmgr -l file.gpx can show you how many track logs are in the file; gpxmgr -w file.gpx can write new files for each non-zero track log. Add -p if you want to be prompted for each filename (otherwise it'll use the name of the track log, which might be something like "ACTIVE\ LOG\ #2").

How, you may wonder, does that help the original poster's need to separate out waypoints from track files? It doesn't. See, my GPS won't save tracklogs and waypoints in the same file, even if you want them that way; you have to use two separate gpsbabel commands to upload a track file and a waypoint file. So I don't actually know what a tracklog-plus-waypoint file looks like. If anyone wants to use gpxmgr to manage waypoints as well as tracks, send me a sample GPX file that combines them both.

Tags: , ,
[ 20:43 Jun 01, 2009    More mapping | permalink to this entry | comments ]

Tue, 03 Mar 2009

Ellie: Plot GPS elevation profiles

Ever since I got the GPS I've been wanting something that plots the elevation data it stores. There are lots of apps that will show me the track I followed in latitude and longitude, but I couldn't find anything that would plot elevations.

But GPX (the XML-based format commonly used to upload track logs) is very straightforward -- you can look at the file and read the elevations right out of it. I knew it wouldn't be hard to write a script to plot them in Python; it just needed a few quiet hours. Sounded like just the ticket for a rainy day stuck at home with a sore throat.

Sure enough, it was fairly easy. I used xml.dom.minidom to parse the file (I'd already had some experience with it in gimplabels for converting gLabels templates), and pylab from matplotlib for doing the plotting. Easy and nice looking.

I even threw in the nice "conditional main" code from Matt Harrison's SCALE7x Python talk, so it should be callable from other Python code.

Here's the page and a screenshot: Ellie: plot elevation from a GPS track.

Tags: , ,
[ 17:57 Mar 03, 2009    More programming | permalink to this entry | comments ]

Tue, 20 Jan 2009

LCA 2009 Tuesday

I missed a lot of the miniconf talks on Tuesday because I wanted to make some last-minute changes to my talk. But I do want to comment on one: Simon Greener's talk on "A Review of Australian Geodata Providers." Of course, I'm not in Australia, but it was quite interesting to hear how similar Australia's problematic geodata siguation is to the situation in the US. His presentation was entertaining, animated and I learned some interesting facts about GPS and geodata in general.

And Dave and I got another good astronomy opportunity with the dark skies at Peppermint Bay at the Speakers' Dinner. Despite occasional intrusive clouds we managed to get a great view of the Large Magellanic Cloud and a decent view of the small one, as well as eta Carinae and the star clouds between Crux and Carina. Pity I'd forgotten to bring my thumpin' travel optics that I'd been using the previous evening: a 6x20 monocular.

Tags: , , ,
[ 17:15 Jan 20, 2009    More conferences/lca2009 | permalink to this entry | comments ]

Sun, 04 Jan 2009

Garmin Vista Cx on Ubuntu "Hardy"

I got myself a GPS unit for Christmas.

I've been resisting the GPS siren song for years -- mostly because I knew it would be a huge time sink involving months of futzing with drivers and software trying to get it to do something useful.

But my experience at an OpenStreetMap mapping party got me fired up about it, and I ordered a Garmin Vista Cx.

Shopping for a handheld GPS is confusing. I was fairly convinced I wanted a Garmin, just because it's the brand used by most people in the open source mapping community so I knew they were likely to work. I wanted one with a barometric altimeter, because I wanted that data from my hikes and bike rides (and besides, it's fun to know how much you've climbed on an outing; I used to have a bike computer with an altimeter and it was a surprisingly good motivator for working harder and getting in better shape).

But Garmin has a bazillion models and I never found any comparison page explaining the differences among the various hiking eTrex models. Eventually I worked it out:

Garmin eTrex models, decoded

Color display. This generally also implies USB connectivity instead of serial, just because the color models are newer.
High precision (a more sensitive satellite receiver).
Takes micro-SD cards. This may not be important for storing tracks and waypoints (you can store quite a long track with the built-in memory) but they mean that you can load extra base maps, like topographic data or other useful features.
Vista, Summit
These models have barometric altimeters and magnetic compasses. (I never did figure out the difference between a Vista and a Summit, except that in the color models (C), Vistas take micro-SD cards (x) while Summits don't, so there's a Summit C and HC while Vistas come in Cx and HCx. I don't know what the difference is between a monochrome Summit and Vista.)
Legend, Venture
These have no altimeter or compass. A Venture is a Legend that comes without the bundled extras like SD card, USB cable and base maps, so it's cheaper.

For me, the price/performance curve pointed to the Vista Cx.

Loading maps

Loading base maps was simplicity itself, and I found lots of howtos on how to use downloadable maps. Just mount the micro-SD card on any computer, make a directory called Garmin, and name the file gmapsupp.img. I used the CloudMade map for California, and it worked great. There are lots of howtos on generating your own maps, too, and I'm looking forward to making some with topographic data (which the CloudMade maps don't have). The most promising howtos I've found so far are the OSM Map On Garmin page on the OSM wiki and the much more difficult, but gorgeous, Hiking Biking Mapswiki page.

Uploading tracks and waypoints

But the real goal was to be able to take this toy out on a hike, then come back and upload the track and waypoint files.

I already knew, from the mapping party, that Garmins have an odd misfeature: you can connect them in usb-storage mode, where they look like an external disk and don't need any special software ... but then you can't upload any waypoints. (In fact, when I tried it with my Vista Cx I didn't even see the track file.) To upload tracks and waypoints, you need to use something that speaks Garmin protocol: namely, the excellent GPSBabel.

So far so good. How do you call GPSbabel? Luckily for me, just before my GPS arrived, Iván Sánchez Ortega posted a useful little gpsbabel script to the OSM newbies list and I thought I was all set.

But once I actually had the Vista in hand, complete with track and waypoints from a walk around the block, it turned out it wasn't quite that simple -- because Ubuntu didn't create the /dev/ttyUSB0 that Iván's script used. A web search found tons of people having that problem on Ubuntu and talking about various workarounds, involving making sure the garmin_usb driver is blacklisted in /etc/modprobe.d/blacklist (it was already), adding a /etc/udev/rules.d/45-garmin.rules file that changes permissions and ownership of ... um, I guess of the file that isn't being created? That didn't make much sense. Anyway, none of it helped.

But finally I found the fix: keep the garmin_usb driver blacklisted use "usb:" as the device to pass to GPSBabel rather than "/dev/ttyUSB0". So the commands are:

gpsbabel -t -i garmin -f usb: -o gpx -F tracks.gpx
gpsbabel -i garmin -f usb: -o gpx -F waypoints.gpx

Like so many other things, it's easy once you know the secret! Viewing tracklogs works great in Merkaartor, though I haven't yet found an app that does anything useful with the elevation data. I may have to write one.

Update: After I wrote this but before I was able to post it, a discussion on the OSM Newbies list with someone who was having similar troubles resulted in this useful wiki page: Garmin on GNU/Linux. It may also be worth checking the Discussion tab on that wiki page for further information.

Update, October 2011:
As of Debian Squeeze or Ubuntu Natty, you need two steps:

  1. Add a line to /etc/modprobe.d/blacklist.conf:
    blacklist garmin_gps
  2. Create a udev file, /etc/udev/rules.d/51-garmin.rules, to set the permissions so that you can access the device without being root. It contains the line:
    ATTRS{idVendor}=="091e", ATTRS{idProduct}=="0003", MODE="0660", GROUP="plugdev"

Then use gpsbabel with usb: and you should be fine.

Tags: , , , ,
[ 16:31 Jan 04, 2009    More mapping | permalink to this entry | comments ]

Sat, 03 Jan 2009

OpenStreetMap mapping parties

Latest obsession: mapping with OpenStreetMap.

Last month, OpenStreetMap and its benefactor company CloudMade held a "mapping party" in Palo Alto. I love maps and mapping (I wrote my own little topographic map viewer when I couldn't find one ready-made) and I've been wanting to know more about the state of open source mapping. A mapping party sounded perfect.

The party was a loosely organized affair. We met at a coffeehouse and discussed basics of mapping and openstreetmap. The hosts tried to show us newbies how OSM works, but that was complicated by the coffeehouse's wireless net being down. No big deal -- turns out the point of a mapping party is to hand out GPSes to anyone who doesn't already have one and send us out to do some mapping.

I attached myself to a couple of CloudMade folks who had some experience already and we headed north on a pedestrian path. We spent a couple of hours walking urban trails and marking waypoints. Then we all converged on a tea shop (whose wireless worked a little better than the one at the coffeehouse, but still not very reliably) for lunch and transfer of track and waypoint files.

This part didn't work all that well. It turned out the units we were using (Garmin Legend HCx) can transfer files in two modes, USB mass storage (the easy way, just move files as if from an external disk) or USB Garmin protocol (the hard way: you have to use software like gpsbabel, or the Garmin software if you're on Windows). And in mass storage mode, you get a file but the waypoints aren't there.

The folks running the event all had Macs, and there were several Linux users there as well, but no Windows laptops. By the time the Macs both had gpsbabel downloaded over the tea shop's flaky net, it was past time for me to leave, so I never did get to see our waypoint files. Still, I could see it was possible (and one of the Linux attendees assured me that he had no trouble with any of the software; in fact, he found it easier than what the Mac people at the party were going through).

But I was still pretty jazzed about how easy OpenStreetMap is to use. You can contribute to the maps even without a GPS. Once you've registered on the site, you just click on the Edit tab on any map, and you see a flash application called "Potlatch" that lets you mark trails, roads or other features based on satellite images or the existing map. I was able to change a couple of mismarked roads near where I live, as well as adding a new trail and correcting the info on an existing one for one of the nearby parks.

If you prefer (as, I admit, I do) to work offline or don't like flash, you can use a Java app, JOSM, or a native app, merkaartor. Very cool! Merkaartor is my favorite so far (because it's faster and works better in standalone mode) though it's still fairly rough around the edges. They're all described on the OSM Map Editing page.

Of course, all this left me lusting after a GPS. But that's another story, to be told separately.

Tags: , ,
[ 13:00 Jan 03, 2009    More mapping | permalink to this entry | comments ]

Fri, 25 Aug 2006

PyTopo 0.5

Belated release announcement: 0.5b2 of my little map viewer PyTopo has been working well, so I released 0.5 last week with only a few minor changes from the beta. I'm sure I'll immediately find six major bugs -- but hey, that's what point releases are for. I only did betas this time because of the changed configuration file format.

I also made a start on a documentation page for the .pytopo file (though it doesn't really have much that wasn't already written in comments inside the script).

Tags: , , , ,
[ 22:10 Aug 25, 2006    More programming | permalink to this entry | comments ]

Sat, 03 Jun 2006

Cleaner, More Flexible Python Map Viewing

A few months ago, someone contacted me who was trying to use my PyTopo map display script for a different set of map data, the Topo! National Parks series. We exchanged some email about the format the maps used.

I'd been wanting to make PyTopo more general anyway, and already had some hacky code in my local version to let it use a local geologic map that I'd chopped into segments. So, faced with an Actual User (always a good incentive!), I took the opportunity to clean up the code, use some of Python's support for classes, and introduce several classes of map data.

I called it 0.5 beta 1 since it wasn't well tested. But in the last few days, I had occasion to do some map exploring, cleaned up a few remaining bugs, and implemented a feature which I hadn't gotten around to implementing in the new framework (saving maps to a file).

I think it's ready to use now. I'm going to do some more testing: after visiting the USGS Open House today and watching Jim Lienkaemper's narrated Virtual Tour of the Hayward Fault, I'm all fired up about trying again to find more online geologic map data. But meanwhile, PyTopo is feature complete and has the known bugs fixed. The latest version is on the PyTopo page.

Tags: , , , ,
[ 18:25 Jun 03, 2006    More programming | permalink to this entry | comments ]

Wed, 13 Apr 2005

PyTopo 0.3

I needed to print some maps for one of my geology class field trips, so I added a "save current map" key to PyTopo (which saves to .gif, and then I print it with gimp-print). It calls montage from Image Magick.

Get yer PyTopo 0.3 here.

Tags: , , ,
[ 17:56 Apr 13, 2005    More programming | permalink to this entry | comments ]

Wed, 06 Apr 2005

PyTopo is usable; pygtk is inefficient

While on vacation, I couldn't resist tweaking pytopo so that I could use it to explore some of the areas we were visiting.

It seems fairly usable now. You can scroll around, zoom in and out to change between the two different map series, and get the coordinates of a particular location by clicking. I celebrated by making a page for it, with a silly tux-peering-over-map icon.

One annoyance: it repaints every time it gets a focus in or out, which means, for people like me who use mouse focus, that it repaints twice for each time the mouse moves over the window. This isn't visible, but it would drag the CPU down a bit on a slow machine (which matters since mapping programs are particularly useful on laptops and handhelds).

It turns out this is a pygtk problem: any pygtk drawing area window gets spurious Expose events every time the focus changes (whether or not you've asked to track focus events), and it reports that the whole window needs to be repainted, and doesn't seem to be distinguishable in any way from a real Expose event. The regular gtk libraries (called from C) don't do this, nor do Xlib C programs; only pygtk.

I filed bug 172842 on pygtk; perhaps someone will come up with a workaround, though the couple of pygtk developers I found on #pygtk couldn't think of one (and said I shouldn't worry about it since most people don't use pointer focus ... sigh).

Tags: , , , ,
[ 17:26 Apr 06, 2005    More programming | permalink to this entry | comments ]

Sun, 27 Mar 2005

Python GTK Topographic Map Program

I couldn't stop myself -- I wrote up a little topo map viewer in PyGTK, so I can move around with arrow keys or by clicking near the edges. It makes it a lot easier to navigate the map directory if I don't know the exact starting coordinates.

It's called PyTopo, and it's in the same place as my earlier two topo scripts.

I think CoordsToFilename has some bugs; the data CD also has some holes, and some directories don't seem to exist in the expected place. I haven't figured that out yet.

Tags: , , , ,
[ 18:53 Mar 27, 2005    More programming | permalink to this entry | comments ]