Shallow Thoughts : : mapping

Akkana's Musings on Open Source Computing, Science, and Nature.

Tue, 20 Aug 2013

Using Google Maps with Python to turn a list of addresses into waypoints

A few days ago I tlaked about how I use making waypoint files for a list of house addresses is OsmAnd. For waypoint files, you need latitude/longitude coordinates, and I was getting those from a web page that used the online Google Maps API to convert an address into latitude and longitude coordinates.

It was pretty cool at first, but pasting every address into the latitude/longitude web page and then pasting the resulting coordinates into the address file, got old, fast. That's exactly the sort of repetitive task that computers are supposed to handle for us.

The lat/lon page used Javascript and the Google Maps API. and I already had a Google Maps API key (they have all sorts of fun APIs for map geeks) ... but I really wanted something that could run locally, reading and converting a local file.

And then I discovered the Python googlemaps package. Exactly what I needed! It's in the Python Package Index, so I installed it with pip install googlemaps. That enabled me to change my waymaker Python script: if the first line of a description wasn't a latitude and longitude, instead it looked for something that might be an address.

Addresses in my data files might be one line or might be two, but since they're all US addresses, I know they'll end with a two-capital-letter state abbreviation and a 5-digit zip code: 2948 W Main St. Anytown, NM 12345. You can find that with a regular expression:

    match = re.search('.*[A-Z]{2}\s+\d{5}$', line)

But first I needed to check whether the first line of the entry was already latitude/longitude coordinates, since I'd already converted some of my files. That uses another regular expression. Python doesn't seem to have a built-in way to search for generic numeric expressions (containing digits, decimal points or +/- symbols) so I made one, since I had to use it twice if I was searching for two numbers with whitespace between them.

    numeric = '[\+\-\d\.]'
    match = re.search('^(%s+)\s+(%s+)$' % (numeric, numeric),
                      line)
(For anyone who wants to quibble, I know the regular expression isn't perfect. For instance, it would match expressions like 23+48..6.1-64.5. Not likely to be a problem in these files, so I didn't tune it further.)

If the script doesn't find coordinates but does find something that looks like an address, it feeds the address into Google Maps and gets the resulting coordinates. That code looks like this:

from googlemaps import GoogleMaps

gmaps = GoogleMaps('YOUR GOOGLE MAPS API KEY HERE')
try:
    lat, lon = gmaps.address_to_latlng(addr)
except googlemaps.GoogleMapsError, e:
    print "Oh, no! Couldn't geocode", addr
    print e

Overall, a nice simple solution made possible with python-googlemaps. The full script is on github: waymaker.

Tags: , , , , , ,
[ 11:24 Aug 20, 2013    More mapping | permalink to this entry | comments ]

Fri, 16 Aug 2013

Offline mapping with lists of waypoints

Dave and I have been doing some exploratory househunting trips, and one of the challenges is how to maintain a list of houses and navigate from location to location. It's basically like geocaching, navigating from one known location to the next.

Sure, there are smartphone apps to do things like "show houses for sale near here" against a Google Maps background. But we didn't want everything, just the few gems we'd picked out ahead of time. And some of the places we're looking are fairly remote -- you can't always count on a consistent signal everywhere as you drive around, let alone a connection fast enough to download map tiles.

Fortunately, I use a wonderful open-source Android program called OsmAnd. It's the best, bar none, at offline mapping: download data files prepared from OpenStreetMap vector data, and you're good to go, even into remote areas with no network connectivity. It's saved our butts more than once exploring remote dirt tracks in the Mojave. And since the maps come from OpenStreetMap, if you find anything wrong with the map, you can fix it.

So the map part is taken care of. What about that list of houses?

Making waypoint files

On the other hand, one of OsmAnd's many cool features is that it can show track logs. I can upload a GPX file from my Garmin, or record a track within OsmAnd, and display the track on OsmAnd's map.

GPX track files can include waypoints. What if I made a GPX file consisting only of waypoints and descriptions for each house?

My husband was already making text files of potentially interesting houses:

404 E David Dr 
Flagstaff, AZ 86001
$355,000
3 Bed 2 Bath
1,673 Sq Ft
0.23 acres
http://blahblah/long_url

2948 W Wilson Dr 
Flagstaff, AZ 86001
$285,000
3 Bed 2 Bath
1,908 Sq Ft
8,000 Sq Ft Lot 
http://blahblah/long_url

... (and so on)
So I just needed to turn those into GPX.

GPX is a fairly straightforward XML format -- I've parsed GPX files for pytopo and for ellie, and generating them from Python should be easier than parsing. But first I needed latitude and longitude coordinates. A quick web search solved that: an excellent page called Find latitude and longitude with Google Maps. You paste the address in and it shows you the location on a map along with latitude and longitude. Thanks to Bernard Vatant at Mondeca!

For each house, I copied the coordinates directly from the page and pasted them into the file. (Though that got old after about the fifth house; I'll write about automating that step in a separate article.)

Then I wrote a script called waymaker that parses a file of coordinates and descriptions and makes waypoint files. Run it like this: waymaker infile.txt outfile.gpx and it will create (or overwrite) a gpx file consisting of those waypoints.

Getting it into OsmAnd

I plugged my Android device into my computer's USB port, mounted it as usb-storage and copied all the GPX files into osmand/tracks (I had to create the tracks subdirectory myself, since I hadn't recorded any tracks. After restarting OsmAnd, it was able to see all the waypoint files.

OsmAnd has a couple of other ways of showing points besides track files. "Favorites" lets you mark a point on the map and save it to various Favorites categories. But although there's a file named favorites.gpx, changes you make to it never show up in the program. Apparently they're cached somewhere else. "POI" (short for Points of Interest) can be uploaded, but only as a .obf OsmAnd file or a .sqlitedb database, and there isn't much documentation on how to create either one. GPX tracks seemed like the easiest solution, and I've been happy with them so far.

Update: I asked on the osmand mailing list; it turns out that on the Favorites screen (Define View, then Favorites) there's a Refresh button that makes osmand re-read favorites.gpx. Works great. It uses pretty much the same format as track files -- I took <wpt></wpt> sequences I'd generated with waymaker and added them to my existing favorites.gpx file, adding appropriate categories. It's nice to have two different ways to display and categorize waypoints within the app.

Using waypoints in OsmAnd

How do you view these waypoints once they're loaded? When you're in OsmAnd's map view, tap the menu button and choose Define View, then GPX track... You'll see a list of all your GPX files; choose the one you want.

You'll be taken back to the map view, at a location and zoom level that shows all your waypoints. Don't panic if you don't see them immediately; sometimes I needed to scroll and zoom around a little before OsmAnd noticed there were waypoints and started drawing them.

Then you can navigate in the usual way. When you get to a waypoint, tap on it to see the description brieftly -- I was happy to find that multiple line descriptions work just fine. Or long-press on it to pop up a persistent description window that will stay up until you dismiss it.

It worked beautifully for our trip, both for houses and for other things like motels and points of interest along the way.

Tags: , , ,
[ 14:58 Aug 16, 2013    More mapping | permalink to this entry | comments ]

Wed, 16 Nov 2011

New trails, and new PyTopo 1.1 release

A new trail opened up above Alum Rock park! Actually a whole new open space preserve, called Sierra Vista -- with an extensive set of trails that go all sorts of interesting places.

Dave and I visit Alum Rock frequently -- we were married there -- so having so much new trail mileage is exciting. We tried to explore it on foot, but quickly realized the mileage was more suited to mountain bikes. Even with bikes, we'll be exploring this area for a while (mostly due to not having biked in far too long, so it'll take us a while to work up to that much riding ... a combination of health problems and family issues have conspired to keep us off the bikes).

Of course, part of the fun of discovering a new trail system is poring over maps trying to figure out where the trails will take us, then taking GPS track logs to study later to see where we actually went.

And as usual when uploading GPS track logs and viewing them in pytopo, I found some things that weren't working quite the way I wanted, so the session ended up being less about studying maps and more about hacking Python.

In the end, I fixed quite a few little bugs, improved some features, and got saved sites with saved zoom levels working far better.

Now, PyTopo 1.0 happened quite a while ago -- but there were two of us hacking madly on it at the time, and pinning down the exact time when it should be called 1.0 wasn't easy. In fact, we never actually did it. I know that sounds silly -- of all releases to not get around to, finally reaching 1.0? Nevertheless, that's what happened.

I thought about cheating and calling this one 1.0, but we've had 1.0 beta RPMs floating around for so long (and for a much earlier release) that that didn't seem right.

So I've called the new release PyTopo 1.1. It seems to be working pretty solidly. It's certainly been very helpful to me in exploring the new trails. It's great for cross-checking with Google Earth: the OpenCycleMap database has much better trail data than Google does, and pytopo has easy track log loading and will work offline, while Google has the 3-D projection aerial imagery that shows where trails and roads were historically (which may or may not correspond to where they decide to put the new trails). It's great to have both.

Anyway, here's the new PyTopo.

Tags: , ,
[ 19:59 Nov 16, 2011    More mapping | permalink to this entry | comments ]

Sun, 24 Apr 2011

WhereCamp 2011

I spent Friday and Saturday at the WhereCamp unconference on mapping, geolocation and related topics.

This was my second year at WhereCamp. It's always a bit humbling. I feel like I'm pretty geeky, and I've written a couple of Python mapping apps and I know spherical geometry and stuff ... but when I get in a room with the folks at WhereCamp I realize I don't know anything at all. And it's all so interesting I want to learn all of it! It's a terrific and energetic unconference. I

I won't try to write up a full report, but here are some highlights.

Several Grassroots Mapping people were there again this year. Jeffrey Warren led people in constructing balloons from tape and mylar space blankets in the morning, and they shot some aerial photos. Then in a late-afternoon session he discussed how to stitch the aerial photos together using Cargen Knitter.

But he also had other projects to discuss: the Passenger Pigeon project to give cameras to people who will be flying over environmental that need to be monitored -- like New York's Gowanus Canal superfund site, next to La Guardia airport. And the new Public Laboratory for Open Technology and Science has a new project making vegetation maps by taking aerial photos with two cameras simultaneously, one normal, one modified for infra-red photography.

How do you make an IR camera? First you have to remove the IR-blocking filter that all digital cameras come with (CCD sensors are very sensitive to IR light). Then you need to add a filter that blocks out most of the visible light. How? Well, it turns out that exposed photographic film (remember film?) makes a good IR-only filter. So you go to a camera store, buy a roll of film, rip it out of the reel while ignoring the screams of the people in the store, then hand it back to them and ask to have it developed. Cheap and easy.

Even cooler, you can use a similar technique to make a spectrometer from a camera, a cardboard box and a broken CD. Jeffrey showed spectra for several common objects, including bacon (actually pancetta, it turns out).
JW: See the dip in the UV? Pork fat is very absorbent in the UV. That's why some people use pork products as sunscreen.
Audience member: Who are these people?
JW: Well, I read about them on the internet.
I ask you, how can you beat a talk like that?

Two Google representatives gave an interesting demo of some of the new Google APIs related to maps and data visualization, in particular Fusion Tables. Motion charts sounded especially interesting but they didn't have a demo handy; there may be one appearing soon in the Fusion Charts gallery. They also showed the new enterprise-oriented Google Earth Builder, and custom street views for Google Maps.

There were a lot of informal discussion sessions, people brainstorming and sharing ideas. Some of the most interesting ones I went to included

Lightning talks included demonstrations and discussions of global Twitter activity as the Japanese quake and tsunami news unfolded, the new CD from OSGeo, the upcoming PII conference -- that's privacy identity innovation -- in Santa Clara.

There were quite a few outdoor game sessions Friday. I didn't take part myself since they all relied on having an iPhone or Android phone: my Archos 5 isn't reliable enough at picking up distant wi-fi signals to work as an always-connected device, and the Stanford wi-fi net was very flaky even with my laptop, with lots of dropped connections.

Even the OpenStreetMap mapping party was set up to require smartphones, in contrast with past mapping parties that used Garmin GPS units. Maybe this is ultimately a good thing: every mapping party I've been to fizzled out after everyone got back and tried to upload their data and discovered that nobody had GPSBabel installed, nor the drivers for reading data off a Garmin. I suspect most mapping party data ended up getting tossed out. If everybody's uploading their data in realtime with smartphones, you avoid all that and get a lot more data. But it does limit your contributors a bit.

There were a couple of lowlights. Parking was very tight, and somewhat expensive on Friday, and there wasn't any info on the site except a cheerfully misleading "There's plenty of parking!" And the lunch schedule on Saturday as a bit of a mess -- no one was sure when the lunch break was (it wasn't on the schedule), so afternoon schedule had to be re-done a couple times while everybody worked it out. Still, those are pretty trivial complaints -- sheesh, it's a free, volunteer conference! and they even provided free meals, and t-shirts too!

Really, WhereCamp is an astoundingly fun gathering. I always leave full of inspiration and ideas, and appreciation for the amazing people and projects presented there. A big thanks to the organizers and sponsors. I can't wait 'til next year -- and I hope I'll have something worth presenting then!

Tags: , ,
[ 22:40 Apr 24, 2011    More mapping | permalink to this entry | comments ]

Wed, 03 Nov 2010

Garmin GPX timestamp bizarreness

My last entry mentioned some work I'd done to one of my mapping programs, Ellie, to gather statistics from the track logs I get from my Garmin GPS.

In the course of working on Ellie, I discovered something phenomenally silly about the GPX files from my Garmin Vista CX, as uploaded with gpsbabel.

Track log points, quite reasonably, have time stamps in "Zulu time" (essentially the same as GMT, give or take some fraction of a second). They look like this:

<trkpt lat="35.289519913" lon="-115.227057561">
  <ele>1441.634277</ele>
  <time>2010-10-14T17:51:35Z</time>
</trkpt>

But the waypoints you set for specific points of interest, even if they're in the same GPX file, have timestamps that have no time zone at all. They look like this:

<wpt lat="35.334813371" lon="-115.178730609">
  <ele>1489.917480</ele>
  <name>001</name>
  <cmt>14-OCT-10 11:18:51AM</cmt>
  <desc>14-OCT-10 11:18:51AM</desc>
  <sym>Flag, Blue</sym>
</wpt>

Notice the waypoint's time isn't actually in a time field -- it's duplicated in two fields, cmt (comment) and desc (description). So it's not really intended to be a time stamp -- but it sure would be handy if you could use it as one.

You might be able to correlate waypoints with track points by comparing coordinates ... unless you spent more than an hour hanging around a particular location, or came back several hours later (perhaps starting and ending your hike at the same place). In that case ... you'd better know what the local time zone was, including daylight savings time.

What a silly omission, considering that the GPS obviously already knows the Zulu time and could just as easily use that!

Tags: , ,
[ 21:09 Nov 03, 2010    More mapping | permalink to this entry | comments ]

Sat, 30 Oct 2010

New versions of mapping programs: Pytopo and Ellie

[pytopo logo] On our recent Mojave trip, as usual I spent some of the evenings reviewing maps and track logs from some of the neat places we explored.

There isn't really any existing open source program for offline mapping, something that works even when you don't have a network. So long ago, I wrote Pytopo, a little program that can take map tiles from a Windows program called Topo! (or tiles you generate yourself somehow) and let you navigate around in that map.

But in the last few years, a wonderful new source of map tiles has become available: OpenStreetMap. On my last desert trip, I whipped up some code to show OSM tiles, but a lot of the code was hacky and empirical because I couldn't find any documentation for details like the tile naming scheme.

Well, that's changed. Upon returning to civilization I discovered there's now a wonderful page explaining the Slippy map tilenames very clearly, with sample code and everything. And that was the missing piece -- from there, all the things I'd been missing in pytopo came together, and now it's a useful self-contained mapping script that can download its own tiles, and cache them so that when you lose net access, your maps don't disappear along with everything else.

Pytopo can show GPS track logs and waypoints, so you can see where you went as well as where you might want to go, and whether that road off to the right actually would have connected with where you thought you were heading.

It's all updated in svn and on the Pytopo page.

Ellie

[Ellie icon]

Most of the pytopo work came after returning from the desert, when I was able to google and find that OSM tile naming page. But while still out there and with no access to the web, I wanted to review the track logs from some of our hikes and see how much climbing we'd done. I have a simple package for plotting elevation from track logs, called Ellie. But when I ran it, I discovered that I'd never gotten around to installing the pylab Python plotting package (say that three times fast!) on this laptop.

No hope of installing the package without a net ... so instead, I tweaked Ellie so that so that without pylab you can still print out statistics like total climb. While I was at it I added total distance, time spent moving and time spent stopped. Not a big deal, but it gave me the numbers I wanted. It's available as ellie 0.3.

Tags: , ,
[ 18:24 Oct 30, 2010    More mapping | permalink to this entry | comments ]

Mon, 01 Jun 2009

A GPX file manager

Someone on the OSM newbies list asked how he could strip waypoints out of a GPX track file. Seems he has track logs of an interesting and mostly-unmapped place that he wants to add to openstreetmap, but there are some waypoints that shouldn't be included, and he wanted a good way of separating them out before uploading.

Most of the replies involved "just edit the XML." Sure, GPX files are pretty simple and readable XML -- but a user shouldn't ever have to do that! Gpsman and gpsbabel were also mentioned, but they're not terribly easy to use either.

That reminded me that I had another XML-parsing task I'd been wanting to write in Python: a way to split track files from my Garmin GPS.

Sometimes, after a day of mapping, I end up with several track segments in the same track log file. Maybe I mapped several different trails; maybe I didn't get a chance to upload one day's mapping before going out the next day. Invariably some of the segments are of zero length (I don't know why the Garmin does that, but it always does). Applications like merkaartor don't like this one bit, so I usually end up editing the XML file and splitting it into segments by hand. I'm comfortable with XML -- but it's still silly.

I already have some basic XML parsing as part of PyTopo and Ellie, so I know the parsing very easy to do. So, spurred on by the posting on OSM-newbies, I wrote a little GPX parser/splitter called gpxmgr. gpxmgr -l file.gpx can show you how many track logs are in the file; gpxmgr -w file.gpx can write new files for each non-zero track log. Add -p if you want to be prompted for each filename (otherwise it'll use the name of the track log, which might be something like "ACTIVE\ LOG\ #2").

How, you may wonder, does that help the original poster's need to separate out waypoints from track files? It doesn't. See, my GPS won't save tracklogs and waypoints in the same file, even if you want them that way; you have to use two separate gpsbabel commands to upload a track file and a waypoint file. So I don't actually know what a tracklog-plus-waypoint file looks like. If anyone wants to use gpxmgr to manage waypoints as well as tracks, send me a sample GPX file that combines them both.

Tags: , ,
[ 19:43 Jun 01, 2009    More mapping | permalink to this entry | comments ]

Sun, 04 Jan 2009

Garmin Vista Cx on Ubuntu "Hardy"

I got myself a GPS unit for Christmas.

I've been resisting the GPS siren song for years -- mostly because I knew it would be a huge time sink involving months of futzing with drivers and software trying to get it to do something useful.

But my experience at an OpenStreetMap mapping party got me fired up about it, and I ordered a Garmin Vista Cx.

Shopping for a handheld GPS is confusing. I was fairly convinced I wanted a Garmin, just because it's the brand used by most people in the open source mapping community so I knew they were likely to work. I wanted one with a barometric altimeter, because I wanted that data from my hikes and bike rides (and besides, it's fun to know how much you've climbed on an outing; I used to have a bike computer with an altimeter and it was a surprisingly good motivator for working harder and getting in better shape).

But Garmin has a bazillion models and I never found any comparison page explaining the differences among the various hiking eTrex models. Eventually I worked it out:

Garmin eTrex models, decoded

C
Color display. This generally also implies USB connectivity instead of serial, just because the color models are newer.
H
High precision (a more sensitive satellite receiver).
x
Takes micro-SD cards. This may not be important for storing tracks and waypoints (you can store quite a long track with the built-in memory) but they mean that you can load extra base maps, like topographic data or other useful features.
Vista, Summit
These models have barometric altimeters and magnetic compasses. (I never did figure out the difference between a Vista and a Summit, except that in the color models (C), Vistas take micro-SD cards (x) while Summits don't, so there's a Summit C and HC while Vistas come in Cx and HCx. I don't know what the difference is between a monochrome Summit and Vista.)
Legend, Venture
These have no altimeter or compass. A Venture is a Legend that comes without the bundled extras like SD card, USB cable and base maps, so it's cheaper.

For me, the price/performance curve pointed to the Vista Cx.

Loading maps

Loading base maps was simplicity itself, and I found lots of howtos on how to use downloadable maps. Just mount the micro-SD card on any computer, make a directory called Garmin, and name the file gmapsupp.img. I used the CloudMade map for California, and it worked great. There are lots of howtos on generating your own maps, too, and I'm looking forward to making some with topographic data (which the CloudMade maps don't have). The most promising howtos I've found so far are the OSM Map On Garmin page on the OSM wiki and the much more difficult, but gorgeous, Hiking Biking Mapswiki page.

Uploading tracks and waypoints

But the real goal was to be able to take this toy out on a hike, then come back and upload the track and waypoint files.

I already knew, from the mapping party, that Garmins have an odd misfeature: you can connect them in usb-storage mode, where they look like an external disk and don't need any special software ... but then you can't upload any waypoints. (In fact, when I tried it with my Vista Cx I didn't even see the track file.) To upload tracks and waypoints, you need to use something that speaks Garmin protocol: namely, the excellent GPSBabel.

So far so good. How do you call GPSbabel? Luckily for me, just before my GPS arrived, Iván Sánchez Ortega posted a useful little gpsbabel script to the OSM newbies list and I thought I was all set.

But once I actually had the Vista in hand, complete with track and waypoints from a walk around the block, it turned out it wasn't quite that simple -- because Ubuntu didn't create the /dev/ttyUSB0 that Iván's script used. A web search found tons of people having that problem on Ubuntu and talking about various workarounds, involving making sure the garmin_usb driver is blacklisted in /etc/modprobe.d/blacklist (it was already), adding a /etc/udev/rules.d/45-garmin.rules file that changes permissions and ownership of ... um, I guess of the file that isn't being created? That didn't make much sense. Anyway, none of it helped.

But finally I found the fix: keep the garmin_usb driver blacklisted use "usb:" as the device to pass to GPSBabel rather than "/dev/ttyUSB0". So the commands are:

gpsbabel -t -i garmin -f usb: -o gpx -F tracks.gpx
gpsbabel -i garmin -f usb: -o gpx -F waypoints.gpx

Like so many other things, it's easy once you know the secret! Viewing tracklogs works great in Merkaartor, though I haven't yet found an app that does anything useful with the elevation data. I may have to write one.

Update: After I wrote this but before I was able to post it, a discussion on the OSM Newbies list with someone who was having similar troubles resulted in this useful wiki page: Garmin on GNU/Linux. It may also be worth checking the Discussion tab on that wiki page for further information.

Update, October 2011:
As of Debian Squeeze or Ubuntu Natty, you need two steps:

  1. Add a line to /etc/modprobe.d/blacklist.conf:
    blacklist garmin_gps
    
  2. Create a udev file, /etc/udev/rules.d/51-garmin.rules, to set the permissions so that you can access the device without being root. It contains the line:
    ATTRS{idVendor}=="091e", ATTRS{idProduct}=="0003", MODE="0660", GROUP="plugdev"
    

Then use gpsbabel with usb: and you should be fine.

Tags: , , , ,
[ 15:31 Jan 04, 2009    More mapping | permalink to this entry | comments ]

Syndicated on:
LinuxChix Live
Ubuntu Women
Women in Free Software
Graphics Planet
DevChix
Ubuntu California
Planet Openbox
Devchix
Planet LCA2009

Friends' Blogs:
Morris "Mojo" Jones
Jane Houston Jones
Dan Heller
Long Live the Village Green
Ups & Downs
DailyBBG

Other Blogs of Interest:
DevChix
Scott Adams
Dave Barry
BoingBoing

Powered by PyBlosxom.