Shallow Thoughts

Akkana's Musings on Open Source Computing and Technology, Science, and Nature.

Wed, 17 Aug 2016

Making New Map Tracks with Google Earth

A few days ago I wrote about track files in maps, specifically Translating track files between mapping formats. I promised to follow up with information on how to create new tracks.

For instance, I have some scans of old maps from the 60s and 70s showing the trails in the local neighborhood. There's no newer version. (In many cases, the trails have disappeared from lack of use -- no one knows where they're supposed to be even though they're legally trails where you're allowed to walk.) I wanted a way to turn trails from the old map into GPX tracks.

My first thought was to trace the old PDF map. A lot of web searching found a grand total of one page that talks about that: How to convert image of map into vector format?. It involves using GIMP to make an image containing just black lines on a white background, saving as uncompressed TIFF, then using a series of commands in GRASS. I made a start on that, but it was looking like it might be a big job that way. Since a lot of the old trails are still visible as faint traces in satellite photos, I decided to investigate tracing satellite photos in a map editor first, before trying the GRASS method.

But finding a working open source map editor turns out to be basically impossible. (Opportunity alert: it actually wouldn't be that hard to add that to PyTopo. Some day I'll try that, but now I was trying to solve a problem and hoping not to get sidetracked.)

The only open source map editor I've found is called Viking, and it's terrible. The user interface is complicated and poorly documented, and I could input only two or three trail segments before it crashed and I had to restart. Saving often, I did build up part of the trail network that way, but it was so slow and tedious restoring between crashes that I gave up.

OpenStreetMap has several editors available, and some of them are quite good, but they're (quite understandably) oriented toward defining roads that you're going to upload to the OpenStreetMap world map. I do that for real trails that I've walked myself, but it doesn't seem appropriate for historical paths between houses, some of which are now fenced off and few of which I've actually tried walking yet.

Editing a track in Google Earth

In the end, the only reasonable map editor I found was Google Earth -- free as in beer, not speech. It's actually quite a good track editor once I figured out how to use it -- the documentation is sketchy and no one who writes about it tells you the important parts, which were, for me:

Click on "My Places" in the sidebar before starting, assuming you'll want to keep these tracks around.

Right-click on My Places and choose Add->Folder if you're going to be creating more than one path. That way you can have a single KML file (Google Earth creates KML/KMZ, not GPX) with all your tracks together.

Move and zoom the map to where you can see the starting point for your path.

Click the "Add Path" button in the toolbar. This brings up a dialog where you can name the path and choose a color that will stand out against the map. Do not hit Return after typing the name -- that will immediately dismiss the dialog and take you out of path editing mode, leaving you with an empty named object in your sidebar. If you forget, like I kept doing, you'll have to right-click it and choose Properties to get back into editing mode.

Iconify, shade or do whatever your window manager allows to get that large, intrusive dialog out of the way of the map you're trying to edit. Shade worked well for me in Openbox.

Click on the starting point for your path. If you forgot to move the map so that this point is visible, you're out of luck: there's no way I've found to move the map at this point. (You might expect something like dragging with the middle mouse button, but you'd be wrong.) Do not in any circumstances be tempted to drag with the left button to move the map: this will draw lots of path points.

If you added points you don't want -- for instance, if you dragged on the map trying to move it -- Ctrl-Z doesn't undo, and there's no Undo in the menus, but Delete removes previous points. Whew.

Once you've started adding points, you can move the map using the arrow keys on your keyboard. And you can always zoom with the mousewheel.

When you finish one path, click OK in its properties dialog to end it.

Save periodically: click on the folder you created in My Places and choose Save Place As... Google Earth is a lot less crashy than Viking, but I have seen crashes.

When you're done for the day, be sure to File->Save->Save My Places. Google Earth apparently doesn't do this automatically; I was forever being confused why it didn't remember things I had done, and why every time I started it it would give me syntax errors on My Places saying it was about to correct the problem, then the next time I'd get the exact same error. Save My Places finally fixed that, so I guess it's something we're expected to do now and then in Google Earth.

Once I'd learned those tricks, the map-making went fairly quickly. I had intended only to trace a few trails then stop for the night, but when I realized I was more than halfway through I decided to push through, and ended up with a nice set of KML tracks which I converted to GPX and loaded onto my phone. Now I'm ready to explore.

[ 17:26 Aug 17, 2016    More mapping | permalink to this entry | comments ]

Sun, 14 Aug 2016

Translating track files between mapping formats

I use map tracks quite a bit. On my Android phone, I use OsmAnd, an excellent open-source mapping tool that can download map data generated from free OpenStreetMap, then display the maps offline, so I can use them in places where there's no cellphone signal (like nearly any hiking trail). At my computer, I never found a decent open-source mapping program, so I wrote my own, PyTopo, which downloads tiles from OpenStreetMap.

In OsmAnd, I record tracks from all my hikes, upload the GPX files, and view them in PyTopo. But it's nice to go the other way, too, and take tracks or waypoints from other people or from the web and view them in my own mapping programs, or use them to find them when hiking.

Translating between KML, KMZ and GPX

Both OsmAnd and PyTopo can show Garmin track files in the GPX format. PyTopo can also show KML and KMZ files, Google's more complicated mapping format, but OsmAnd can't. A lot of track files are distributed in Google formats, and I find I have to translate them fairly often -- for instance, lists of trails or lists of waypoints on a new hike I plan to do may be distributed as KML or KMZ.

The command-line gpsbabel program does a fine job translating KML to GPX. But I find its syntax hard to remember, so I wrote a shell alias:

kml2gpx () {
        gpsbabel -i kml -f $1 -o gpx -F $1:t:r.gpx
so I can just type kml2gpx file.kml and it will create a file.gpx for me.

More often, people distribute KMZ files, because they're smaller. They're just gzipped KML files, so the shell alias is only a little bit longer:

kmz2gpx () {
        gunzip -c $1 > $kmlfile
        gpsbabel -i kml -f $kmlfile -o gpx -F $kmlfile:t:r.gpx

Of course, if you ever have a need to go from GPX to KML, you can reverse the gpsbabel arguments appropriately; and if you need KMZ, run gzip afterward.

UTM coordinates

A couple of people I know use a different format, called UTM, which stands for Universal Transverse Mercator, for waypoints, and there are some secret lists of interesting local features passed around in that format.

It's a strange system. Instead of using latitude and longitude like most world mapping coordinate systems, UTM breaks the world into 60 longitudinal zones. UTM coordinates don't usually specify their zone (at least, none of the ones I've been given ever have), so if someone gives you a UTM coordinate, you need to know what zone you're in before you can translate it to a latitude and longitude. Then a pair of UTM coordinates specifies easting, and northing which tells you where you are inside the zone. Wikipedia has a map of UTM zones.

Note that UTM isn't a file format: it's just a way of specifying two (really three, if you count the zone) coordinates. So if you're given a list of UTM coordinate pairs, gpsbabel doesn't have a ready-made way to translate them into a GPX file. Fortunately, it allows a "universal CSV" (comma separated values) format, where the first line specifies which field goes where. So you can define a UTM UniCSV format that looks like this:

Trailhead,13,0395145,3966291,Trailhead on Buckman Rd
Sierra Club TH,13,0396210,3966597,Alternate trailhead in the arroyo
then translate it like this:
gpsbabel -i unicsv -f filename.csv -o gpx -F filename.gpx
I (and all the UTM coordinates I've had to deal with) are in zone 13, so that's what I used for that example and I hardwired that into my alias, but if you're near a zone boundary, you'll need to figure out which zone to use for each coordinate.

I also know someone who tends to send me single UTM coordinate pairs, because that's what she has her Garmin configured to show her. For instance, "We'll be using the trailhead at 0395145 3966291". This happened often enough, and I got tired of looking up the UTM UniCSV format every time, that I made another shell function just for that.

utm2gpx () {
        unicsv=`mktemp /tmp/point-XXXXX.csv` 
        echo "name,utm_z,utm_e,utm_n,comment" >> $unicsv
        printf "Point,13,%s,%s,point" $1 $2 >> $unicsv
        gpsbabel -i unicsv -f $unicsv -o gpx -F $gpxfile
        echo Created $gpxfile
So I can say utm2gpx 0395145 3966291, pasting the two coordinates from her email, and get a nice GPX file that I can push to my phone.

What if all you have is a printed map, or a scan of an old map from the pre-digital days? That's part 2, which I'll post in a few days.

Now posted: Making New Map Tracks with Google Earth.

[ 10:29 Aug 14, 2016    More mapping | permalink to this entry | comments ]

Tue, 09 Aug 2016

Double Rainbow, with Hummingbirds

A couple of days ago we had a spectacular afternoon double rainbow. I was out planting grama grass seeds, hoping to take take advantage of a rainy week, but I cut the planting short to run up and get my camera.

[Double rainbow]

[Hummingbirds and rainbow] And then after shooting rainbow shots with the fisheye lens, it occurred to me that I could switch to the zoom and take some hummingbird shots with the rainbow in the background. How often do you get a chance to do that? (Not to mention a great excuse not to go back to planting grass seeds.)

(Actually, here, it isn't all that uncommon since we get a lot of afternoon rainbows. But it's the first time I thought of trying it.)

Focus is always chancy when you're standing next to the feeder, waiting for birds to fly by and shooting whatever you can. Next time maybe I'll have time to set up a tripod and remote shutter release. But I was pretty happy with what I got.

Photos: Double rainbow, with hummingbirds.

Tags: , , ,
[ 19:40 Aug 09, 2016    More nature | permalink to this entry | comments ]

Sat, 06 Aug 2016

Adding a Back button in Python Webkit-GTK

I have a little browser script in Python, called quickbrowse, based on Python-Webkit-GTK. I use it for things like quickly calling up an anonymous window with full javascript and cookies, for when I hit a page that doesn't work with Firefox and privacy blocking; and as a quick solution for calling up HTML conversions of doc and pdf email attachments.

Python-webkit comes with a simple browser as an example -- on Debian it's installed in /usr/share/doc/python-webkit/examples/ But it's very minimal, and lacks important basic features like command-line arguments. One of those basic features I've been meaning to add is Back and Forward buttons.

Should be easy, right? Of course webkit has a go_back() method, so I just have to add a button and call that, right? Ha. It turned out to be a lot more difficult than I expected, and although I found a fair number of pages asking about it, I didn't find many working examples. So here's how to do it.

Add a toolbar button

In the WebToolbar class (derived from gtk.Toolbar): In __init__(), after initializing the parent class and before creating the location text entry (assuming you want your buttons left of the location bar), create the two buttons:

        backButton = gtk.ToolButton(gtk.STOCK_GO_BACK)
        backButton.connect("clicked", self.back_cb)
        self.insert(backButton, -1)

        forwardButton = gtk.ToolButton(gtk.STOCK_GO_FORWARD)
        forwardButton.connect("clicked", self.forward_cb)
        self.insert(forwardButton, -1)

Now create those callbacks you just referenced:

   def back_cb(self, w):

    def forward_cb(self, w):

That's right, you can't just call go_back on the web view, because GtkToolbar doesn't know anything about the window containing it. All it can do is pass signals up the chain.

But wait -- it can't even pass signals unless you define them. There's a __gsignals__ object defined at the beginning of the class that needs all its signals spelled out. In this case, what you need is

       "go-back-requested": (gobject.SIGNAL_RUN_FIRST,
                              gobject.TYPE_NONE, ()),
       "go-forward-requested": (gobject.SIGNAL_RUN_FIRST,
                              gobject.TYPE_NONE, ()),
Now these signals will bubble up to the window containing the toolbar.

Handle the signals in the containing window

So now you have to handle those signals in the window. In WebBrowserWindow (derived from gtk.Window), in __init__ after creating the toolbar:

        toolbar.connect("go-back-requested", self.go_back_requested_cb,
        toolbar.connect("go-forward-requested", self.go_forward_requested_cb,

And then of course you have to define those callbacks:

def go_back_requested_cb (self, widget, content_pane):
    # Oops! What goes here?
def go_forward_requested_cb (self, widget, content_pane):
    # Oops! What goes here?

But whoops! What do we put there? It turns out that WebBrowserWindow has no better idea than WebToolbar did of where its content is or how to tell it to go back or forward. What it does have is a ContentPane (derived from gtk.Notebook), which is basically just a container with no exposed methods that have anything to do with web browsing.

Get the BrowserView for the current tab

Fortunately we can fix that. In ContentPane, you can get the current page (meaning the current browser tab, in this case); and each page has a child, which turns out to be a BrowserView. So you can add this function to ContentPane to help other classes get the current BrowserView:

    def current_view(self):
        return self.get_nth_page(self.get_current_page()).get_child()

And now, using that, we can define those callbacks in WebBrowserWindow:

def go_back_requested_cb (self, widget, content_pane):
def go_forward_requested_cb (self, widget, content_pane):

Whew! That's a lot of steps for something I thought was going to be just adding two buttons and two callbacks.

Tags: , , ,
[ 16:45 Aug 06, 2016    More programming | permalink to this entry | comments ]

Tue, 05 Jul 2016

GIMP at Texas LinuxFest

I'll be at Texas LinuxFest in Austin, Texas this weekend. Friday, July 8 is the big day for open source imaging: first a morning Photo Walk led by Pat David, from 9-11, after which Pat, an active GIMP contributor and the driving force behind the PIXLS.US website and discussion forums, gives a talk on "Open Source Photography Tools". Then after lunch I'll give a GIMP tutorial. We may also have a Graphics Hackathon/Q&A session to discuss all the open-source graphics tools in the last slot of the day, but that part is still tentative. I'm hoping we can get some good discussion especially among the people who go on the photo walk.

Lots of interesting looking talks on Saturday, too. I've never been to Texas LinuxFest before: it's a short conference, just two days, but they're packing a lot into those two days and but it looks like it'll be a lot of fun.

Tags: , ,
[ 18:37 Jul 05, 2016    More conferences | permalink to this entry | comments ]

Sun, 03 Jul 2016

Midsummer Nature Notes from Traveling

A few unusual nature observations noticed over the last few weeks ...

First, on a trip to Washington DC a week ago (my first time there). For me, the big highlight of the trip was my first view of fireflies -- bright green ones, lighting once or twice then flying away, congregating over every park, lawn or patch of damp grass. What fun!

Predatory grackle


But the unusual observation was around mid-day, on the lawn near the Lincoln Memorial. A grackle caught my attention as it flashed by me -- a male common grackle, I think (at least, it was glossy black, relatively small and with only a moderately long tail).

It turned out it was chasing a sparrow, which was dodging and trying to evade, but unsuccessfully. The grackle made contact, and the sparrow faltered, started to flutter to the ground. But the sparrow recovered and took off in another direction, the grackle still hot on its tail. The grackle made contact again, and again the sparrow recovered and kept flying. But the third hit was harder than the other two, and the sparrow went down maybe fifteen or twenty feet away from me, with the grackle on top of it.

The grackle mantled over its prey like a hawk and looked like it was ready to begin eating. I still couldn't quite believe what I'd seen, so I stepped out toward the spot, figuring I'd scare the grackle away and I'd see if the sparrow was really dead. But the grackle had its eye on me, and before I'd taken three steps, it picked up the sparrow in its bill and flew off with it.

I never knew grackles were predatory, much less capable of killing other birds on the wing and flying off with them. But a web search on grackles killing birds got quite a few hits about grackles killing and eating house sparrows, so apparently it's not uncommon.

Daytime swarm of nighthawks

Then, on a road trip to visit friends in Colorado, we had to drive carefully past the eastern slope of San Antonio Mountain as a flock of birds wheeled and dove across the road. From a distance it looked like a flock of swallows, but as we got closer we realized they were far larger. They turned out to be nighthawks -- at least fifty of them, probably considerably more. I've heard of flocks of nighthawks swarming around the bugs attracted to parking lot streetlights. And I've seen a single nighthawk, or occasionally two, hawking in the evenings from my window at home. But I've never seen a flock of nighthawks during the day like this. An amazing sight as they swoop past, just feet from the car's windshield.

Flying ants

[Flying ant courtesy of Jen Macke]

Finally, the flying ants. The stuff of a bad science fiction movie! Well, maybe if the ants were 100 times larger. For now, just an interesting view of the natural world.

Just a few days ago, Jennifer Macke wrote a fascinating article in the PEEC Blog, "Ants Take Wing!" letting everyone know that this is the time of year for ants to grow wings and fly. (Jen also showed me some winged lawn ants in the PEEC ant colony when I was there the day before the article came out.) Both males and females grow wings; they mate in the air, and then the newly impregnated females fly off, find a location, shed their wings (leaving a wing scar you can see if you have a strong enough magnifying glass) and become the queen of a new ant colony.

And yesterday morning, as Dave and I looked out the window, we saw something swarming right below the garden. I grabbed a magnifying lens and rushed out to take a look at the ones emerging from the ground, and sure enough, they were ants. I saw only black ants. Our native harvester ants -- which I know to be common in our yard, since I've seen the telltale anthills surrounded by a large bare area where they clear out all vegetation -- have sexes of different colors (at least when they're flying): females are red, males are black. These flying ants were about the size of harvester ants but all the ants I saw were black. I retreated to the house and watched the flights with binoculars, hoping to see mating, but all the flyers I saw seemed intent on dispersing. Either these were not harvester ants, or the females come out at a different time from the males. Alas, we had an appointment and had to leave so I wasn't able to monitor them to check for red ants. But in a few days I'll be watching for ants that have lost their wings ... and if I find any, I'll try to identify queens.

Tags: , , ,
[ 09:28 Jul 03, 2016    More nature | permalink to this entry | comments ]

Sun, 26 Jun 2016

How to un-deny a host blocked by denyhosts

We had a little crisis Friday when our server suddenly stopped accepting ssh connections.

The problem turned out to be denyhosts, a program that looks for things like failed login attempts and blacklists IP addresses.

But why was our own IP blacklisted? It was apparently because I'd been experimenting with a program called mailsync, which used to be a useful program for synchronizing IMAP folders with local mail folders. But at least on Debian, it has broken in a fairly serious way, so that it makes three or four tries with the wrong password before it actually uses the right one that you've configured in .mailsync. These failed logins are a good way to get yourself blacklisted, and there doesn't seem to be any way to fix mailsync or the c-client library it uses under the covers.

Okay, so first, stop using mailsync. But then how to get our IP off the server's blacklist? Just editing /etc/hosts.deny didn't do it -- the IP reappeared there a few minutes later.

A web search found lots of solutions -- you have to edit a long list of files, but no two articles had the same file list. It appears that it's safest to remove the IP from every file in /var/lib/denyhosts.

So here are the step by step instructions.

First, shut off the denyhosts service:

service denyhosts stop

Go to /var/lib/denyhosts/ and grep for any file that includes your IP:

grep *

(If you aren't sure what your IP is as far as the outside world is concerned, Googling what's my IP will helpfully tell you, as well as giving you a list of other sites that will also tell you.)

Then edit each of these files in turn, removing your IP from them (it will probably be at the end of the file).

When you're done with that, you have one more file to edit: remove your IP from the end of /etc/hosts.deny

You may also want to add your IP to /etc/hosts.allow, but it may not make much difference, and if you're on a dynamic IP it might be a bad idea since that IP will eventually be used by someone else.

Finally, you're ready to re-start denyhosts:

service denyhosts start

Whew, un-blocked. And stay away from mailsync. I wish I knew of a program that actually worked to keep IMAP and mbox mailboxes in sync.

[ 12:59 Jun 26, 2016    More linux | permalink to this entry | comments ]

Sat, 18 Jun 2016

Cave 6" as a Quick-Look Scope

I haven't had a chance to do much astronomy since moving to New Mexico, despite the stunning dark skies. For one thing, those stunning dark skies are often covered with clouds -- New Mexico's dramatic skyscapes can go from clear to windy to cloudy to hail or thunderstorms and back to clear and hot over the course of a few hours. Gorgeous to watch, but distracting for astronomy, and particularly bad if you want to plan ahead and observe on a particular night. The Pajarito Astronomers' monthly star parties are often clouded or rained out, as was the PEEC Nature Center's moon-and-planets star party last week.

That sort of uncertainty means that the best bet is a so-called "quick-look scope": one that sits by the door, ready to be hauled out if the sky is clear and you have the urge. Usually that means some kind of tiny refractor; but it can also mean leaving a heavy mount permanently set up (with a cover to protect it from those thunderstorms) so it's easy to carry out a telescope tube and plunk it on the mount.

I have just that sort of scope sitting in our shed: an old, dusty Cave Astrola 6" Newtonian on an equatorian mount. My father got it for me on my 12th birthday. Where he got the money for such a princely gift -- we didn't have much in those days -- I never knew, but I cherished that telescope, and for years spent most of my nights in the backyard peering through the Los Angeles smog.

Eventually I hooked up with older astronomers (alas, my father had passed away) and cadged rides to star parties out in the Mojave desert. Fortunately for me, parenting standards back then allowed a lot more freedom, and my mother was a good judge of character and let me go. I wonder if there are any parents today who would let their daughter go off to the desert with a bunch of strange men? Even back then, she told me later, some of her friends ribbed her -- "Oh, 'astronomy'. Suuuuuure. They're probably all off doing drugs in the desert." I'm so lucky that my mom trusted me (and her own sense of the guys in the local astronomy club) more than her friends.

The Cave has followed me through quite a few moves, heavy, bulky and old fashioned as it is; even when I had scopes that were bigger, or more portable, I kept it for the sentimental value. But I hadn't actually set it up in years. Last week, I assembled the heavy mount and set it up on a clear spot in the yard. I dusted off the scope, cleaned the primary mirror and collimated everything, replaced the finder which had fallen out somewhere along the way, set it up ... and waited for a break in the clouds.

[Hyginus Rille by Michael Karrer] I'm happy to say that the optics are still excellent. As I write this (to be posted later), I just came in from beautiful views of Hyginus Rille and the Alpine Valley on the moon. On Jupiter the Great Red Spot was just rotating out. Mars, a couple of weeks before opposition, is still behind a cloud (yes, there are plenty of clouds). And now the clouds have covered the moon and Jupiter as well. Meanwhile, while I wait for a clear view of Mars, a bat makes frenetic passes overhead, and something in the junipers next to my observing spot is making rhythmic crunch, crunch, crunch sounds. A rabbit chewing something tough? Or just something rustling in the bushes?

I just went out again, and now the clouds have briefly uncovered Mars. It's the first good look I've had at the Red Planet in years. (Tiny achromatic refractors really don't do justice to tiny, bright objects.) Mars is the most difficult planet to observe: Dave liks to talk about needing to get your "Mars eyes" trained for each Mars opposition, since they only come every two years. But even without my "Mars eyes", I had no trouble seeing the North pole with dark Acidalia enveloping it, and, in the south, the sinuous chain of Sini Sabaeus, Meridiani, Margaritifer, and Mare Erythraeum. (I didn't identify any of these at the time; instead, I dusted off my sketch pad and sketched what I saw, then compared it with XEphem's Mars view afterward.)

I'm liking this new quick-look telescope -- not to mention the childhood memories it brings back.

Tags: ,
[ 08:53 Jun 18, 2016    More science/astro | permalink to this entry | comments ]