Shallow Thoughts : : Aug

Akkana's Musings on Open Source Computing and Technology, Science, and Nature.

Wed, 28 Aug 2013

Python scripts for Android

Python on Android. Wouldn't that make so many things so much easier?

I've known for a long time about SL4A, but when I read, a year or two ago, that Google officially disclaimed support for languages other than Java and C and didn't want their employees working on projects like SL4A, I decided it wasn't a good bet.

But recently I heard from someone who had just discovered SL4A and its Python support and talked about it like a going thing. I had an Android scripting problem I really wanted to solve, and decided it was time to take another look.

It turns out SL4A and its Python interpreter are still being maintained, and indeed, I was able to solve my problem that way. But the documentation was scanty at best. So here are some shortcuts.

Getting Python running on Android

How do you install it in the first place? Took me three or four tries: it turns out it's extremely picky about the order in which you do things, and the documentation doesn't warn you about that. Follow these steps:

  1. Enable "Unknown Sources" under Application settings if you haven't already.
  2. Download both sl4a_r6.apk and PythonForAndroid_r4.apk
  3. Install sl4a from the apk. Do not install Python yet.
  4. Find SL4A in Applications and run it. It will say "no matches found" (i.e. no scripts) but that's okay: the important thing is that it creates the directory where the scripts will live, /sdcard/sl4a/scripts, without which PythonForAndroid would fail to install.
  5. Install PythonForAndroid from the apk.
  6. Find Python for Android in Applications and run it. Tap Install. This will install the sample scripts, and you'll be ready to go.

Make a shortcut on the home screen:

You've written a script and it does what you want. But to run it, you have to run SL4A, choose the Python interpreter, scroll around to find the script, tap on it, and indicate whether or not you want to see the console. Way too many steps!

Turns out you can make a shortcut on the home screen to an SL4A script, like this: (thanks to this tip):

This will give you the familiar twin-snake Python icon on your home screen. There doesn't seem to be any way to change this to a different icon.

Wait, what about UI?

Well, that still seems to be a big hole in the whole SL4A model. You can write great scripts that print to the console. You can even do a few specialized things, like popup menus, messages (what the Python Android module calls makeToast()) and notifications. The test.py sample script is a great illustration of how to use all those features, plus a lot more.

But what if you want to show a window, put a few buttons in it, let the user control things? Nobody seems to have thought about that possibility. I mean, it's not "sorry, we haven't had time to implement this", it isn't even mentioned as something someone would want to do on an Android device. Boggle.

The only possibility I've found is that there is apparently a way to use Android's WebView class from Python. I have not tried this yet; when I do, I'll write it up separately.

WebView may not be the best way to do UI. I've spent many hours tearing my hair out over its limitations even when called from Java. But still, it's something. And one very interesting thing about it is that it provides an easy way to call up an HTML page, either local or remote, from an Android home screen icon. So that may be the best reason yet to check out SL4A.

Tags: , ,
[ 22:31 Aug 28, 2013    More programming | permalink to this entry | ]

Sat, 24 Aug 2013

A nifty shell redirection trick: process substitution

I love shell pipelines, and flatter myself that I'm pretty good at them. But a discussion last week on the Linuxchix Techtalk mailing list on finding added lines in a file turned up a terrific bash/zsh shell redirection trick I'd never seen before: 

join -v 2 <(sort A.txt) <(sort B.txt)

I've used backquotes, and their cognate $(), plenty. For instance, you can do things like PS1=$(hostname): or PS1=`hostname`: to set your prompt to the current hostname: the shell runs the hostname command, takes its output, and substitutes that output in place of the backquoted or parenthesized expression.

But I'd never seen that <(...) trick before, and immediately saw how useful it was. Backquotes or $() let you replace arguments to a command with a program's output -- they're great for generating short strings for programs that take all their arguments on the command line. But they're no good for programs that need to read a file, or several files. <(...) lets you take the output of a command and pass it to a program as though it was the contents of a file. And if you can do it more than once in the same command -- as in Little Girl's example -- that could be tremendously useful.

Playing with it to see if it really did what it looked like it did, and what other useful things I could do with it, I tried this (and it worked just fine): 

$ diff <(echo hello; echo there) <(echo hello; echo world)
2c2
< there
---
> world
It acts as though I had two files, which each have "hello" as their first line; but one has "there" as the second line, while the other has "world". And diff shows the difference. I don't think there's any way of doing anything like that with backquotes; you'd need to use temp files.

Of course, I wanted to read more about it -- how have I gone all these years without knowing about this? -- and it looks like I'm not the only one who didn't know about it. In fact, none of the pages I found on shell pipeline tricks even mentioned it.

It turns out it's called "process substitution" and I found it documented in Chapter 23 of the Advanced Bash-Scripting Guide.

I tweeted it, and a friend who is a zsh master gave me some similar cool tricks. For instance, in zsh echo hi > >(cat) > >(cat -n) lets you pipe the output of a command to more than one other command.

That's zsh, but in bash (or zsh too, of course), you can use >() and tee to do the same thing: echo hi | tee >(cat) | cat -n

If you want a temp file to be created automatically, one you can both read and write, you can use =(foo) (zsh only?)

Great stuff! Some other pages that discuss some of these tricks:

Tags: , , ,
[ 19:23 Aug 24, 2013    More linux/cmdline | permalink to this entry | ]

Tue, 20 Aug 2013

Using Google Maps with Python to turn a list of addresses into waypoints

A few days ago I wrote about how I use making waypoint files for a list of house addresses is OsmAnd. For waypoint files, you need latitude/longitude coordinates, and I was getting those from a web page that used the online Google Maps API to convert an address into latitude and longitude coordinates.

It was pretty cool at first, but pasting every address into the latitude/longitude web page and then pasting the resulting coordinates into the address file, got old, fast. That's exactly the sort of repetitive task that computers are supposed to handle for us.

The lat/lon page used Javascript and the Google Maps API. and I already had a Google Maps API key (they have all sorts of fun APIs for map geeks) ... but I really wanted something that could run locally, reading and converting a local file.

And then I discovered the Python googlemaps package. Exactly what I needed! It's in the Python Package Index, so I installed it with pip install googlemaps. That enabled me to change my waymaker Python script: if the first line of a description wasn't a latitude and longitude, instead it looked for something that might be an address.

Addresses in my data files might be one line or might be two, but since they're all US addresses, I know they'll end with a two-capital-letter state abbreviation and a 5-digit zip code: 2948 W Main St. Anytown, NM 12345. You can find that with a regular expression: 

    match = re.search('.*[A-Z]{2}\s+\d{5}$', line)

But first I needed to check whether the first line of the entry was already latitude/longitude coordinates, since I'd already converted some of my files. That uses another regular expression. Python doesn't seem to have a built-in way to search for generic numeric expressions (containing digits, decimal points or +/- symbols) so I made one, since I had to use it twice if I was searching for two numbers with whitespace between them. 

    numeric = '[\+\-\d\.]'
    match = re.search('^(%s+)\s+(%s+)$' % (numeric, numeric),
                      line)
(For anyone who wants to quibble, I know the regular expression isn't perfect. For instance, it would match expressions like 23+48..6.1-64.5. Not likely to be a problem in these files, so I didn't tune it further.)

If the script doesn't find coordinates but does find something that looks like an address, it feeds the address into Google Maps and gets the resulting coordinates. That code looks like this: 

from googlemaps import GoogleMaps

gmaps = GoogleMaps('YOUR GOOGLE MAPS API KEY HERE')
try:
    lat, lon = gmaps.address_to_latlng(addr)
except googlemaps.GoogleMapsError, e:
    print "Oh, no! Couldn't geocode", addr
    print e

Overall, a nice simple solution made possible with python-googlemaps. The full script is on github: waymaker.

Tags: , , , , , , ,
[ 12:24 Aug 20, 2013    More mapping | permalink to this entry | ]

Sun, 18 Aug 2013

Nominated for FOSS Force Blog Competition

Shallow Thoughts has been nominated as a competitor in round two of the Foss Force Best Personal Linux or FOSS Blog Competition. There are plenty of excellent blogs on the list and I'm flattered to be included.

If you have a moment, take a look and vote for your favorite (whether or not it's Shallow Thoughts). You can vote for up to two. If nothing else, it's a good excuse to check out some excellent articles on free software from a variety of writers. Voting ends on Monday.

Tags:
[ 12:05 Aug 18, 2013    More blogging | permalink to this entry | ]

Learning to Sing

[House finch] I was awakened at 6:30 this morning by what sounded like a young house finch learning to sing, just outside my window. It got me thinking.

Every fall, songbirds which have stopped singing during high summer start up again, briefly, to sing for a few weeks before weather gets cold. A discussion several years ago on a local birding list concluded that nobody knows for sure why birds sing in autumn -- are they confused about the weather and think it's spring again, hoping for a last fling before the cold weather sets in, or what? There's a a wonderful ditty about it, "The Autumnal Recrudescence of the Amatory Urge", apparently written in the 1970s by Susan Stiles.

It's too early in the year right now for autumnal anything -- it's still quite warm. But lying there in bed listening to the exploratory notes of a bird clearly not yet confident in his song, I got to thinking about how birds learn their songs.

In most birds it's not innate: young male birds learn singing while still nestlings from listening to their father sing, much like human babies learn the rhythms of their native language from hearing their parents talk; and if you raise a songbird in a nest of another species, they will often learn the wrong song, or end up with some hybrid song that doesn't attract females of either species. (A good overview: The Development of Birdsong on Nature.) More recently, there have been all sorts of interesting studies on how young birds learn their local dialect, since a species' song varies quite a bit from one location to another.

But ... not all birds sing much once the eggs are laid, do they? They sing their hearts out while acquiring a territory and trying to attract a female; but once nesting starts, I don't remember hearing much activity from the house finches. Mockingbirds are an exception: I've seen mockers singing day and night even after they're feeding nestlings, though not all male mockers are quite so industrious. But I thought most species stopped singing much once the nest was built and eggs laid.

But if that's true, when do the young males learn their songs? Even if the father does sing a little, off and on, while the nestlings are being raised, that's not very much time to learn. Suppose the adults started singing again in the fall before the family disperses. Wouldn't that be an advantage to the young males who are just learning their songs? If a fledgling, off the nest and mostly able to care for himself, is "babbling", trying exploratory notes while learning what sounds he can make, wouldn't it be helpful to have a few nearby males who occasionally burst into song even if it's out of season?

Maybe the "Autumnal Recrudescence" isn't birds being confused about the weather at all. Maybe it's an evolutionary aid to help the young birds crystallize their songs before heading into their first winter. By singing in autumn, the males help their sons crystallize their songs for the next year, which helps the sons be more successful when it's time to look for a mate next spring.

Just a theory ... but I think it makes some sense, and I'll be listening to this autumn's chorus with new interest.

Tags: ,
[ 11:57 Aug 18, 2013    More nature/birds | permalink to this entry | ]

Fri, 16 Aug 2013

Offline mapping with lists of waypoints

Dave and I have been doing some exploratory househunting trips, and one of the challenges is how to maintain a list of houses and navigate from location to location. It's basically like geocaching, navigating from one known location to the next.

Sure, there are smartphone apps to do things like "show houses for sale near here" against a Google Maps background. But we didn't want everything, just the few gems we'd picked out ahead of time. And some of the places we're looking are fairly remote -- you can't always count on a consistent signal everywhere as you drive around, let alone a connection fast enough to download map tiles.

Fortunately, I use a wonderful open-source Android program called OsmAnd. It's the best, bar none, at offline mapping: download data files prepared from OpenStreetMap vector data, and you're good to go, even into remote areas with no network connectivity. It's saved our butts more than once exploring remote dirt tracks in the Mojave. And since the maps come from OpenStreetMap, if you find anything wrong with the map, you can fix it.

So the map part is taken care of. What about that list of houses?

Making waypoint files

On the other hand, one of OsmAnd's many cool features is that it can show track logs. I can upload a GPX file from my Garmin, or record a track within OsmAnd, and display the track on OsmAnd's map.

GPX track files can include waypoints. What if I made a GPX file consisting only of waypoints and descriptions for each house?

My husband was already making text files of potentially interesting houses: 

404 E David Dr 
Flagstaff, AZ 86001
$355,000
3 Bed 2 Bath
1,673 Sq Ft
0.23 acres
http://blahblah/long_url

2948 W Wilson Dr 
Flagstaff, AZ 86001
$285,000
3 Bed 2 Bath
1,908 Sq Ft
8,000 Sq Ft Lot 
http://blahblah/long_url

... (and so on)
So I just needed to turn those into GPX.

GPX is a fairly straightforward XML format -- I've parsed GPX files for pytopo and for ellie, and generating them from Python should be easier than parsing. But first I needed latitude and longitude coordinates. A quick web search solved that: an excellent page called Find latitude and longitude with Google Maps. You paste the address in and it shows you the location on a map along with latitude and longitude. Thanks to Bernard Vatant at Mondeca!

For each house, I copied the coordinates directly from the page and pasted them into the file. (Though that got old after about the fifth house; I'll write about automating that step in a separate article.)

Then I wrote a script called waymaker that parses a file of coordinates and descriptions and makes waypoint files. Run it like this: waymaker infile.txt outfile.gpx and it will create (or overwrite) a gpx file consisting of those waypoints.

Getting it into OsmAnd

I plugged my Android device into my computer's USB port, mounted it as usb-storage and copied all the GPX files into osmand/tracks (I had to create the tracks subdirectory myself, since I hadn't recorded any tracks. After restarting OsmAnd, it was able to see all the waypoint files.

OsmAnd has a couple of other ways of showing points besides track files. "Favorites" lets you mark a point on the map and save it to various Favorites categories. But although there's a file named favorites.gpx, changes you make to it never show up in the program. Apparently they're cached somewhere else. "POI" (short for Points of Interest) can be uploaded, but only as a .obf OsmAnd file or a .sqlitedb database, and there isn't much documentation on how to create either one. GPX tracks seemed like the easiest solution, and I've been happy with them so far.

Update: I asked on the osmand mailing list; it turns out that on the Favorites screen (Define View, then Favorites) there's a Refresh button that makes osmand re-read favorites.gpx. Works great. It uses pretty much the same format as track files -- I took <wpt></wpt> sequences I'd generated with waymaker and added them to my existing favorites.gpx file, adding appropriate categories. It's nice to have two different ways to display and categorize waypoints within the app.

Using waypoints in OsmAnd

How do you view these waypoints once they're loaded? When you're in OsmAnd's map view, tap the menu button and choose Define View, then GPX track... You'll see a list of all your GPX files; choose the one you want.

You'll be taken back to the map view, at a location and zoom level that shows all your waypoints. Don't panic if you don't see them immediately; sometimes I needed to scroll and zoom around a little before OsmAnd noticed there were waypoints and started drawing them.

Then you can navigate in the usual way. When you get to a waypoint, tap on it to see the description brieftly -- I was happy to find that multiple line descriptions work just fine. Or long-press on it to pop up a persistent description window that will stay up until you dismiss it.

It worked beautifully for our trip, both for houses and for other things like motels and points of interest along the way.

Tags: , , , , , ,
[ 15:58 Aug 16, 2013    More mapping | permalink to this entry | ]

Sun, 11 Aug 2013

GPIO tutorial for the BeagleBone Black

Want to get started controlling hardware from your BeagleBone Black? I've found a lot of the documentation and tutorials a little sketchy, so here's what I hope is a quick start guide.

I used the Adafruit Python GPIO library for my initial hacking. It's easy to set up: once you have your network set up, run these commands: 

opkg update && opkg install python-pip python-setuptools python-smbus
pip install Adafruit_BBIO

Pinout diagrams

First, where can you plug things in? The BBB has two huge header blocks, P8 and P9, but trying to find pinout diagrams for them is a problem. Don't blindly trust any diagram you find on the net; compare it against several others, and you may find there are big differences. I've found a lot of mislabeled BBB diagrams out there.

The best I've found so far are the two tables at elinux.org/BeagleBone. No pictures, but the tables are fairly readable and seem to be correct.

The official BeagleBone Black hardware manual is the reference you're actually supposed to use. It's a 121-page PDF full of incomprehensible and unexplained abbreviations. Good luck! The pin tables for P8 and P9 are on pp. 80 and 82. P8 and P8 are identified on p. 78.

Blinking an LED: basic GPIO output

For basic GPIO output, you have a wide choice of pins. Use the tables to identify power and ground, then pick a GPIO pin that doesn't seem to have too many other uses.

The Adafruit library can identify pins either by their location on the P8 and P9 headers, e.g. "P9_11", or by GPIO number, e.g. "GPIO0_26". Except -- with the latter designation, what's that extra zero between GPIO and _26? Is it always 0? Adafruit doesn't explain it. So for now I'm sticking to the PN_NN format.

I plugged my LED and resistor into ground (there are lots of ground terminals -- I used pin 0 on the P9 header) and pin 11 on P9. It's one line to enable it, and then you can turn it on and off: 

import Adafruit_BBIO.GPIO as GPIO

GPIO.setup("P9_11", GPIO.OUT)
GPIO.output("P9_11", GPIO.HIGH)
GPIO.output("P9_11", GPIO.LOW)
GPIO.output("P9_11", GPIO.HIGH)
Or make it blink: 
import time
while True:
    GPIO.output("P9_11", GPIO.HIGH)
    time.sleep(.5)
    GPIO.output("P9_11", GPIO.LOW)
    time.sleep(.5)

Fading an LED: PWM output

PWM is harder. Mostly because it's not easy to find out which pins can be used for GPIO. All the promotional literature on the BBB says it has 8 GPIO outputs -- but which ones are those?

If you spend half an hour searching for "pwm" in that long PDF manual and collecting a list of pins with "pwm" in their description, you'll find 13 of them on P9 and 12 on P8. So that's no help.

After comparing a bunch of references and cross-checking pin numbers against the descriptions in the hardware manual, I think this is the list: P9_14 P9_16 P9_21 P9_22 P9_28 P9_31 P8_13 P8_19

I haven't actually verified all of them yet, though.

Once you've found a pin that works for PWM, the rest is easy. Start it and set an initial frequency with PWM.start(pin, freq), and then you can change the duty cycle with set_duty_cycle(pin, cycle) where cycle is a number between 0 and 100. The duty cycle is the reverse of what you might expect: if you have an LED plugged in, a duty cycle of 0 will be brightest, 100 will be dimmest.

You can also change the frequency with PWM.set_frequency(pin, freq). I'm guessing freq is in Hertz, but they don't actually say.

When you're done, you should call PWM.stop() and PWM.cleanup().

Here's how to fade an LED from dim to bright ten times: 

import Adafruit_BBIO.PWM as PWM

PWM.start("P9_14", 50)

for j in range(10):
    for i in range(100, 0, -1):
        PWM.set_duty_cycle("P9_14", i)
        time.sleep(.02)

PWM.stop("P9_14")
PWM.cleanup()

Tags: , , ,
[ 13:36 Aug 11, 2013    More hardware | permalink to this entry | ]