Shallow Thoughts : tags : android

PyTopo can Show GPS from Image Files Now (30DayMapChallenge #15, My Data)

For Day 15 of the 30 Day Map Challenge, "My Data", I'm highlighting a feature I added to PyTopo last week: the ability to read GPS tags in image files.

JPEG, and probably other image formats as well, lets you store GPS coordinates inside the EXIF (EXchangeable Image File format) metadata stored within each image file.

Read more ...

Android Studio Workarounds, and Command-Line Gradle Builds

I got a new phone. (Not something that happens often.)

Fun, right? Well, partly, but also something I'd been dreading. I had a feeling that my ancient RSS reader, FeedViewer, which I use daily to read all my news feeds, probably wouldn't work under a modern Android (I wrote it for KitKat and it was last updated under Marshmallow). And that was correct.

Read more ...

Create a signed app with Cordova

I wrote last week about developing apps with PhoneGap/Cordova. But one thing I didn't cover. When you type cordova build, you're building only a debug version of your app. If you want to release it, you have to sign it. Figuring out how turned out to be a little tricky.

Most pages on the web say you can sign your apps by creating platforms/android/ant.properties with the same keystore information in it that you'd put in an ant build, then running cordova build android --release

But Cordova completely ignored my ant.properties file and went on creating a debug .apk file and no signed one.

I found various other purported solutions on the web, like creating a build.json file in the app's top-level directory ... but that just made Cordova die with a syntax error inside one of its own files). This is the only method that worked for me:

Create a file called platforms/android/release-signing.properties, and put this in it:

storeFile=/path/to/your-keystore.keystore
storeType=jks
keyAlias=some-key
// if you don't want to enter the password at every build, use this:
keyPassword=your-key-password
storePassword=your-store-password

Then cordova build android --release finally works, and creates a file called platforms/android/build/outputs/apk/android-release.apk

Cross-Platform Android Development Toolkits: Kivy vs. PhoneGap / Cordova

Although Ant builds have made Android development much easier, I've long been curious about the cross-platform phone development apps: you write a simple app in some common language, like HTML or Python, then run something that can turn it into apps on multiple mobile platforms, like Android, iOS, Blackberry, Windows phone, UbuntoOS, FirefoxOS or Tizen.

Last week I tried two of the many cross-platform mobile frameworks: Kivy and PhoneGap.

Kivy lets you develop in Python, which sounded like a big plus. I went to a Kivy talk at PyCon a year ago and it looked pretty interesting. PhoneGap takes web apps written in HTML, CSS and Javascript and packages them like native applications. PhoneGap seems much more popular, but I wanted to see how it and Kivy compared. Both projects are free, open source software.

If you want to skip the gory details, skip to the summary: how do Kivy and PhoneGap compare?

PhoneGap

I tried PhoneGap first. It's based on Node.js, so the first step was installing that. Debian has packages for nodejs, so apt-get install nodejs npm nodejs-legacy did the trick. You need nodejs-legacy to get the "node" command, which you'll need for installing PhoneGap.

Now comes a confusing part. You'll be using npm to install ... something. But depending on which tutorial you're following, it may tell you to install and use either phonegap or cordova.

Cordova is an Apache project which is intertwined with PhoneGap. After reading all their FAQs on the subject, I'm as confused as ever about where PhoneGap ends and Cordova begins, which one is newer, which one is more open-source, whether I should say I'm developing in PhoneGap or Cordova, or even whether I should be asking questions on the #phonegap or #cordova channels on Freenode. (The one question I had, which came up later in the process, I asked on #phonegap and got a helpful answer very quickly.) Neither one is packaged in Debian.

After some searching for a good, comprehensive tutorial, I ended up on a The Cordova tutorial rather than a PhoneGap one. So I typed:

sudo npm install -g cordova

Once it's installed, you can create a new app, add the android platform (assuming you already have android development tools installed) and build your new app:

cordova create hello com.example.hello HelloWorld
cordova platform add android
cordova build

Oops!

Error: Please install Android target: "android-22"

So I fired up the Android SDK manager, selected android-22 for install, accepted the license ... and waited ... and waited. In the end it took over two hours to download the android-22 SDK; the system image is 13Gb! So that's a bit of a strike against PhoneGap.

While I was waiting for android-22 to download, I took a look at Kivy.

Kivy

As a Python enthusiast, I wanted to like Kivy best. Plus, it's in the Debian repositories: I installed it with sudo apt-get install python-kivy python-kivy-examples

They have a nice quickstart tutorial for writing a Hello World app on their site. You write it, run it locally in python to bring up a window and see what the app will look like. But then the tutorial immediately jumps into more advanced programming without telling you how to build and deploy your Hello World. For Android, that information is in the Android Packaging Guide. They recommend an app called Buildozer (cute name), which you have to pull from git, build and install.

buildozer init
buildozer android debug deploy run

The file buildozer.spec in your project's directory lets you uncomment and customize variables like:

# (int) Android SDK version to use
android.sdk = 21

# (str) Android NDK directory (if empty, it will be automatically downloaded.)
# android.ndk_path = 

# (str) Android SDK directory (if empty, it will be automatically downloaded.)
# android.sdk_path = 

Unlike a lot of Android build packages, buildozer will not inherit variables like ANDROID_SDK, ANDROID_NDK and ANDROID_HOME from your environment; you must edit buildozer.spec.

But that doesn't help with ant. Fortunately, when I inspected the Python code for buildozer itself, I discovered there was another variable that isn't mentioned in the default spec file. Just add this line:

android.ant_path = /usr/bin

Next, buildozer gave me a slew of compilation errors:

kivy/graphics/opengl.c: No such file or directory
 ... many many more lines of compilation interspersed with errors
kivy/graphics/vbo.c:1:2: error: #error Do not use this file, it is the result of a failed Cython compilation.

I had to ask on #kivy to solve that one. It turns out that the current version of cython, 0.22, doesn't work with kivy stable. My choices were to uninstall kivy and pull the development version from git, or to uninstall cython and install version 0.21.2 via pip. I opted for the latter option. Either way, there's no "make clean", so removing the dist and build directories let me start over with the new cython.

apt-get purge cython
sudo pip install Cython==0.21.2
rm -rf ./.buildozer/android/platform/python-for-android/dist
rm -rf ./.buildozer/android/platform/python-for-android/build

Buildozer was now happy, and proceeded to download and build Python-2.7.2, pygame and a large collection of other Python libraries for the ARM platform. Apparently each app packages the Python language and all libraries it needs into the Android .apk file.

Eventually I ran into trouble because I'd named my python file hello.py instead of main.py; apparently this is something you're not allowed to change and they don't mention it in the docs, but that was easily solved. Then I ran into trouble again:

Exception: Unable to find capture version in ./main.py (looking for `__version__ = ['"](.*)['"]`)

The .apk file it created was quite large because of all the embedded Python libraries: for the little 77-line pong demo, /usr/share/kivy-examples/tutorials/pong in the Debian kivy-examples package, the apk came out 7.3Mb. For comparison, my FeedViewer native java app, roughly 2000 lines of Java plus a few XML files, produces a 44k apk.

The next step was to make a real mini app. But when I looked through the Kivy examples, they all seemed highly specialized, and I couldn't find any documentation that addressed issues like what widgets were available or how to lay them out. How do I add a basic text widget? How do I put a button next to it? How do I get the app to launch in portrait rather than landscape mode? Is there any way to speed up the very slow initialization?

I'd spent a few hours on Kivy and made a Hello World app, but I was having trouble figuring out how to do anything more. I needed a change of scenery.

PhoneGap, redux

By this time, android-22 had finally finished downloading. I was ready to try PhoneGap again.

This time,

cordova platforms add android
cordova build

Cordova eventually produced a 1.8Mb "hello world" apk -- a quarter the size of the Kivy package, though 20 times as big as a native Java app. Deployed on Android, it initialized much faster than the Kivy app, and came up in portrait mode but rotated correctly if I rotated the phone.

Editing the HTML, CSS and Javascript was fairly simple. You'll want to replace pretty much all of the default CSS if you don't want your app monopolized by the Cordova icon.

The only tricky part was file access: opening a file:// URL didn't work. I asked on #phonegap and someone helpfully told me I'd need the file plugin. That was easy to find in the documentation, and I added it like this:

cordova plugin search file
cordova plugin add org.apache.cordova.file

My final apk, for a small web app I use regularly on Android, was almost the same size as their hello world example: 1.8Mb. And it works great: phonegap had no problem playing an audio clip, something that was tricky when I was trying to do the same thing from a native Android java WebView class.

Summary: How do Kivy and PhoneGap compare?

This has been a long article, I know. So how do Kivy and PhoneGap compare, and which one will I be using?

They both need a large amount of disk space for the development environment. I wish I had good numbers to give you, but I was working with both systems at the same time, and their packages are scattered all over the disk so I haven't found a good way of measuring their size. I suspect PhoneGap is quite a bit bigger, because it uses gradle rather than ant and because it insists on android-22.

On the other hand, PhoneGap wins big on packaged application size: its .apk files are a quarter the size of Kivy's.

PhoneGap definitely wins on documentation. Kivy has seemingly lots of documentation, but its tutorials jumped around rather than following a logical sequence, and I had trouble finding answers to basic questions like "How do I display a text field with a button?" PhoneGap doesn't need that, because the UI is basic HTML and CSS -- limited though they are, at least most people know how to use them.

Finally, PhoneGap wins on startup speed. For my very simple test app, startup was more or less immediate, while the Kivy Hello World app required several seconds of startup time on my Galaxy S4.

Kivy is an interesting project. I like the ant-based build, the straightforward .spec file, and of course the Python language. But it still has some catching up to do in performance and documentation. For throwing together a simple app and packaging it for Android, I have to give the win to PhoneGap.

Command-line builds for Android using ant

I recently needed to update an old Android app that I hadn't touched in years. My Eclipse setup is way out of date, and I've been hearing about more and more projects switching to using command-line builds. I wanted to ditch my fiddly, difficult to install Eclipse setup and switch to something easier to use.

Some of the big open-source packages, like OsmAnd, have switched to gradle for their Java builds. So I tried to install gradle -- and on Debian, apt-get install gradle wanted to pull in a total of 153 packages! Maybe gradle wasn't the best option to pursue.

But there's another option for command-line android builds: ant. When I tried apt-get install ant, since I already have Java installed (I think the relevant package is openjdk-7-jdk), it installed without needing a single additional package. For a small program, that's clearly a better way to go!

Then I needed to create a build directory and move my project into it. That turned out to be fairly easy, too -- certainly compared to the hours it spent setting up an Eclipse environment. Here's how to set up your ant Android build:

First install the Android "Stand-alone SDK Tools" from Installing the Android SDK. This requires a fair amount of clicking around, accepting licenses, and waiting for a long download.

Now install an SDK or two. Use android sdk to install new SDK versions, and android list targets to see what versions you have installed.

Create a new directory for your project, cd into it, and then:

android create project --name YourProject --path . --target android-19 --package tld.yourdomain.YourProject --activity YourProject

When this is done, type ant with no arguments to make sure the directory structure was created properly. If it doesn't print errors, that's a good sign.

Check that local.properties has sdk.dir set correctly. It should have picked that up from your environment.

There will be a stub source file in src/tld/yourdomain/YourProject.java. Edit it as needed, or, if you're transferring a project from another build system such as eclipse, copy the existing .java files to that directory.

If you have custom icons for your project, or other resources like layout or menu files, put them in the appropriate directories under res. The directory structure is the same as in eclipse, but unlike an eclipse build, you can edit the files at any time without the build mysteriously breaking.

Signing your app

Now you'll need a key to sign your app. Eclipse generates a debugging key automatically, but ant doesn't. It's better to use a real key anyway, since debugging keys expire and need to be regenerated periodically.

If you don't already have a key, generate one with:

keytool -genkey -v -keystore my-key.keystore -alias mykey -keyalg RSA -sigalg SHA1withRSA -keysize 2048 -validity 10000

Now create a file called ant.properties containing these two lines:

key.store=/path/to/my-key.keystore
key.alias=mykey

If you forget your key alias, you can find out with this command and the password:

keytool -list -keystore /path/to/my-key.keystore

Optionally, you can also include your key's password:

key.store.password=xxxx
key.alias.password=xxxx

Building

Finally, you're ready to build!

ant release

If you get an error like:

AndroidManifest.xml:6: error: Error: No resource found that matches the given name (at 'icon' with value '@drawable/ic_launcher').

After ant release, your binary will be in bin/YourProject-release.apk. If you have an adb connection, you can (re)install it with: adb install -r bin/YourProject-release.apk

Done! So much easier than eclipse, and you can use any editor you want, and check your files into any version control system.

That just leaves the coding part. If only Java development were as easy as Python or C ...

Tips for passing Google's "Mobile Friendly" tests

I saw on Slashdot that Google is going to start down-rating sites that don't meet its criteria of "mobile-friendly": Are you ready for Google's 'Mobilegeddon' on Tuesday?. And from the the Slashdot discussion, it was pretty clear that Google's definition included some arbitrary hoops to jump through.

So I headed over to Google's Mobile-friendly test to check out some of my pages.

Now, most of my website seemed to me like it ought to be pretty mobile friendly. It's size agnostic: I don't specify any arbitrary page widths in pixels, so most of my pages can resize down as far as necessary (I was under the impression that was what "responsive design" meant for websites, though I've been doing it for many years and it seems now that "responsive design" includes a whole lot of phone-specific tweaks and elaborate CSS for moving things around based on size.) I also don't set font sizes that might make the page less accessible to someone with vision problems -- or to someone on a small screen with high pixel density. So I was pretty confident.

I shouldn't have been. Basically all of my pages failed. And in chasing down some of the problems I've learned a bit about Google's mobile rules, as well as about some weird quirks in how current mobile browsers render websites.

Basically, all of my pages failed with the same three errors:

• Text too small to read
• Links too close together
• Mobile viewport not set

What? I wasn't specifying text size at all -- if the text is too small to read with the default font, surely that's a bug in the mobile browser, not a bug in my website. Same with links too close together, when I'm using the browser's default line spacing.

But it turned out that the first two points were meaningless. They were just a side effect of that third error: the mobile viewport.

The mandatory meta viewport tag

It turns out that any page that doesn't add a new meta tag, called "viewport", will automatically fail Google's mobile friendly test and be downranked accordingly. What's that all about?

Apparently it's originally Apple's fault. iPhones, by default, pretend their screen is 980 pixels wide instead of the actual 320 or 640, and render content accordingly, and so they shrink everything down by a factor of 3 (980/320). They do this assuming that most website designers will set a hard limit of 980 pixels (which I've always considered to be bad design) ... and further assuming that their users care more about seeing the beautiful layout of a website than about reading the website's text.

And Google apparently felt, at some point during the Android development process, that they should copy Apple in this silly behavior. I'm not sure when Android started doing this; my Android 2.3 Samsung doesn't do it, so it must have happened later than that.

Anyway, after implementing this, Apple then introduced a meta tag you can add to an HTML file to tell iPhone browsers not to do this scaling, and to display the text at normal text size. There are various forms for this tag, but the most common is:

<meta name="viewport" content="width=device-width, initial-scale=1">

Just to be clear, Google is telling us that in order not to have our pages downgraded, we have to add a new tag to every page on the web to tell mobile browsers not to do something silly that they shouldn't have been doing in the first place, and which Google implemented to copy a crazy thing Apple was doing.

How width and initial-scale relate

Documentation on how width and initial-scale relate to each other, and which takes precedence, are scant. Apple's documentation on the meta viewport tag says that setting initial-scale=1 automatically sets width=device-width. That implies that the two are basically equivalent: that they're only different if you want to do something else, like set a page width in pixels (use width=) or set the width to some ratio of the device width other than 1 (use initial-scale=.

That means that using initial-scale=1 should imply width=device-width -- yet nearly everyone on the web seems to use both. So I'm doing that, too. Apparently there was once a point to it: some older iPhones had a bug involving switching orientation to landscape mode, and specifying both initial-scale=1 and width=device-width helped, but supposedly that's long since been fixed.

initial-scale=2, by the way, sets the viewport to half what it would have been otherwise; so if the width would have been 320, it sets it to 160, so you'll see half as much. Why you'd want to set initial-scale to anything besides 1 in a web page, I don't know.

If the width specified by initial-scale conflicts with that specified by width, supposedly iOS browsers will take the larger of the two, while Android won't accept a width directive less than 320, according to Quirks mode: testing Meta viewport.

It would be lovely to be able to test this stuff; but my only Android device is running Android 2.3, which doesn't do all this silly zooming out. It does what a sensible small-screen device should do: it shows text at normal, readable size by default, and lets you zoom in or out if you need to.

(Only marginally related, but interesting if you're doing elaborate stylesheets that take device resolution into account, is A List Apart's discussion, A Pixel Identity Crisis.)

Control width of images

Once I added meta viewport tags, most of my pages passed the test. But I was seeing something else on some of my photo pages, as well as blog pages where I have inline images:

• Content wider than screen
• Links too close together

Image pages are all about showing an image. Many of my images are wider than 320 pixels ... and thus get flagged as too wide for the screen. Note the scrollbars, and how you can only see a fraction of the image.

There's a simple way to fix this, and unlike the meta viewport thing, it actually makes sense. The solution is to force images to be no wider than the screen with this little piece of CSS:

<style type="text/css">
  img { max-width: 100%; height: auto; }
</style>

I've been using similar CSS in my RSS reader for several months, and I know how much better it made the web, on news sites that insist on using 1600 pixel wide images inline in stories. So I'm happy to add it to my photo pages. If someone on a mobile browser wants to view every hair in a squirrel's tail, they can still zoom in to the page, or long-press on the image to view it at full resolution. Or rotate to landscape mode.

The CSS rule works for those wide page banners too. Or you can use overflow: hidden if the right side of your banner isn't all that important.

Anyway, that takes care of the "page too wide" problem. As for the "Links too close together" even after I added the meta viewport tag, that was just plain bad HTML and CSS, showing that I don't do enough testing on different window sizes. I fixed it so the buttons lay out better and don't draw on top of each other on super narrow screens, which I should have done long ago. Likewise for some layout problems I found on my blog.

So despite my annoyance with the whole viewport thing, Google's mandate did make me re-examine some pages that really needed fixing, and should have improved my website quite a bit for anyone looking at it on a small screen. I'm glad of that.

It'll be a while before I have all my pages converted, especially that business of adding the meta tag to all of them. But readers, if you see usability problems with my site, whether on mobile devices or otherwise, please tell me about them!

Python scripts for Android

Python on Android. Wouldn't that make so many things so much easier?

I've known for a long time about SL4A, but when I read, a year or two ago, that Google officially disclaimed support for languages other than Java and C and didn't want their employees working on projects like SL4A, I decided it wasn't a good bet.

But recently I heard from someone who had just discovered SL4A and its Python support and talked about it like a going thing. I had an Android scripting problem I really wanted to solve, and decided it was time to take another look.

It turns out SL4A and its Python interpreter are still being maintained, and indeed, I was able to solve my problem that way. But the documentation was scanty at best. So here are some shortcuts.

Getting Python running on Android

How do you install it in the first place? Took me three or four tries: it turns out it's extremely picky about the order in which you do things, and the documentation doesn't warn you about that. Follow these steps:

1. Enable "Unknown Sources" under Application settings if you haven't already.
2. Download both sl4a_r6.apk and PythonForAndroid_r4.apk
3. Install sl4a from the apk. Do not install Python yet.
4. Find SL4A in Applications and run it. It will say "no matches found" (i.e. no scripts) but that's okay: the important thing is that it creates the directory where the scripts will live, /sdcard/sl4a/scripts, without which PythonForAndroid would fail to install.
5. Install PythonForAndroid from the apk.
6. Find Python for Android in Applications and run it. Tap Install. This will install the sample scripts, and you'll be ready to go.

Make a shortcut on the home screen:

You've written a script and it does what you want. But to run it, you have to run SL4A, choose the Python interpreter, scroll around to find the script, tap on it, and indicate whether or not you want to see the console. Way too many steps!

Turns out you can make a shortcut on the home screen to an SL4A script, like this: (thanks to this tip):

• Hit the add icon button ("+") on the main screen.
• tap on Shortcuts
• scroll down to Scripts
• choose your script
• choose the icon indicating whether you want to show the console while the script is running

This will give you the familiar twin-snake Python icon on your home screen. There doesn't seem to be any way to change this to a different icon.

Wait, what about UI?

Well, that still seems to be a big hole in the whole SL4A model. You can write great scripts that print to the console. You can even do a few specialized things, like popup menus, messages (what the Python Android module calls makeToast()) and notifications. The test.py sample script is a great illustration of how to use all those features, plus a lot more.

But what if you want to show a window, put a few buttons in it, let the user control things? Nobody seems to have thought about that possibility. I mean, it's not "sorry, we haven't had time to implement this", it isn't even mentioned as something someone would want to do on an Android device. Boggle.

The only possibility I've found is that there is apparently a way to use Android's WebView class from Python. I have not tried this yet; when I do, I'll write it up separately.

WebView may not be the best way to do UI. I've spent many hours tearing my hair out over its limitations even when called from Java. But still, it's something. And one very interesting thing about it is that it provides an easy way to call up an HTML page, either local or remote, from an Android home screen icon. So that may be the best reason yet to check out SL4A.

Playing an MP3 file from an Android app

I've mostly been enormously happy with my upgrade from my old Archos 5 to the Samsung Galaxy Player 5.0. The Galaxy does everything I always wanted the Archos to do, all those things the Archos should have done but couldn't because of its buggy and unsupported Android 1.6.

That is, I've been happy with everything except one thing: my birdsong app no longer worked.

I have a little HTML app based on my "tweet" python script which lets you choose from a list of birdsong MP3 files. (The actual MP3 files are ripped from the excellent 4-CD Stokes Field Guide to Western Bird Songs set.) The HTML app matches bird names as you type in characters. (If you're curious, an earlier test version is at tweet.html.)

On the Archos, I ran that under my WebClient Android app (I had to modify the HTML to add a keyboard, since in Android 1.6 the soft keyboard doesn't work in WebView text fields). I chose a bird, and WebView passed off the MP3 file to the Archos' built-in audio player. Worked great.

On the Samsung Galaxy, no such luck. Apparently Samsung's built-in media player can only play files it has indexed itself. If you try to use it to play an arbitrary file, say, "Song_Sparrow.mp3", it will say: unknown file type. No matter that the file ends in .mp3 ... and no matter that I've called intent.setDataAndType(Uri.parse(url), "audio/mpeg"); ... and no matter that the file is sitting on the SD cad and has in fact been indexed already by the media player. You didn't navigate to it via the media player's UI, so it refuses to play it.

I haven't been able to come up with an answer to how to make Samsung's media player more flexible, and I was just starting a search for alternate Android MP3 player apps, when I ran across Play mp3 in SD Card, using Android's MediaPlayer and Error creating MediaPlayer with Uri or file in assets which gave me the solution. Instead of using an intent and letting WebView call up a music application, you can use an Android MediaPlayer to play your file directly.

Here's what the code looks like, inside setWebViewClient() which is itself inside onCreate():

            @Override
            public boolean shouldOverrideUrlLoading(WebView view, String url) {
                if (url.endsWith(".mp3")) {
                    MediaPlayer mediaPlayer = new MediaPlayer();
                    try {
                        mediaPlayer.setDataSource(getApplicationContext(), Uri.parse(url));
                        mediaPlayer.prepare();
                        mediaPlayer.start();
                    }
                    catch (IllegalArgumentException e) { showMessage("Illegal argument exception on " + url); }
                    catch (IllegalStateException e) { showMessage("Illegal State exception on " + url); }
                    catch (IOException e) { showMessage("I/O exception on " + url); }
                }
            }

showMessage() is my little wrapper that pops up an error message dialog. Of course, you can handle other types, not just files ending in .mp3.

And now I can take the Galaxy out on a birdwalk and use it to help me identify bird songs.

Android WebView can't goBack from a page with iframes

I've been fighting a bug in Android's WebView class for ages: on some pages, clicking FeedViewer's back arrow (which calls WebView::goBack()) doesn't go back to the previous page. Instead, it jumps to some random position in the current page. If you repeat it, eventually, after five or so tries (depending on the page), eventually goBack() will finally work and you'll be back at the previous page.

It was especially frustrating in that it didn't happen everywhere -- only in pages from certain sites. I saw it all the time in pages from the Los Angeles Times and from Make Magazine, but only rarely on other sites.

But I finally tracked it down: it's because those pages include the HTML <iframe> tag. Apparently, if a WebView is on a page (at least if it's a local page) that contains N iframes, the first N calls to goBack will jump somewhere in the document -- probably the location of the associated iframe, though I haven't verified that -- and only on the N+1st call will the WebView actually go back to the previous page.

The oddest thing is, this doesn't seem to be reported anywhere. Android's bug tracker finds nothing for webview iframe goback, and web searching hasn't found a hint of it, even though I see this in Android versions from 1.6 through 2.3.5. How is it possible that other people haven't noticed this? I wonder if it only happens on local file:// URLs, and not many people use those.

In any case, I'm very happy to have found the answer at last. It was easy enough to modify FeedMe to omit iframes (and who wants iframes in simplified HTML anyway?), and it's great to have a Back button that works on any page.

Mounting a Samsung Galaxy Player on Linux

My new toy: a Samsung Galaxy Player 5.0!

So far I love it. It's everything my old Archos 5 wanted to be when it grew up, except the Archos never grew up. It's the same size, a little lighter weight, reliable hardware (no random reboots), great battery life, fast GPS, modern Android 2.3, and the camera is even not too bad (though it certainly wouldn't tempt me away from my Canon).

For the record, Dave got a Galaxy Player 4.0, and it's very nifty too, and choosing between them was a tough choice -- the 4-inch is light and so easy to hold, and it uses replaceable batteries, while the 5-inch's screen is better for reading and maps.

USB-storage devices don't register

I love the Galaxy ... but there's one thing that bugs me about it. When I plug it in to Linux, dmesg reports two new storage devices, one for main storage and one for the SD card. Just like most Android devices, so far.

The difference is that these Samsung devices aren't fully there. They don't show up in /proc/partitions or in /dev/disk/by-uuid, dmesg doesn't show sizes for them, and, most important, they can't be mounted by UUID from an fstab entry, like

UUID=62B0-C667   /droidsd    vfat   user,noauto,exec,fmask=111,shortname=lower 0 0

I found that if I mounted it explicitly using the device pathname -- mount /dev/sdf /mnt -- then subsequently the device shows up normally, even after I unmount it. So I could check dmesg to find the device name, mount it as root, unmount as root, then mount it again as myself using my fstab entry. What a pain!

A kernel expert I asked thought it looked like the Samsung is pretending to be a removable device, but only "plugging in" when the system actually tries to access it. Annoying. So how do you get Linux to "access" it?

Udev: still an exercise in frustration

The obvious solution is a udev rule. Some scrutiny of /lib/udev/rules.d/60-persistent-storage.rules found some rules that did this intriguing thing: IMPORT{program}="ata_id --export $tempnode".

Naturally, this mysterious ata_id is undocumented. It's hidden in /lib/udev/ata_id, and I found this tiny ata_id man page online since there's none available in Ubuntu. Running ata_id /dev/sdf seemed to do what I needed: it made the device show up in /proc/partitions and /dev/disk/by-uuid, and after that, I could mount it without being root.

I created a file named /etc/udev/rules.d/59-galaxy.rules, with the rule:

KERNEL=="sd[b-g]", SUBSYSTEMS=="usb", ATTRS{idVendor}=="04e8", SYMLINK+="galaxy-%k-%n", IMPORT{program}="ata_id --export $tempnode"

When I tested it with udevadm test /block/sdf, not only did the rule trigger correctly, but it actually ran ata_id and the device became visible -- even though udevadm test states clearly no programs will be run. How do I love udev -- let me count the ways.

But a reboot revealed that udev was not actually running the rule when I actually plugged the Galaxy in -- the devices did not become visible. Did I mention how much I love udev?

Much simpler: a shell alias

But one thing I'd noticed in all this: side by side with /dev/disk/by-uuid is a directory called /dev/disk/by-id. And the Samsung devices did show up there, with names like usb-Android_UMS_Composite_c197022a2b41c1ae-0:0.

Faced with the prospect of spending the rest of the day trying random udev rules, rebooting each time since that's the only way to test udev, I decided to cheat and find another way. I made a shell alias:

alias galaxy='sudo sh -c "for d in /dev/disk/by-id/usb-Android*; do /lib/udev/ata_id --export \$d; done"'

Typing galaxy will now cause the Samsung to register both devices; and from then on I can mount and unmount them without needing root, using my normal fstab entries.

Update: This works for the Nook's main storage, too -- just add x/dev/disk/by-id/usb-B_N_Ebook_Disk* to the list -- but it doesn't work reliably for the Nook's SD card. The SD card does show up in /dev/disk/by-id along with main storage; but running ata_id on it doesn't make its UUID appear. I may just change my fstab entry to refer to the /dev/disk/by-id device directly.

Quick Guide to Android ADB

I've been fiddling with several new Android devices, which means I have to teach myself how to use adb all over again.

adb is the Android Debug Bridge, and it's great for debugging. It lets you type commands on your desktop keyboard rather than tapping them into the soft keyboard in Android's terminal emulator, it gives you a fast way to install apps, and most important, it lets you get Java stack traces from crashing programs.

Alas, the documentation is incomplete and sometimes just plain wrong. Since I don't need adb very often, I always forget how to use it between sessions, and it takes some googling to figure out the tricks. Here are the commands I've found most helpful.

Start the server

First you have to start the adb, and that must be done as root. But adb isn't a system program and probably lives in some path like /home/username/path/to/android-sdk-linux_x86/tools. Even if you put it in your own path, it may not be in root's. You can probably run it with the explicit path:

$ sudo /path/to/android-sdk-linux_x86/tools/adb start-server

# export PATH=$PATH:/path/to/android/android-sdk-linux_x86/tools
# adb start-server

If you're also running eclipse, that probably won't work the first time, because eclipse may also have started an adb server (that gets in the way when you try to run adb manually). if you don't see "* daemon started successfully *", try killing the server and restarting it:

# adb kill-server
# adb start-server
* daemon not running. starting it now on port 5037 *
* daemon started successfully *

Keep trying until you see that "* daemon started successfully *" message.

Connecting

$ adb usb

Occasionally, this will give "error: closed". Don't panic -- sometimes this actually means "I noticed something connected on USB and automatically connected to it, so no need to connect again." It's mysterious, and no one seems to have an explanation for what's really happening. Anyway, try running some adb commands and you may find you're actually connected.

Shell and install

The most fun is running an interactive shell on your Android device.

$ adb shell

You can also install apps. On some devices (like the Nook, where I haven't found a way to allow install from non-market sources), it's the only way to install an apk file.

$ adb install /path/to/appname.apk

If the app is already installed, you'll get an error. Theoretically you can also do adb uninstall first, but when I tried that it just printed "Failure". But you can use -r for "reinstall":

$ adb install -r /path/to/appname.apk

There is no mention of either uninstall or -r in the online adb documentation, though adb -h mentions it.

Update: To uninstall, you need the full name of the package. To get the names of installed packages (another undocumented command), do this: adb shell pm list packages

Debug crashes with logcat

Finally, for debugging crashes, you can start up a logcat and see system messages, including stack traces from crashing apps:

$ adb logcat

Logcat is great for fixing reproducible crashes. Sadly, it's not so good for random/sporadic crashes that happen during the course of using the device.

You're supposed to be able to do adb logcat -s AppName if you're only interested in debugging messages from one app, but that doesn't work for me -- I get no output even when the app runs and crashes.

Reading, converting and editing EPUB ebooks

There are tons of places to get EPUB ebooks -- I won't try to list them all, but Project Gutenberg is a good place to start. The next question was how to read them.

Reading EPUB books: Aldiko or FBReader

I've already mentioned Aldiko in my post on Android as an RSS reader. It's not so good for reading short RSS feeds, but it's excellent for ebooks.

But Aldiko has one fatal flaw: it insists on keeping its books in one place, and you can't change it. When I tried to add a big technical book, Aldiko spun for several minutes with no feedback, then finally declared it was out of space on the device. Frustrating, since I have a nearly empty 8-gigabyte micro-SD card and there's no way to get Aldiko to use it. Fiddling with symlinks didn't help.

A reader gave me a tip a while back that I should check out FBReader. I'd been avoiding it because of a bad experience with the early FBReader on the Nokia 770 -- but it's come a long way since then, and FBReaderJ, the Android port, works very nicely. It's as good a reader as Aldiko (except I wish the line spacing were more configurable). It has better navigation: I can see how far along in the book I am or jump to an arbitrary point, tasks Aldiko makes quite difficult. Most important, it lets me keep my books anywhere I want them. Plus it's open source.

Creating EPUB books: Calibre and ebook-convert

I hadn't had the tablet for long before I encountered an article that was only available as PDF. Wouldn't it be nice to read it on my tablet?

Of course, Android has lots of PDF readers. But most of them aren't smart about things like rewrapping lines or changing fonts and colors, so it's an unpleasant experience to try to read PDF on a five-inch screen. Could I convert the PDF to an EPUB?

Sadly, there aren't very many open-source options for handling EPUB. For converting from other formats, you have one choice: Calibre. It's a big complex GUI program for organizing your ebook library and a whole bunch of other things I would never want to do, and it has a ton of prerequisites, like Qt4. But the important thing is that it comes with a small Python script called ebook-convert.

ebook-convert has no built-in help -- it takes lots of options, but to find out what they are, you have to go to the ebook-convert page on Calibre's site. But here's all you typically need

ebook-convert --authors "Mark Twain" --title "Huckleberry Finn" infile.pdf huckfinn.epub

ebook-convert infile.pdf huckfinn.epub --authors "Mark Twain" --title "Huckleberry Finn"

Pretty easy; the only hard part is remembering that it's --authors and not --author.

Calibre (and ebook-convert) can take lots of different input formats, not just PDF. If you're converting ebooks, you need it. I wish ebook-convert was available by itself, so I could run it on a server; I took a quick stab at separating it, but even once I separated out the Qt parts it still required Python libraries not yet available on Debian stable. I may try again some day, but for now, I'll stick to running it on desktop systems.

Editing EPUB books: Sigil

But we're not quite done yet. Calibre and ebook-convert do a fairly good job, but they're not perfect. When I tried converting my GIMP book from a PDF, the chapter headings were a mess and there was no table of contents. And of course I wanted the cover page to be right, instead of the default Calibre image. I needed a way to edit it.

EPUB is an XML format, so in theory I could have fixed this with a text editor, but I wanted to avoid that if possible.

And I found Sigil. Wikipedia claims it's the only application that can edit EPUB books.

There's no sigil package in Ubuntu (though Arch has one), but it was very easy to install from the sigil website.

And it worked beautifully. I cleaned up the font problems at the beginnings of chapters, added chapter breaks where they were missing, and deleted headings that didn't belong. Then I had Sigil auto-generate a table of contents from headers in the document. I was also able to fix the title and put the real book cover on the title page.

It all worked flawlessly, and the ebook I generated with Sigil looks very nice and has working navigation when I view it in FBReaderJ (it's still too big for Aldiko to handle). Very impressive. If you've ever wanted to generate your own ebook, or edit one you already have, you should definitely check out Sigil.

Getting rid of extra whitespace from Eclipse

I found lots of suggestions on the web for eliminating extra whitespace, and several places to configure this within Eclipse, but most of them don't do anything. Here's the one that actually worked:

Window->Preferences
Jave->Editor->Save Actions
Enable Perform the selected actions on save.
Enable Additional actions.
Click Configure.
In the Code Organizing tab., enable Remove trailing whitespace for All lines.
Review all the other options there, since it will all happen automatically whenever you save -- make sure there isn't anything there you don't want.
Dismiss the Configure window.
Review the other options under Save Actions, since these will also happen automatically now.
Don't forget to click Apply in the Save Actions preference page.

Whew! There are other places to set this, in various Code style and Cleanup options, but all all the others require taking some action periodically, like Source->Clean up...

By the way, while you're changing whitespace preferences, you may also want the Insert spaces for tabs preference under General->Editors->Text Editors.

An easy way to check whether you've succeeded in exorcising the spaces -- eclipse doesn't show them all, even when you tell it to -- is to :set hlsearch in vim, then search for a space. (Here are some other ways to show spaces in vim.) In emacs, you can M-x set-variable show-trailing-whitespace to true, but that doesn't show spaces on blank lines; for that you might want whitespace.el or similar packages.

Developing for Android

I wrote yesterday about my quest for an app for reading news feeds and other timely information from the web. And how existing ebook readers didn't meet that need. That meant I would have to write something.

Android development is done in Java, using Eclipse as an IDE. Let me just state up front that (a) I dislike Java (and have forgotten most of what I once knew about it) and (b) I hate IDEs -- they make you use their crippled editor instead of your own, they control what you can put where on the screen, and they're always popping up windows that get in your way.

Okay, not an auspicious beginning. But let's try to be open-minded, follow the instructions and see what happens.

I installed Eclipse from eclipse.org after being advised that the version on Ubuntu is out of date. Then I installed all the various Android plug-ins and SDKs and set them up (there is no single page that lists all the steps, so I did a lot of googling). It took maybe an hour or so to get it all installed to the point where it could produce its "Hello world".

And then ... wow! Hello world worked right off the bat, and the emulator worked for basic testing. Hmm, okay ... how about if we use HTML as a format ... is there a standard HTML display component? Sure enough -- I added a WebView to my app and right away I had a working HTML reader. Okay, how about a row of buttons and a status bar on top? Sure, no problem.

The standard Android online docs aren't great -- they're a wonderful example of how to write seemingly comprehensive documentation that somehow manages to tell you nothing of what you actually need to know. But that's not as bad as it sounds, because there are lots of forums and tutorials to fill in the gaps. Stack Overflow is particularly good for Android tips.

And yes, I did some swearing at Eclipse and spent too much time googling how to disable features, like the "Content Assist" that periodically freezes the whole UI for a minute or so in the middle of your typing a line of code, while it thinks about some unhelpful and irrelevant hints to offer you in a popup. Turn it off in the prefs under Java/Editor. (Eclipse's actual useful hints, like the ones you get when you hover over something that's red because of an error, will still work. I found them very helpful.)

More specifically: Java/Editor/Content Assist/Hovers, and turn off Combined Hover and maybe anything else that happens without a modifier key. You probably also want to turn off Enable Auto Activation under Java/Editor/Content Assist. And possibly others -- I kept turning things off until the popups and delays went away, and I haven't found anything explaining how all these parameters relate.

Okay, so there were snags, and it's frustrating how there are almost no open source apps for this open source OS. (Yes, my app will be.) But here's the thing: in about 4 days, starting from nothing, I had a little RSS reader that did everything I needed. I've been adding features since then. Android doesn't have a reasonable battery status monitor? Fine, my reader can show battery percentage in the status bar. Android doesn't dim the screen enough? Fine, I can dim it further inside the application (an idea borrowed from Aldiko).

After less than a week of work I have an RSS reader that's better than my Palms running Plucker ever were. And that says a lot about the ease of the Android programming environment. I'm impressed!

Update: The source, an apk, and a brief discussion of how I use my feed reader are now here: FeedViewer.

Android tablet as an ebook/RSS reader

I've been searching for years for something to replace the aging and unsupported Palm platform. I've been using Palms for many years to read daily news feeds; first on the proprietary Avantgo service, but mostly using the open source Plucker.

I don't normally have access to a network when I'm reading -- I might be a passenger in a car or train, in a restaurant, standing in line at the market, or in the middle of the Mojave desert. So I run a script once a day on a network-connected computer to gather up a list of feeds, package it up and transfer it to the mobile device, so I have something to read whenever I find spare time.

For years I used Sitescooper on the host end to translate HTML pages into a mobile format, and eventually became its primary maintainer. But that got cumbersome, and I wrote a simpler RSS feed reader, feedme.

But on the reader side, that still left me buying old PalmOS Clies on ebay. Surely there was a better option?

I've been keeping an eye on ebook readers and tablets for a while now. But the Plucker reader has several key features not available in most ebook reader apps:

1. An easy, open-source way of automatically translating RSS and HTML pages into something the reader can understand;
2. Delete documents after you've read them, without needing to switch to a separate application;
3. Random access to document, e.g. jump to the beginning or end, or 60% in;
4. Follow links: nearly all RSS sites, whether news sites or blogs, are set up as an index page with links to individual story pages;
5. Save external links if you click on them while offline, so you can fetch them later.

Most modern apps seem to assume either (a) that you'll be reading only books packaged commercially, or (b) that you're reading web pages and always have a net connection. Which meant that I'd probably have to roll my own; and that pointed to Android tablets rather than dedicated ebook readers.

Android as a reader

All the reviews I read pointed to Aldiko as the best e-reader on Android, so I installed it first thing. And indeed, it's a wonderful reader. The font is beautiful, and you can adjust size and color easily, including a two-click transition between configurable "day" and "night" schemes. It's easy to turn pages (something surprisingly difficult in most Android apps, since the OS seems to have no concept of "Page down"). It's easy to import new documents and easy to delete them after reading them.

So how about those other requirements? Not so good. Aldiko uses epub format, and it's possible (after much research) to produce those using ebook-convert, a command-line script you can get as part of the huge Calibre package. Alas, Calibre requires all sorts of extraneous packages like Qt even if you're never going to use the GUI; but once you get past that, the ebook-convert script works pretty well.

Except that links don't work, much. Sometimes they do, but mostly they do nothing. I don't know if this is a problem with Calibre's ebook-convert, Aldiko's reader, or the epub format itself, but you can't rely on links from the index page actually jumping anywhere. Aldiko also doesn't have a way to jump to a set point, so once you're inside a story you can't easily go back to the title page (sometimes BACK works, sometimes it doesn't).

And of course there's no way to save external links for later.

So Aldiko is a good book reader, but it wouldn't solve my feed-reading problem.

And that meant I had to write my own reader, and it was time to delve into the world of Android development. And it was surprisingly easy ... which I'll cover in a separate post. For now, I'll skip ahead and ruin the punch line by saying I have a lovely little feed-reading app, and my Archos and Android are working out great.

Archos 5 Android Tablet review

For the past couple weeks I've been using a small Android tablet, an Archos 5. I use it primarily as an ebook and RSS feed reader (more about that separately), though of course I've played with assorted games and other apps too.

I've been trying to wait for the slew of cheap Android tablets the media assure us is coming out any day now. Except "any day now" never turns into "now". And I wanted something suitable for reading: small enough to fit in a jacket pocket and hold in one hand, yet large enough to fit a reasonable amount of text on the screen. A 4-5-inch screen seemed ideal.

There's nothing in the current crop fitting that description, but there's a year-old model, the Archos 5. It has a 4.8-inch screen, plus some other nice hardware like GPS. And it seems to have a fair community behind it, at archosfans.com.

I have the 16G flash version. I've had it for a couple of weeks now and I'm very happy so far. I'm not sure I'd recommend it to a newbie (due to the Android Marketplace's ban on tablets -- see below), but it's a lovely toy for someone fairly tech savvy.

My review turned out quite long, too long for a blog post. So if you're interested in the details of what's good and what's bad, you'll find the details in my Archos 5 Android Tablet review.

Android/Eclipse Spellchecker is a bit confused

What do you think -- should I change all the "Android" references to "Undried"?