Shallow Thoughts : tags : ebook

Downloading all the Books in a Humble Bundle

Humble Bundle has a great bundle going right now (for another 15 minutes -- sorry, I meant to post this earlier) on books by Nebula-winning science fiction authors, including some old favorites of mine, and a few I'd been meaning to read.

I like Humble Bundle a lot, but one thing about them I don't like: they make it very difficult to download books, insisting that you click on every single link (and then do whatever "Download this link / yes, really download, to this directory" dance your browser insists on) rather than offering a sane option like a tarball or zip file. I guess part of their business model includes wanting their customers to get RSI. This has apparently been a problem for quite some time; a web search found lots of discussions of ways of automating the downloads, most of which apparently no longer work (none of the ones I tried did).

But a wizard friend on IRC quickly came up with a solution: some javascript you can paste into Firefox's console. She started with a quickie function that fetched all but a few of the files, but then modified it for better error checking and the ability to get different formats.

In Firefox, open the web console (Tools/Web Developer/Web Console) and paste this in the single-line javascript text field at the bottom.

// How many seconds to delay between downloads.
var delay = 1000;
// whether to use window.location or window.open
// window.open is more convenient, but may be popup-blocked
var window_open = false;
// the filetypes to look for, in order of preference.
// Make sure your browser won't try to preview these filetypes.
var filetypes = ['epub', 'mobi', 'pdf'];

var downloads = document.getElementsByClassName('download-buttons');
var i = 0;
var success = 0;

function download() {
  var children = downloads[i].children;
  var hrefs = {};
  for (var j = 0; j < children.length; j++) {
    var href = children[j].getElementsByClassName('a')[0].href;
    for (var k = 0; k < filetypes.length; k++) {
      if (href.includes(filetypes[k])) {
        hrefs[filetypes[k]] = href;
        console.log('Found ' + filetypes[k] + ': ' + href);
      }
    }
  }
  var href = undefined;
  for (var k = 0; k < filetypes.length; k++) {
    if (hrefs[filetypes[k]] != undefined) {
      href = hrefs[filetypes[k]];
      break;
    }
  }
  if (href != undefined) {
    console.log('Downloading: ' + href);
    if (window_open) {
      window.open(href);
    } else {
      window.location = href;
    }
    success++;
  }
  i++;
  console.log(i + '/' + downloads.length + '; ' + success + ' successes.');
  if (i < downloads.length) {
    window.setTimeout(download, delay);
  }
}
download();

If you have "Always ask where to save files" checked in Preferences/General, you'll still get a download dialog for each book (but at least you don't have to click; you can hit return for each one). Even if this is your preference, you might want to consider changing it before downloading a bunch of Humble books.

Anyway, pretty cool! Takes the sting out of bundles, especially big ones like this 42-book collection.

Hacking / Customizing a Kobo Touch ebook reader: Part II, Python

I wrote last week about tweaking a Kobo e-reader's sqlite database by hand.

But who wants to remember all the table names and type out those queries? I sure don't. So I wrote a Python wrapper that makes it much easier to interact with the Kobo databases.

Happily, Python already has a module called sqlite3. So all I had to do was come up with an API that included the calls I typically wanted -- list all the books, list all the shelves, figure out which books are on which shelves, and so forth.

The result was kobo_utils.py, which includes a main function that can list books, shelves, or shelf contents.

You can initialize kobo_utils like this:

import kobo_utils

koboDB = KoboDB("/path/where/your/kobo/is/mounted")
koboDB.connect("/path/to/KoboReader.sqlite")

Then you can list books thusly:

koboDB.list_books()

koboDB.list_shelves()

shelves = koboDB.get_dlist("Shelf", selectors=[ "Name" ])
for shelf in shelves:
    print shelf["Name"]

What I really wanted, though, was a way to organize my library, taking the tags in each of my epub books and assigning them to an appropriate shelf on the Kobo, creating new shelves as needed. Using kobo_utils.py plus the Python epub library I'd already written, that ended up being quite straightforward: shelves_by_tag.

Hacking / Customizing a Kobo Touch ebook reader: Part I, sqlite

I've been enjoying reading my new Kobo Touch quite a lot. The screen is crisp, clear and quite a bit whiter than my old Nook; the form factor is great, it's reasonably responsive (though there are a few places on the screen where I have to tap harder than other places to get it to turn the page), and I'm happy with the choice of fonts.

But as I mentioned in my previous Kobo article, there were a few tweaks I wanted to make; and I was very happy with how easy it was to tweak, compared to the Nook. Here's how.

Mount the Kobo

When you plug the Kobo in to USB, it automatically shows up as a USB-Storage device once you tap "Connect" on the Kobo -- or as two storage devices, if you have an SD card inserted.

Like the Nook, the Kobo's storage devices show up without partitions. For instance, on Linux, they might be /dev/sdb and /dev/sdc, rather than /dev/sdb1 and /dev/sdc1. That means they also don't present UUIDs until after they're already mounted, so it's hard to make an entry for them in /etc/fstab if you're the sort of dinosaur (like I am) who prefers that to automounters.

Instead, you can use the entry in /dev/disk/by-id. So fstab entries, if you're inclined to make them, might look like:

/dev/disk/by-id/usb-Kobo_eReader-3.16.0_N905K138254971:0 /kobo   vfat user,noauto,exec,fmask=133,shortname=lower 0 0
/dev/disk/by-id/usb-Kobo_eReader-3.16.0_N905K138254971:1 /kobosd vfat user,noauto,exec,fmask=133,shortname=lower 0 0

One other complication, for me, was that the Kobo is one of a few devices that don't work through my USB2 powered hub. Initially I thought the Kobo wasn't working, until I tried a cable plugged directly into my computer. I have no idea what controls which devices work through the hub and which ones don't. (The Kobo also doesn't give any indication when it's plugged in to a wall charger, nor does

The sqlite database

Once the Kobo is mouted, ls -a will show a directory named .kobo. That's where all the good stuff is: in particular, KoboReader.sqlite, the device's database, and Kobo/Kobo eReader.conf, a human-readable configuration file.

Browse through Kobo/Kobo eReader.conf for your own amusement, but the remainder of this article will be about KoboReader.sqlite.

I hadn't used sqlite before, and I'm certainly no SQL expert. But a little web searching and experimentation taught me what I needed to know.

First, make a local copy of KoboReader.sqlite, so you don't risk overwriting something important during your experimentation. The Kobo is apparently good at regenerating data it needs, but you might lose information on books you're reading.

To explore the database manually, run: sqlite3 KoboReader.sqlite

Some useful queries

Here are some useful sqlite commands, which you can generalize to whatever you want to search for on your own Kobo. Every query (not .tables) must end with a semicolon.

Show all tables in the database:

.tables

Show all column names in a table:

PRAGMA table_info(content);

Show the names of all your shelves/collections:

SELECT Name FROM Shelf;

Show everything in a table:

SELECT * FROM Shelf;

Show all books assigned to shelves, and which shelves they're on:

SELECT ShelfName,ContentId FROM ShelfContent;

Show all books you have installed:

SELECT Title,Attribution,ContentID FROM content WHERE BookTitle is null ORDER BY Title;

sqlite> SELECT Title,BookTitle from content WHERE ContentID LIKE "%hamlet%";
HAMLET, PRINCE OF DENMARK|Hamlet
PERSONS REPRESENTED.|Hamlet
ACT I.|Hamlet
Scene II. Elsinore. A room of state in the Castle.|Hamlet
Scene III. A room in Polonius's house.|Hamlet
Scene IV. The platform.|Hamlet
Scene V. A more remote part of the Castle.|Hamlet
Act II.|Hamlet
  [ ... and so on ... ]
ACT V.|Hamlet
Scene II. A hall in the Castle.|Hamlet
Hamlet|

Show all books by an author:

SELECT Title,Attribution,ContentID FROM content WHERE BookTitle is null
AND Attribution LIKE "%twain%" ORDER BY Title;

SELECT Title,Attribution,ContentID,BookTitle FROM content WHERE Attribution LIKE "%Doyle%" AND Title LIKE "%Adventures%";
UPDATE content SET Title="Sherlock Stories 1: The Adventures of Sherlock Holmes" WHERE Attribution LIKE "%Doyle%" AND Title LIKE "%Adventures%";
    

DELETE from content WHERE Attribution LIKE "%Doyle%" AND ContentID="file:///mnt/sd/memoirs-holmes.epub";
    

Of course, it's a lot handier to have a program that knows these queries so you don't have to type them in every time (especially since the sqlite3 app has no history or proper command-line editing). But this has gotten long enough, so I'll write about that separately.

Switching to a Kobo e-reader

For several years I've kept a rooted Nook Touch for reading ebooks. But recently it's become tough to use. Newer epub books no longer work work on any version of FBReader still available for the Nook's ancient Android 2.1, and the Nook's built-in reader has some fatal flaws: most notably that there's no way to browse books by subject tag, and it's painfully slow to navigate a library of 250 books when have to start from the As and you need to get to T paging slowly forward 6 books at a time.

The Kobo Touch

But with my Nook unusable, I borrowed Dave's Kobo Touch to see how it compared. I like the hardware: same screen size as the Nook, but a little brighter and sharper, with a smaller bezel around it, and a spring-loaded power button in a place where it won't get pressed accidentally when it's packed in a suitcase -- the Nook was always coming on while in its case, and I didn't find out until I pulled it out to read before bed and discovered the battery was too low.

The Kobo worked quite nicely as a reader, though it had a few of the same problems as the Nook. They both insist on justifying both left and right margins (Kobo has a preference for that, but it doesn't work in any book I tried). More important is the lack of subject tags. The Kobo has a "Shelves" option, called "Collections" in some versions, but adding books to shelves manually is tedious if you have a lot of books. (But see below.)

It also shared another Nook problem: it shows overall progress in the book, but not how far you are from the next chapter break. There's a choice to show either book progress or chapter progress, but not both; and chapter progress only works for books in Kobo's special "kepub" format (I'll write separately about that). I miss FBReader's progress bar that shows both book and chapter progress, and I can't fathom why that's not considered a necessary feature for any e-reader.

But mostly, Kobo's reader was better than the Nook's. Bookmarks weren't perfect, but they basically worked, and I didn't even have to spent half an hour reading the manual to use them (like I did with the Nook). The font selection was great, and the library navigation had one great advantage over the Nook: a slider so you could go from A to T quickly.

I liked the Kobo a lot, and promptly ordered one of my own.

It's not all perfect

There were a few disadvantages. Although the Kobo had a lot more granularity in its line spacing and margin settings, the smallest settings were still a lot less tight than I wanted. The Nook only offered a few settings but the smallest setting was pretty good.

Also, the Kobo can only see books at the top level of its microSD card. No subdirectories, which means that I can't use a program like rsync to keep the Kobo in sync with my ebooks directory on my computer. Not that big a deal, just a minor annoyance.

More important was the subject tagging, which is really needed in a big library. It was pretty clear Shelves/Collections were what I needed; but how could I get all my books into shelves without laboriously adding them all one by one on a slow e-ink screen?

It turns out Kobo's architecture makes it pretty easy to fix these problems.

Customizing Kobo

While the rooted Nook community has been stagnant for years -- it was a cute proof of concept that, in the end, no one cared about enough to try to maintain it -- Kobo readers are a lot easier to hack, and there's a thriving Kobo community on MobileReads which has been trading tips and patches over the years -- apparently with Kobo's blessing.

The biggest key to Kobo's customizability is that you can mount it as a USB storage device, and one of the files that exposes is the device's database (an sqlite file). That means that well supported programs like Calibre can update shelves/collections on a Kobo, access its book list, and other nifty tricks; and if you want more, you can write your own scripts, or even access the database by hand.

I'll write separately about some Python scripts I've written to display the database and add books to shelves, and I'll just say here that the process was remarkably straightforward and much easier than I usually expect when learning to access a new device.

There's lots of other customizing you can do. There are ways of installing alternative readers on the Kobo, or installing Python so you can write your own reader. I expected to want that, but so far the built-in reader seems good enough.

You can also patch the OS. Kobo updates are distributed as tarballs of binaries, and there's a very well designed, documented and supported (by users, not by Kobo) patching script distributed on MobileReads for each new Kobo release. I applied a few patches and was impressed by how easy it was. And now I have tight line spacing and margins, a slightly changed page number display at the bottom of the screen (still only chapter or book, not both), and a search that defaults to my local book collection rather than the Kobo store.

Stores and DRM

Oh, about the Kobo store. I haven't tried it yet, so I can't report on that. From what I read, it's pretty good as e-bookstores go, and a lot of Nook and Sony users apparently prefer to buy from Kobo. But like most e-bookstores, the Kobo store uses DRM, which makes it a pain (and is why I probably won't be using it much).

They use Adobe's DRM, and at least Adobe's Digital Editions app works in Wine under Linux. Amazon's app no longer does, and in case you're wondering why I didn't consider a Kindle, that's part of it. Amazon has a bad reputation for removing rights to previously purchased ebooks (as well as for spying on their customers' reading habits), and I've experienced it personally more than once.

Not only can I no longer use the Kindle app under Wine, but Amazon no longer lets me re-download the few Kindle books I've purchased in the past. I remember when my mother used to use the Kindle app on Android regularly; every few weeks all her books would disappear and she'd have to get on the phone again to Amazon to beg to have them back. It just isn't worth the hassle. Besides, Kindles can't read public library books (those are mostly EPUBs with Adobe DRM); and a Kindle would require converting my whole EPUB library to MOBI. I don't see any up side, and a lot of down side.

The Adobe scheme used by Kobo and Nook is better, but I still plan to avoid books with DRM as much as possible. It's not the stores' fault, and I hope Kobo does well, because they look like a good company. It's the publishers who insist on DRM. We can only hope that some day they come to their senses, like music publishers finally did with MP3 versus DRMed music. A few publishers have dropped DRM already, and if we readers avoid buying DRMed ebooks, maybe the message will eventually get through.

Python module for reading EPUB e-book metadata

Three years ago I wanted a way to manage tags on e-books in a lightweight way, without having to maintain a Calibre database and fire up the Calibre GUI app every time I wanted to check a book's tags. I couldn't find anything, nor did I find any relevant Python libraries, so I reverse engineered the (simple, XML-bsaed) EPUB format and wrote a Python script to show or modify epub tags.

I've been using that script ever since. It's great for Project Gutenberg books, which tend to be overloaded with tags that I don't find very useful for categorizing books ("United States -- Social life and customs -- 20th century -- Fiction") but lacking in tags that I would find useful ("History", "Science Fiction", "Mystery").

But it wasn't easy to include it in other programs. For the last week or so I've been fiddling with a Kobo ebook reader, and I wanted to write programs that could read epub and also speak Kobo-ese. (I'll write separately about the joys of Kobo hacking. It's really a neat little e-reader.)

So I've factored my epubtag script into a usable Python module, so as well as being a standalone program for viewing epub book data, it's easy to use from other programs. It's available on GitHub: epubtag.py: parse EPUB metadata and view or change subject tags.

Viewing and modifying epub ebook tags

My epub Books folder is starting to look like my physical bookshelf at home -- huge and overflowing with books I hope to read some day. Mostly free books from the wonderful Project Gutenberg and DRM-free books from publishers and authors who support that model.

With the Nook's standard library viewer that's impossible to manage. All you can do is sort all those books alphabetically by title or author and laboriously page through, some five books to a page, hoping the one you want will catch your eye. Worse, sometimes books show up in the author view but don't show up in the title view, or vice versa. I guess Barnes & Noble think nobody keeps more than ten or so books on their shelves.

Fortunately on my rooted Nook I have the option of using better readers, like FBreader and Aldiko, that let me sort by tags. If I want to read something about the Civil War, or Astronomy, or just relax with some Science Fiction, I can browse by keyword.

Well, in theory. In practice, tagging of ebooks is inconsistent and not very useful.

For instance, the Gutenberg tags for Othello are:

• Tragedies
• Othello (Fictitious character) -- Drama
• Jealousy -- Drama
• Interracial marriage -- Drama
• Venice (Italy) -- Drama
• Muslims -- Drama

• Satire
• England -- Fiction
• Married women -- Fiction
• Female friendship -- Fiction
• Social classes -- Fiction
• British -- Europe -- Fiction
• Waterloo, Battle of, Waterloo, Belgium, 1815 -- Fiction

The Prince and the Pauper's tag list looks like:

• Edward VI, King of England, 1537-1553 -- Fiction
• Impostors and imposture -- Fiction
• Social classes -- Fiction
• Poor children -- Fiction
• Lookalikes -- Fiction
• Princes -- Fiction
• Boys -- Fiction
• London (England) -- Fiction
• Historical fiction

• Sea stories
• Saltwater fishing -- Fiction
• Children of the rich -- Fiction
• Bildungsromans
• Fishing boats -- Fiction
• Teenage boys -- Fiction
• Rescues -- Fiction
• Fishers -- Fiction
• Grand Banks of Newfoundland -- Fiction

I can understand wanting to tag details like this, but few of those tags are helpful when I'm browsing books on my little handheld device. I can't imagine sitting down to read and thinking, "Let's see, what books do I have on Interracial marriage? Or Saltwater fishing? No, on second thought I'd rather read some fiction set in the time of Edward VI, King of England, 1537-1553."

And of course, with over 90 books loaded on my ebook readers, it means I have hundreds of entries in my tags list, with few of them including more than one book.

Clearly what I needed to do was to change the tags on my ebooks.

Viewing and modifying epub tags

That ought to be simple, right? But ebooks are still a very young technology, and there's surprisingly little software devoted to them. Calibre can probably do it if you don't mind maintaining your whole book collection under calibre; but I like to be able to work on files one at a time or in small groups. And I couldn't find a program that would let me do that.

What to do? Well, epub is a fairly simple XML format, right? So modifying it with Python shouldn't that hard.

Managing epub in Python

An epub file is a collection of XML files packaged in a zip archive. So I unzipped one of my epub books and poked around. I found the tags in a file called content.opf, inside a <metadata> tag. They look like this:

<dc:subject>Science fiction</dc:subject>

So I could use Python's zipfile module to access the content.opf file inside the zip archive, then use the xml.dom.minidom parser to get to the tags. Writing a script to display existing tags was very easy.

What about replacing the old, unweildy tag list with new, simple tags?

It's easy enough to add nodes in Python's minidom. So the trick is writing it back to the epub file. The zipfile module doesn't have a way to modify a zip file in place, so I created a new zip archive and copied files from the old archive to the new one, replacing content.opf with a new version.

Python's difficulty with character sets in XML

But I hit a snag in writing the new content.opf. Python's XML classes have a toprettyxml() method to write the contents of a DOM tree. Seemed simple, and that worked for several ebooks ... until I hit one that contained a non-ASCII character. Then Python threw a UnicodeEncodeError: 'ascii' codec can't encode character u'\u2014' in position 606: ordinal not in range(128).

Of course, there are ways (lots of them) to encode that output string -- I could do

ozf.writestr(info, dom.toprettyxml().encode(encoding, 'xmlcharrefreplace'))

writestr(info, dom.toprettyxml(encoding=encoding)

Since you need a charset to avoid the UnicodeEncodeError, the only options are (1) always use a fixed charset, like utf-8, for content.opf, or (2) open content.opf and parse the charset line by hand after Python has already parsed the rest of the file. Yuck! So I chose the first option ... I can always revisit that if the utf-8 in content.opf ever causes problems.

The final script

Charset difficulties aside, though, I'm quite pleased with my epubtags.py script. It's very handy to be able to print tags on any .epub file, and after cleaning up the tags on my ebooks, it's great to be able to browse by category in FBreader. Here's the program: epubtag.py.

Downloading Adobe-protected books to a Nook using Linux

University of Chicago Press has a Carl Zimmer book, A Planet of Viruses, as their free monthly e-book.

I know Zimmer is a good writer. but the ebook, despite being free, is encumbered with Adobe's version of DRM, which unlocks via a Windows or Mac program. I use Linux, and wanted to read the book on a Nook. Was I out of luck?

Happily, the instruction page they sent when I signed up for the book helpfully included a section for Linux users. Hooray, U. Chicago! It said Adobe Digital Editions will run under Wine, the Windows emulator. I'd been meaning to try that anyway, and a Carl Zimmer book seemed like the perfect excuse.

And overall, it worked pretty well, with only a few snags. Here are the steps I had to follow:

Authorizing a book using Adobe Digital Editions in Linux on Wine

Install wine (on Ubuntu, I used apt-get install wine).

Download the Adobe Digital Editions setup.exe

Run: wine setup.exe (this should install ADE inside your .wine directory)

Copy the file, e.g. URLLink.acsm, into .wine/drive_c/My\ Documents/ Don't bother trying to open it with ADE -- the program won't open anything except PDF and epub. Curiously, the only ways to open the file from ADE are to drag the file onto the ADE window or to pass it as a commandline argument:
wine start .wine/drive_c/My\ Documents/URLLink.acsm

Now ADE should download your book and display it. You can read it there, if you want. But you won't want to -- it's not a good reading interface.

Authorizing a device with Adobe Digital Editions under Wine

• Plug in your ebook reader.
• Mount the device wherever you like -- /media/nook, /nook or whatever.
• With ADE not running (quit it if it's running), map a drive letter to the mount point:
  • Run winecfg
  • Click the Drives tab
  • Click Add...
  • Choose a drive letter (I chose D:)
  • Under Path: type in the mount point, like /nook
  • Click Show Advanced
  • Set the Type: to Floppy disk
  • Click OK to save it
• Now the drive is mapped. Re-run ADE:
  wine .wine/drive_c/Program\ Files/Adobe/Adobe\ Digital\ Editions/digitaleditions.exe
  ADE should now see the device and ask you if you want to authorize it.
• In ADE, drag the chosen book onto the left sidebar entry for the device.
• umount the reader ... and now your new book should show up in the library.

In theory, the drive letter should stay mapped, so you should be able to use it for opening future books. Just remember to mount your device to the same location before running ADE under wine.

Nook Touch: the good, the bad, and the crazy

Here's a preliminary review, based on a few days of playing with it.

Nice size, nice screen

The Nook Touch feels very light. It's a little heavier than a paperback, but it's easy to hold, and the rubbery back feels nice in the hand. The touchscreen works well enough for book reading, though you wouldn't want to try to play video games or draw pictures on it.

It's very easy to turn pages, either with the hardware buttons on the bezel or a tap on the edges of the screen. Page changes are much faster than with older e-ink readers like the original Nook or the Sony Pocket: the screen still flashes black with each page change, but only very briefly.

I'd wondered how a non-backlit e-ink display would work in dim light, since that's one thing you can't test in stores. It turns out it's not as good as a paper book -- neither as sharp nor as contrasty -- but still readable with my normal dim bedside lighting.

Changing fonts, line spacing and margins is easy once you figure out that you need to tap on the screen to get to that menu. Navigating within a book is also via that tap-on-page menu -- it gives you a progress meter you can drag, or a "jump to page" option. Which is a good thing. This is sadly very important (see below).

Searching within books isn't terribly convenient. I wanted to figure out from the user manual how to set a bookmark, and I couldn't find anything that looked helpful in the user manual's table of contents, so I tried searching for "bookmark". The search results don't show much context, so I had to try them one at a time, and there's no easy way to go back and try the next match. (Turns out you set a bookmark by tapping in the upper right corner, and then the bookmark applies to the next several pages.)

Plan to spend some quality time reading the full-length manual (provided as a pre-installed ebook, naturally) learning tricks like this: a lot of the UI isn't very discoverable (though it's simple enough once you learn it) so you'll miss a lot if you rely on what you can figure out by tapping around.

Off to a tricky start with minor Wi-fi issues

When we first powered up, we hit a couple of problems right off with wireless setup.

First, it had no way to set a static IP address. The only way we could get the Nook connected was to enable DHCP on the router.

But even then it wouldn't connect. We'd re-type the network password and hit "Connect"; the "Connect" button would flash a couple of times, leaving an "incorrect password" message at the top of the screen. This error message never went away, even after going back to the screen with the list of networks available, so it wasn't clear whether it was retrying the connection or not.

Finally through trial and error we found the answer: to clear a failed connection, you have to "Forget" the network and start over. So go back to the list of wireless networks, choose the right network, then tap the "Forget" button. Then go back and choose the network again and proceed to the connect screen.

Connecting to a computer

The Nook Touch doesn't come with much in the way of starter books -- just two public-domain titles, plus its own documentation -- so the first task was to download a couple of Project Gutenberg books that Mom had been reading on her Treo.

The Nook uses a standard micro-USB cable for both charging and its USB connection. Curiously, it shows up as a USB device with no partitions -- you have to mount sdb, not sdb1. Gnome handled that and mounted it without drama. Copying epub books to the Nook was just a matter of cp or drag-and-drop -- easy.

Getting library books may be moot

One big goal for this device is reading ebooks from the public library, and I had hoped to report on that. But it turns out to be a more difficult proposition than expected. There are all the expected DRM issues to surmount, but before that, there's the task of finding an ebook that's actually available to check out, getting the library's online credentials straightened out, and so forth. So that will be a separate article.

The fatal flaw: forgetting its position

Alas, the review is not all good news. While poking around, reading a page here and there, I started to notice that I kept getting reset back to the beginning of a book I'd already started. What was up?

For a while I thought it was my imagination. Surely remembering one's place in a book you're reading is fundamental to a device designed from the ground up as a book reader. But no -- it clearly was forgetting where I'd left off. How could that be?

It turns out this is a known and well reported problem with what B&N calls "side-loaded" content -- i.e. anything you load from your computer rather than download from their bookstore. With side-loaded books, apparently connecting the Nook to a PC causes it to lose its place in the book you're reading! (also discussed here and here).

There's no word from Barnes & Noble about this on any of the threads, but people writing about it speculate that when the Nook makes a USB connection, it internally unmounts its filesystems -- and forgets anything it knew about what was on those filesystems.

I know B&N wants to drive you to their site to buy all your books ... and I know they want to keep you online checking in with their store at every opportunity. But some people really do read free books, magazines and other "side loaded" content. An ebook reader that can't handle that properly isn't much of a reader.

It's too bad. The Nook Touch is a nice little piece of hardware. I love the size and light weight, the daylight-readable touchscreen, the fast page flips. Mom is being tolerant about her new toy, since she likes it otherwise -- "I'll just try to remember what page I was on." But come on, Barnes & Noble: a dedicated ebook reader that can't remember where you left off reading your book? Seriously?

Reading, converting and editing EPUB ebooks

There are tons of places to get EPUB ebooks -- I won't try to list them all, but Project Gutenberg is a good place to start. The next question was how to read them.

Reading EPUB books: Aldiko or FBReader

I've already mentioned Aldiko in my post on Android as an RSS reader. It's not so good for reading short RSS feeds, but it's excellent for ebooks.

But Aldiko has one fatal flaw: it insists on keeping its books in one place, and you can't change it. When I tried to add a big technical book, Aldiko spun for several minutes with no feedback, then finally declared it was out of space on the device. Frustrating, since I have a nearly empty 8-gigabyte micro-SD card and there's no way to get Aldiko to use it. Fiddling with symlinks didn't help.

A reader gave me a tip a while back that I should check out FBReader. I'd been avoiding it because of a bad experience with the early FBReader on the Nokia 770 -- but it's come a long way since then, and FBReaderJ, the Android port, works very nicely. It's as good a reader as Aldiko (except I wish the line spacing were more configurable). It has better navigation: I can see how far along in the book I am or jump to an arbitrary point, tasks Aldiko makes quite difficult. Most important, it lets me keep my books anywhere I want them. Plus it's open source.

Creating EPUB books: Calibre and ebook-convert

I hadn't had the tablet for long before I encountered an article that was only available as PDF. Wouldn't it be nice to read it on my tablet?

Of course, Android has lots of PDF readers. But most of them aren't smart about things like rewrapping lines or changing fonts and colors, so it's an unpleasant experience to try to read PDF on a five-inch screen. Could I convert the PDF to an EPUB?

Sadly, there aren't very many open-source options for handling EPUB. For converting from other formats, you have one choice: Calibre. It's a big complex GUI program for organizing your ebook library and a whole bunch of other things I would never want to do, and it has a ton of prerequisites, like Qt4. But the important thing is that it comes with a small Python script called ebook-convert.

ebook-convert has no built-in help -- it takes lots of options, but to find out what they are, you have to go to the ebook-convert page on Calibre's site. But here's all you typically need

ebook-convert --authors "Mark Twain" --title "Huckleberry Finn" infile.pdf huckfinn.epub

ebook-convert infile.pdf huckfinn.epub --authors "Mark Twain" --title "Huckleberry Finn"

Pretty easy; the only hard part is remembering that it's --authors and not --author.

Calibre (and ebook-convert) can take lots of different input formats, not just PDF. If you're converting ebooks, you need it. I wish ebook-convert was available by itself, so I could run it on a server; I took a quick stab at separating it, but even once I separated out the Qt parts it still required Python libraries not yet available on Debian stable. I may try again some day, but for now, I'll stick to running it on desktop systems.

Editing EPUB books: Sigil

But we're not quite done yet. Calibre and ebook-convert do a fairly good job, but they're not perfect. When I tried converting my GIMP book from a PDF, the chapter headings were a mess and there was no table of contents. And of course I wanted the cover page to be right, instead of the default Calibre image. I needed a way to edit it.

EPUB is an XML format, so in theory I could have fixed this with a text editor, but I wanted to avoid that if possible.

And I found Sigil. Wikipedia claims it's the only application that can edit EPUB books.

There's no sigil package in Ubuntu (though Arch has one), but it was very easy to install from the sigil website.

And it worked beautifully. I cleaned up the font problems at the beginnings of chapters, added chapter breaks where they were missing, and deleted headings that didn't belong. Then I had Sigil auto-generate a table of contents from headers in the document. I was also able to fix the title and put the real book cover on the title page.

It all worked flawlessly, and the ebook I generated with Sigil looks very nice and has working navigation when I view it in FBReaderJ (it's still too big for Aldiko to handle). Very impressive. If you've ever wanted to generate your own ebook, or edit one you already have, you should definitely check out Sigil.

Android tablet as an ebook/RSS reader

I've been searching for years for something to replace the aging and unsupported Palm platform. I've been using Palms for many years to read daily news feeds; first on the proprietary Avantgo service, but mostly using the open source Plucker.

I don't normally have access to a network when I'm reading -- I might be a passenger in a car or train, in a restaurant, standing in line at the market, or in the middle of the Mojave desert. So I run a script once a day on a network-connected computer to gather up a list of feeds, package it up and transfer it to the mobile device, so I have something to read whenever I find spare time.

For years I used Sitescooper on the host end to translate HTML pages into a mobile format, and eventually became its primary maintainer. But that got cumbersome, and I wrote a simpler RSS feed reader, feedme.

But on the reader side, that still left me buying old PalmOS Clies on ebay. Surely there was a better option?

I've been keeping an eye on ebook readers and tablets for a while now. But the Plucker reader has several key features not available in most ebook reader apps:

1. An easy, open-source way of automatically translating RSS and HTML pages into something the reader can understand;
2. Delete documents after you've read them, without needing to switch to a separate application;
3. Random access to document, e.g. jump to the beginning or end, or 60% in;
4. Follow links: nearly all RSS sites, whether news sites or blogs, are set up as an index page with links to individual story pages;
5. Save external links if you click on them while offline, so you can fetch them later.

Most modern apps seem to assume either (a) that you'll be reading only books packaged commercially, or (b) that you're reading web pages and always have a net connection. Which meant that I'd probably have to roll my own; and that pointed to Android tablets rather than dedicated ebook readers.

Android as a reader

All the reviews I read pointed to Aldiko as the best e-reader on Android, so I installed it first thing. And indeed, it's a wonderful reader. The font is beautiful, and you can adjust size and color easily, including a two-click transition between configurable "day" and "night" schemes. It's easy to turn pages (something surprisingly difficult in most Android apps, since the OS seems to have no concept of "Page down"). It's easy to import new documents and easy to delete them after reading them.

So how about those other requirements? Not so good. Aldiko uses epub format, and it's possible (after much research) to produce those using ebook-convert, a command-line script you can get as part of the huge Calibre package. Alas, Calibre requires all sorts of extraneous packages like Qt even if you're never going to use the GUI; but once you get past that, the ebook-convert script works pretty well.

Except that links don't work, much. Sometimes they do, but mostly they do nothing. I don't know if this is a problem with Calibre's ebook-convert, Aldiko's reader, or the epub format itself, but you can't rely on links from the index page actually jumping anywhere. Aldiko also doesn't have a way to jump to a set point, so once you're inside a story you can't easily go back to the title page (sometimes BACK works, sometimes it doesn't).

And of course there's no way to save external links for later.

So Aldiko is a good book reader, but it wouldn't solve my feed-reading problem.

And that meant I had to write my own reader, and it was time to delve into the world of Android development. And it was surprisingly easy ... which I'll cover in a separate post. For now, I'll skip ahead and ruin the punch line by saying I have a lovely little feed-reading app, and my Archos and Android are working out great.

Archos 5 Android Tablet review

For the past couple weeks I've been using a small Android tablet, an Archos 5. I use it primarily as an ebook and RSS feed reader (more about that separately), though of course I've played with assorted games and other apps too.

I've been trying to wait for the slew of cheap Android tablets the media assure us is coming out any day now. Except "any day now" never turns into "now". And I wanted something suitable for reading: small enough to fit in a jacket pocket and hold in one hand, yet large enough to fit a reasonable amount of text on the screen. A 4-5-inch screen seemed ideal.

There's nothing in the current crop fitting that description, but there's a year-old model, the Archos 5. It has a 4.8-inch screen, plus some other nice hardware like GPS. And it seems to have a fair community behind it, at archosfans.com.

I have the 16G flash version. I've had it for a couple of weeks now and I'm very happy so far. I'm not sure I'd recommend it to a newbie (due to the Android Marketplace's ban on tablets -- see below), but it's a lovely toy for someone fairly tech savvy.

My review turned out quite long, too long for a blog post. So if you're interested in the details of what's good and what's bad, you'll find the details in my Archos 5 Android Tablet review.