Shallow Thoughts : tags : palm

Feedme 0.7

I've been enjoying my Android tablet e-reader for a couple of months now ... and it's made me realize some of the shortcomings in FeedMe. So of course I've been making changes along the way -- quite a few of them, from handling multiple output file types (html, plucker, ePub or FictionBook) to smarter handling of start, end and skip patterns to a different format of the output directory.

It's been fairly solid for a few weeks now, so it's time to release ... FeedMe 0.7.

Tags: RSS, python, programming, palm
[ 22:32 Jan 31, 2011    More programming | permalink to this entry | ]

The cure for unbeamable PalmOS files

Okay, I know Dave and I are probably the last two people on the face of the earth who use PalmOS. But we still do, to read news and ebooks prepared with Plucker, only there was one big frustration: none of our Plucker files were beamable. So if I downloaded a really interesting article and wanted to share it with Dave, I couldn't (unless we went back to our desktop machines and transferred it that way.) The Palm just said Unhandled exception, error code = 5395.

I tried all sorts of things in the feedme code that calls Plucker -- adding --beamable, moving it earlier or later in the argument list, trying other arguments. --beamable is supposed to affect the "copy protect bit", but checking on the Palm devices confirmed the copy protect bit wasn't set, and still beaming didn't work.

And I just stumbled on the answer tonight, and since it's not documented anywhere, I must document it in case somewhere, there's someone else who still uses PalmOS and Plucker struggling with this problem.

It turns out that a document with a colon in the name can't be beamed. On any PalmOS device we've tried, regardless of generation or manufacturer. This of course isn't documented anywhere I can find.

So Fri: BBC World News isn't beamable. But simply give it a colonoscopy -- rename it to Fri BBC World News -- and beaming works great.

Sheesh! Ain't debugging grand?

Tags: palm, plucker
[ 20:43 Feb 12, 2010    More tech | permalink to this entry | ]

Un-unicode: translating web pages to plain ASCII

It's not that I'm a dumb provincial American, really!

I mean, okay, I am a dumb provincial American. But not completely. I know about Unicode, I know what UTF-8 and ISO-8859-1 and -15 are, I even know how to type Spanish characters like ñ and á in email (at least in Ubuntu; I can't seem to make it work in Gentoo).

The real problem is PalmOS -- I've never found any way to create Plucker files for my Palm that display anything beyond the standard ASCII character set. (I'm not clear whether to blame that on Palm or Plucker. Doesn't matter.)

So when I use a program like Sitescooper or my new FeedMe RSS reader to read daily news on my Palm, I'm forever seeing lines like this:

the weather phenomenon known as ÅoEl Ni€oÅq is

It's tiresome to try to read stuff like that.

Strangely, I've found no libraries to do this, in any language. There are lots of ways to translate from one character encoding into another -- but no way to degrade from nonASCII characters to the nearest ASCII equivalent. Googling finds lots of people asking for them -- I'm far from the only one who wants this. There are various partial hacks, but nothing ready-to-go.

Oh, well, welcome to the programming world. Time to roll my own. I started from some nice tricks I picked up in the web discussions I found, and ended up with something reasonably compact. Of course, the table of fallback characters will grow.

But my ace in the hole, this time, is that my little function has a way of logging errors. When it sees a character it doesn't recognize, it can log the character code to a file, making it easy to add a translation for that character. That was always the problem with similar hacks I'd attempted to add to mutt or plucker or sitescooper in the past: figuring out each new character and what its intended meaning was, so I could add it to the translation table.

Here it is: ununicode.

Call it like this:

import ununicode

ununicode.toascii(str, errfilename=os.path.join("/path/to/errfile"))

There's also a minimal test script provided (which will also grow with time as I accumulate good samples).

Tags: unicode, i18n, charsets, ascii, palm
[ 20:48 Oct 21, 2009    More programming | permalink to this entry | ]

Gathering RSS files for a Palm PDA: FeedMe

For years I've been reading daily news feeds on a series of PalmOS PDAs, using a program called Sitescooper that finds new pages on my list of sites, downloads them, then runs Plucker to translate them into Plucker's open Palm-compatible ebook format.

Sitescooper has an elaborate series of rules for trying to get around the complicated formatting in modern HTML web pages. It has an elaborate cache system to figure out what it's seen before. When sites change their design (which most news sites seem to do roughly monthly), it means going in and figuring out the new format and writing a new Sitescooper site file. And it doesn't understand RSS, so you can't use the simplified RSS that most sites offer. Finally, it's no longer maintained; in fact, I was the last maintainer, after the original author lost interest.

Several weeks ago, bma tweeted about a Python RSS reader he'd hacked up using the feedparser package. His reader targeted email, not Palm, but finding out about feedparser was enough to get me started. So I wrote FeedMe (Carla Schroder came up with the all-important name).

I've been using it for a couple of weeks now and I'm very happy with the results. It's still quite rough, of course, but it's already producing better files than Sitescooper did, and it seems more maintainable. Time will tell.

Of course it needs to be made more flexible, adjusted so that it can produce formats besides Plucker, and so on. I'll get to it.

And the only site I miss now, because it doesn't offer an RSS feed, is Linux Planet. Maybe I'll find a solution for that eventually.

Tags: RSS, python, programming, palm, plucker
[ 21:08 Oct 20, 2009    More programming | permalink to this entry | ]

Reading Palm Datebook files on Linux

Over the years, I've kept a few sets of records in the Datebook app on my PalmOS PDA -- health records and such. I've been experimenting with a few python plotting packages (pycha, CairoPlot and a few others) and I wanted to try plotting one of my Datebook databases.

Not so fast. It seems that it's been a year or more since I last crunched any of this data -- and in the time since then, pilot-link has bumped its version numbers and is now shipping libpisock.so.9 instead of .8.

So what? Well, the problem is that Linux hasn't offered any way to read Palm Datebook files for years. The pilot-link package offered on most distros used to include a program called pilot-datebook, but it was deleted from the source several years ago. Apparently it was hard to maintain.

Back when it first disappeared, I built the previous version of the source, stuck the pilot-datebook binary in ~/bin/linux and have been using it ever since. Which worked fine -- until libpisock.so.8 was no longer there. (Linking .9 to .8 didn't work either.) This is all the more ironic because I don't need pilot-datebook to talk to the PDA with libpisock -- all I want to do is parse the format of a file I've already uploaded.

Off to hunt for an old version of the source. I started at pilot-link.org, but gave up after a while -- they don't seem to have source there except for the latest couple of versions, nor do they have any documentation. Ironically, in their FAQ the very first question is "How can I read the databook entries from a Palm backup?" but the FAQ page is broken and the "answer" is actually another unrelated FAQ question.

Anyway, no help there. I tried googling for old tarballs but there doesn't seem to be anything like archive.org for source code. All I found was the original pilot-datebook page, with a tarball that you insert into a copy of pilot-link 0.9.5 then modify the Makefile. Might work but that's really old.

So I fell back on old distributions. I guessed that Ubuntu Dapper was old enough that it might still have pilot-datebook. So I went to the Dapper pilot-link source and downloaded the source tarball (curiously, they don't offer src debs -- you have to download the tarball and patches separately).

Of course, it doesn't build on Ubuntu Jaunty. It had various entertaining errors ranging from wanting a mysterious tcl.m4 file not present in the code ... to not being able to find <iostream.h< because all the C++ stdlib files have recently been renamed to remove the .h ... to a change in the open() system call where I needed to add permissions argument for O_CREAT.

But I did get it working! So now I have a pilot-datebook program that builds and runs on Ubuntu Jaunty, and parses my DatebookDB.pdb file.

Since I bet I'm not the only one in the world who occasionally wants to read a Palm Datebook file, I've put my working version of the source here: pilot-link_0.11.8.jaunty.tar.gz.

After the usual configure and make, if all you want is pilot-datebook, cd src/pilot-datebook then copy both pilot-datebook and the directory .libs to wherever you want to install them.

And yeah, it would be better to write a standalone program that just parsed the format. But it's hard to justify that for what's essentially a dead platform. The real solution is to quit using a Palm for this, import the data into some common format and keep it on my Linux workstation from now on.

Tags: palm, linux, programming, patch
[ 12:39 Aug 31, 2009    More linux | permalink to this entry | ]

Feisty Fawn: The Adventure Continues, with the Visor Driver

When we left off, I had just found a workaround for my Feisty Fawn installer problems and had gotten the system up and running.

By now, it was late in the day, time for my daily Sitescooper run to grab some news to read on my Treo PDA. The process starts with making a backup (pilot-xfer -s). But pilot-xfer failed because it couldn't find the device, /dev/ttyUSB1. The system was seeing the device connection -- dmesg said

[ 1424.598770] usb 5-2.3: new full speed USB device using ehci_hcd and address 4
[ 1424.690951] usb 5-2.3: configuration #1 chosen from 1 choice

"configuration #1"? What does that mean? I poked around /etc/udev a bit and found this rule in rules.d/60-symlinks.rules:

# Create /dev/pilot symlink for Palm Pilots
KERNEL=="ttyUSB*", ATTRS{product}=="Palm Handheld*|Handspring *|palmOne Handheld", \
             SYMLINK+="pilot"

Oh, maybe they were calling it /dev/pilot1? But no, there was nothing matching /dev/*pilot*, just as there was nothing matching /dev/ttyUSB*.

But this time googling led me right to the bug, bug 108512. Turns out that for some reason (which no one has investigated yet), feisty doesn't autoload the visor module when you plug in a USB palm device the way other distros always have. The temporary workaround is sudo modprobe visor; the long-term workaround is to add visor to /etc/modules.

On the subject of Feisty's USB support, though, I do have some good news to report.

My biggest motivation for upgrading from edgy was because USB2 had stopped working a few months ago -- bug 54419. I hoped that the newer kernel in Feisty might fix the problem.

So once I had the system up and running, I plugged my trusty hated-by-edgy MP3 player into the USB2 hub, and checked dmesg. It wasn't working -- but the error message was actually useful. Rather than obscure complaints like end_request: I/O error, dev sde, sector 2033440 or device descriptor read/64, error -110 or 3:0:0:0: rejecting I/O to dead device it had a message (which I've since lost) about "insufficient power". Now that's something I might be able to do something about!

So I dug into my bag o' cables and found a PS/2 power adaptor that fit my USB2 hub, plugged it in, plugged the MP3 player into the hub, and voila! it was talking on USB2 again.

Tags: linux, ubuntu, udev, palm, pda, usb
[ 21:10 May 13, 2007    More linux | permalink to this entry | ]