Shallow Thoughts : : May

Command-line Arduino development

I've been doing more Arduino development lately. But I don't use the Arduino Java development environment -- programming is so much easier when you have a real editor, like emacs or vim, and key bindings to speed everything up.

I've found very little documentation on how to do command-line Arduino development, and most of the Makefiles out there are old and no longer work. So I've written up a tutorial. It ended up too long for a blog post, so I've made it a separate article:

Command-line Arduino development.

Tags: hardware, arduino, cmdline, programming, maker
[ 14:45 May 30, 2011    More programming | permalink to this entry | ]

PII 2011: Privacy, Identity and Innovation conference

I'm just now finding time to write up some of my notes from PII: Privacy, Identity and Innovation last week. PII was a fabulous conference, fascinating and well run. It was amazing to be in a room with so many people who actually care about these issues.

There were two days of speakers and panels, most of them in the same room, which surprised me: usually conferences have multiple tracks to give you lots of choices. But I ended up being glad for the single track. Almost all of the speakers and panels were interesting, including some I might not have chosen on my own. I had my laptop along with some projects I figured I'd work on during the boring sessions -- but that never happened. I didn't even get time during lunch or breaks -- too many fascinating people to talk to in the hallways.

Then Saturday was "Privacy Camp", a less formal "unconference" full of round-table discussions about some of the issues raised during the regular conference. Conversations were lively and informative.

Usually after a conference I have a couple of suggestions for improvement. For PII I really can't come up with anything. The website was very informative (they even had detailed parking information), everything ran pretty close to on time, rooms were easy to find, they had an A/V crew recording everything, and wow, that Thursday lunch. Plus: Best. Badgeholders. Ever. Great job, PII organizers!

And I couldn't help but notice the gender balance: a third of the speakers were women, and by my rough count-of-nearby-tables, women were close to 40% of the attendees. At a tech conference! That's about double most conferences. Most of the women I talked to were entrepreneurs, many with a history of successful startups already, plus some researchers and a few developers.

The opening talk was worth getting up early for: Julia Angwin, the journalist who wrote the Wall Street Journal's excellent "What they know" articles, discussing the research that led to to the series and what they've learned from it.

Later, once the panel discussions got started, the biggest takeaway from the conference was a question mentioned early on: "Were users surprised? When were they surprised?" Sometimes companies say they care about privacy, but haven't thought much about user expectations. Asking yourself this question is a great test of how well you're really protecting user privacy.

Privacy statements don't work

One of the panels I wouldn't have chosen that was unexpectedly interesting discussed web site privacy statements. First, M. Ryan Calo of the Stanford privacy center presented a study on user behavior with regard to privacy statements. They tried several different types, on websites of very different designs, to see what worked best for users. The upshot? "We couldn't test how well various privacy statements worked, because no users clicked on them. Zero."

Then Aleecia McDonald of Mozilla presented a study where they tried structuring privacy statements in different ways to make the information clearer to users. How can you improve on the "natural-language" policy you see on most websites, consisting of several pages of dense obfuscated text? They tried hierarchies where they showed the basics and let users click through to the details; interactive pages where you could expand and contract sections or mouse over a category to see more; colored tables, cute icons, the works. They found that most of the seemingly easier formats were actually worse than the long natural-language expositions no one reads.

If you make the page interactive, users won't expand the sections and won't find the important mouseovers. If you make sub-pages, users won't click through. If you use icons, users won't know what they mean. But too often, they'll end up thinking they understand, making assumptions about the details that don't match what's really in the policy. So most simplified, "user-friendly" policies are actually worse than a dense wall of text.

The only style that tested slightly better than natural-language policies was the "Nutrition label" style, where they presented several aspects of privacy with ratings for how good or bad the site was.

I felt sorry for the two panelists after Ryan and Aleecia, who were there to show off their cool hierarchical privacy statement page designs. They'd obviously put a lot of work into trying to make their policies clearer ... but we'd just been convincingly shown how ineffective such policies really are.

How to be stupid much faster

One panel discussed big data collection, and some of the ways data can be misused. Someone (Beth Givens?) related a story of a family arrested for marijuana growing after their power company's algorithms flagged them as suspicious for their heavy late-night use of power. Turns out they just had two teenagers who liked to stay up late playing video games. Terence Craig, in my favorite quote from the conference, quipped: "It used to be that it took weeks to accumulate that data. Now you can be stupid much faster."

I enjoyed a workshop given by Brian Kennish of Disconnect and Calvin Pappas of SelectOut about their projects. Disconnect arose from a chrome browser extension, Facebook Disconnect, to block Facebook tracking from widgets on third-party sites. SelectOut also arose from a chrome extension, making it easy for users to opt out of all the major advertising networks at once. The workshop turned into a lively discussion of opt-out versus do-not-track solutions, and what future directions might be.

In another workshop, Martin Ortlieb described a Google study comparing attitudes toward privacy of people in several countries. Someone in the audience asked a question about data being collected and held by government agencies versus private companies. Martin commented that attitudes in the study tended toward "I'd rather companies have my data, because then the government might regulate how it's used. If the government has it, no company's going to regulate it."

Assorted notes

Someone mentioned that Mozilla didn't seem to be taking "Do not track" very seriously, hiding it in the Advanced preferences tab, not under Privacy where you'd expect it. Why? Later we heard that Mozilla is listening to those concerns, and Firefox 5 will move Do Not Track to the Privacy tab.

Esther Dyson: "Personal data can be traded; reputation can't. Reputation is not a currency." She was responding to someone who described a business model involving trading reputation points.

M. Ryan Calo: The government doesn't need a warrant to access your webmail if it's older than 6 months, something most webmail users don't realize.

Finally, Raman Khanna observed: kids get tattoos, then when they're older they pay a lot more for laser removal services. There will be data services like that. "You were stupid when you were in college, and you put all this info online. We'll clean it up for you."

A good insight, and it reminded me of the old threat they used to give us in school (do they still say this to kids?) "This is going on your permanent record." Nobody was ever sure what this permanent record was or why anyone would want to look at it. I wonder if mine still exists somewhere?

Tags: conferences, pii2011, privacy
[ 11:32 May 27, 2011    More conferences | permalink to this entry | ]

Vim/Emacs tip: use modelines for files that need special behavior

Most of the time when I edit a text file in vim, I want lines to wrap automatically as I type, at somewhere around 70 columns wide. So I set textwidth=70 in .vimrc.

But sometimes that isn't appropriate. For instance, I have a procmail rules file where I put common spam patterns that programs like spamassassin don't seem to be able to catch. So I might have lines like:

*^Subject:.*(Ink Cartridges|Hummingbird Vine|who's who of executives|Avandia|Botox|Your Email ID|Zoosk|Best airfares on the internet|UGG Boots|police training)

... and so on -- you get the idea. I can't have lines breaking in the middle, because then the procmail rule wouldn't work. So every time I add a new phrase, I have to :set tw=0 (or one of the other umpteen ways one can tell vim not to wrap lines) first.

But you can set special behavior for one specific file by adding a special comment called a "modeline" as the first line of the file.

Procmail treats any line starting with a hash, #, as a comment, and vim recognizes # as a comment. So I can add this as the first line of the procmail file:

# vim: set tw=0:

then vim will see that and un-set that default text width I specify in .vimrc.

Vim understands most common comment styles, so it should understand lines like /* vim: set tw=0: */ and // vim: set tw=0: and ; vim: set tw=0: as well.

But to make this work I had to do one more thing: in .vimrc, I had to add

set modeline

Apparently on some versions of vim this is on by default; in others it's turned off for security reasons (someone could put an evil modeline into a file which would make your vim do something when you edited it). Definitely something to be aware of, but if you mostly edit files you created yourself on your local machine, and no one else uses your machine, it's your choice whether to worry about it.

Emacs has modelines too

Emacs has mode lines too, though it calls them Local variables lines. For instance, C++ files in Mozilla's source tend to start with:

/* -*- Mode: C++; tab-width: 2; indent-tabs-mode: nil; c-basic-offset: 2 -*- */

It's awfully handy to be able to define specific indentation style for the files within a project, making it easy for emacs users, at least, to follow your preferred coding style. If only all editors understood them!

Tags: editors, vim, emacs, tip
[ 21:26 May 25, 2011    More linux/editors | permalink to this entry | ]

Packaging Python for MeeGo (or other RPM-based distros)

Writing Python scripts for MeeGo is easy. But how do you package a Python script in an RPM other MeeGo users can install?

It turned out to be far easier than I expected. Python and Ubuntu had all the tools I needed.

First you'll need a .desktop file describing your app, if you don't already have one. This gives window managers the information they need to show your icon and application name so the user can run it. Here's the one I wrote for PyTopo: pytopo.desktop.

Of course, you'll also want a desktop icon. Most other applications on MeeGo seemed to use 48x48 pixel PNG images, so that's what I made, though it seems to be quite flexible -- an SVG is ideal.

With your script, desktop file and an icon, you're ready to create a package.

Create a setup.py file describing your package, as in the distutils simple example or the more detailed distutils setup script page. For a sample standalone script with a desktop file and icon, you can take a look at my PyTopo setup.py.

Starting from the Python setup script, Python's distutils can generate RPM or even Windows packages -- assuming you have the appropriate tools installed on your machine.

I'm on an Ubuntu (Debian-based) machine, and all the docs imply you have to be on an RPM-based distro to make an RPM. Happily, that's not true: Ubuntu has RPM tools you can install.

$ sudo apt-get install rpm

Then let Python do its thing:

$ python setup.py bdist_rpm

Python generates the spec file and everything else needed and builds a multiarch RPM that's ready to install on MeeGo. You can install it by copying it to the MeeGo device with scp dist/PyTopo-1.0-1.noarch.rpm meego@address.of.device:/tmp/. Then, as root on the device, install it with rpm -i /tmp/PyTopo-1.0-1.noarch.rpm. You're done!

To see a working example, you can browse my latest PyTopo source (only what's in SVN; it needs a few more tweaks before it's ready for a formal release). Or try the RPM I made for MeeGo: PyTopo-1.0-1.noarch.rpm. I'd love to hear whether this works on other RPM-based distros.

What about Debian packages?

Curiously, making a Debian package on Debian/Ubuntu is much less straightforward even if you're starting on a Debian/Ubuntu machine. Distutils can't do it on its own. There's a Debian Python package recipe, but it begins with a caution that you shouldn't use it for a package you want to submit. For that, you probably have to wade through the Complete Ubuntu Packaging Guide. Clearly, that will need a separate article.

Tags: python, meego, programming, rpm, redhat
[ 18:44 May 20, 2011    More programming | permalink to this entry | ]

How to make a patch that might get accepted into Ubuntu, using bzr

Update and warning: My bzr diff was not accepted. It turns out this particular package doesn't accept that format. Apparently different packages within Ubuntu require different types of patches, and there's no good way to find out besides submitting one type of patch and seeing if it's rejected or ignored. In the end, I did get a patch accepted, and will write up separately how that patch was generated.

The process of submitting bugs and patches to Ubuntu can be deeply frustrating. Even if you figure out how to fix a bug and attach a patch, the patch can sit in Launchpad for years with no attention, as this ubuntu-devel-discuss thread attests.

The problem is that there are a lot of bugs and not enough people qualified to review patches and check them in. To make things easier for the packagers, sometimes people are told to "make a debdiff" or "make a ppa". But it's tough to find good instructions on how to do these things. There are partial instructions at Contributing and on the Packaging Guide -- but both pages are aimed at people who want to become regular packagers of new apps, not someone who just has one patch for a specific bug, and they're both missing crucial steps. Apparently there's a new and better packaging guide being written, but it's not publically available yet.

These days, Bazaar (bzr), not debdiff, is considered the best way to make a patch easy for Ubuntu developers to review. With a lot of help from #ubuntu-women, and particularly Maco (THANKS!), I worked through the steps to submit a patch I'd posted to bug 370735 two years ago for gmemusage. Here's what I needed to do.

Set up the tools

First, install some build tools you'll need, if you don't already have them:

sudo apt-get install bzr bzr-builddeb pbuilder

You will also need a Launchpad account:

• set up a new account
• set up and register a GPG key
• set up and register an SSH key

and connect bzr to your Launchpad account:

bzr whoami "Firstname Lastname <yourname@example.com>"
bzr launchpad-login your-acct

Check out the code

Create a directory where you'll do the work:

mkdir pkgname
cd pkgname

Check out the source from bzr:

bzr branch lp:ubuntu/pkgname pkgname

Make a bzr branch for your fixes. It's probably a good idea to include the bug number or other specifics in the branch name:

bzr branch pkgname pkgname-fix-bugnum
cd pkgname-fix-bugnum

Now you can apply the patch, e.g. patch <../mypatch.diff, or edit source files directly.

Make a package you can test

Making a package from a bzr directory requires several steps.

Making a source package is easy:

bzr bd -S -- -uc -us

This will show up as ../pkgname_version.dsc.

But if you want something you can install and test, you need a binary package. That's quite a bit more trouble to generate. You'll be using pbuilder to create a minimal install of Ubuntu in a chroot environment, so the build isn't polluted by any local changes you have on your own machine.

First create the chroot: this takes a while, maybe 10 minutes or so, or a lot longer if you have a slow network connection. You'll also need some disk space: on my machine it used 168M in /var/cache (plus more for the next step). Since it uses /var/cache, it needs sudo to write there:

sudo pbuilder --create natty

Now build a .deb binary package from your .dsc source package:

sudo pbuilder --build ../pkgname_version.dsc

pbuilder will install a bunch of additional packages, like X and other libraries that are needed to build your package but weren't included in the minimal pbuilder setup.

And then once it's done with the build, it removes them all again. Apparently there's a way to make it cache them so you'll have them if you need to build again, but I'm not sure how.

pbuilder --build gives lots of output, but none of that output tells you where it's actually creating the .deb. Look in /var/cache/pbuilder/result for it.

And now you can finally try installing it:

sudo  dpkg -i /var/cache/pbuilder/result/pkgname_blahblah.deb

You can now test your fix, and make sure you fixed the problem and didn't break anything else.

Check in your bzr branch

Once you're confident your fix is good. it's time to check it in.

Make a new changelog entry:

dch -i

This will open your editor of choice, where you should explain briefly what you changed and why. If it's a fix for a Launchpad bug, list the bug number like this: (LP: #370735).

If you're proposing a fix for an Ubuntu that's already released, you also need to add -proposed to the release name in the top line in the changelog, e.g.:

pkgname (0.2-11ubuntu1) natty-proposed; urgency=low

Also, pay attention to that ubuntu1 part of the version string if the entry prior to yours doesn't include "ubuntu" in the version. If you're proposing a change to a stable release, change that to ubuntu0.1; if it's for the current development release, it's okay to leave it at ubuntu1 (more details on this Packaging page).

Finally, you can check it in to your local repository:

debcommit

and push it to Launchpad:

bzr push lp:~yourname/ubuntu/natty/pkgname/pkgname-fix-bugnum

Notify possible sponsors

You'll want to make sure your patch gets on the sponsorship queue, so someone can review it and check in the fix.

bzr lp-open

(For me, this opened chromium even though firefox is my preferred browser. To use Firefox, I had to: sudo update-alternatives --config x-www-browser first. Boo chromium for making itself default without asking me.)

You should now have a launchpad page open in your browser. Click on "Propose for merging into another branch" and include a description of your change and why it should be merged. This, I'm told, notifies potential sponsors who can review your patch and approve it for check-in.

Whew! That's a lot of steps. You could argue that it's harder to prepare a patch for Ubuntu than it was to fix the bug in the first place. Stay tuned ... I'll let you know when and if my patch actually gets approved.

Tags: ubuntu, bugs, open source
[ 15:38 May 16, 2011    More linux | permalink to this entry | ]

Children of the Code -- Derived Python projects

I got some fun email today -- two different people letting me know about new projects derived from my Python code.

One is M-Poker, originally based on a PyQt tutorial I wrote for Linux Planet. Ville Jyrkkä has taken that sketch and turned it into a real poker program. And it uses PySide now -- the new replacement for PyQt, and one I need to start using for MeeGo development. So I'll be taking a look at M-Poker myself and maybe learning things from it. There are some screenshots on the blog A Hacker's Life in Finland.

The other project is xkemu, a Python module for faking keypresses, grown out of pykey, a Python version of my Crikey keypress generation program. xkemu-server.py looks like a neat project -- you can run it and send it commands to generate key presses, rather than just running a script each time.

(Sniff) My children are going out into the world and joining other projects. I feel so proud. :-)

Tags: programming, python
[ 21:04 May 13, 2011    More programming | permalink to this entry | ]

Firefox 4: Fixing middlemouse.content load, and hacking jars

Mostly the transition to Firefox4 has pretty smooth. But there's been one big hassle: middlemouse content load URL doesn't work as well as it used to.

Middlemouse content load is a great Firefox feature on Linux and other Unix platforms. You see a URL somewhere that doesn't have clickable URLs -- say, a plaintext mail message, or something somebody typed in IRC. You highlight it with the mouse -- no need to Copy explicitly -- X does that automatically whenever you highlight text). Then move to the Firefox window and click the middle mouse button somewhere in the content window -- anywhere as long as it's not over a link or text input -- and Firefox goes straight to the URL you pasted.

A few silly Linux distros, like Ubuntu, disable this feature by default. You can turn it back on by going to about:config, searching for middlemouse, and setting middlemouse.contentLoadURL to true.

Except, in Firefox 4, much of the time nothing happens. In Firefox 4, contentLoadURL only works if the URL you pasted is a complete URL, like http://example.com. This is completely silly, because most of the time, if you had a complete URL, it would have been clickable already in whatever window you originally found it in. When you need contentLoadURL is when someone types "Go to example.com if you want to see this". It's also great for when you get those horrible Facebook or Stumbleupon or Google URLs like http://www.facebook.com/l/bfd4f/example.com/somelink and you want just the real link (example.com/somelink), without the added cruft.

Hacking the jar

Hooray! It turns out the offending code is in browser.js, so it's hackable without needing to recompile all of Firefox. You just need to unpack omni.jar, patch browser.js, then zip up a new omni.jar.

In other words, something like this (on Ubuntu Natty, starting in an empty directory):

$ cp /usr/lib/firefox-4.0/omni.jar ~/omni-jar.sav
$ unzip /usr/lib/firefox-4.0/omni.jar
  [ edit or patch chrome/browser/content/browser/browser.js ]
$ rm -f /tmp/new-omni.jar; zip /tmp/new-omni.jar `find .`
$ sudo cp /tmp/new-omni.jar /usr/lib/firefox-4.0/omni.jar

Getting Firefox to notice code changes

Except, as I was testing this, I discovered: I could make changes and most of the time Firefox wouldn't see them. I would put in something obvious like alert("Hello, world");, verify that the alert was really in omni.jar, run Firefox, click the middle mouse button and -- no alert. Where was Firefox getting the code it was actually running, if not from omni.jar?

I'll spare you the agonizing details of the hunt and just say that eventually I discovered that if I ran Firefox from a different profile on the same machine, I got a different result. It turns out that if you remove either of two files, extensions.sqlite and XUL.mfasl, Firefox4 will re-read the new code in omni.jar.

Removing XUL.mfasl seems to be a little safer: extensions.sqlite contains some details of which extensions are enabled. Of course, back up both files first before experimenting with removing them.

Why these files are keeping a cache of code that's already in omni.jar is anybody's guess.

The Patch: fix contentLoadURL

Oh, and the change? Mikachu came up with a cleaner fix than mine, so this is his. It accepts partial URLs like example.com and also bookmarklet keywords:

--- browser.js.sav      2011-05-07 16:40:03.672540242 -0700
+++ omni/chrome/browser/content/browser/browser.js      2011-05-08 16:29:28.943313984 -0700
@@ -10597,12 +10597,10 @@
   clipboard.replace(/\s*\n\s*/g, "");
 
   let url = getShortcutOrURI(clipboard);
-  try {
-    makeURI(url);
-  } catch (ex) {
-    // Not a valid URI.
-    return;
-  }
+  var URIFixup = Components.classes["@mozilla.org/docshell/urifixup;1"]
+                           .getService(Components.interfaces.nsIURIFixup);
+  url = URIFixup.createFixupURI(url, 1).spec;
+  // 1 is FIXUP_FLAG_ALLOW_KEYWORD_LOOKUP
 
   try {
     addToUrlbarHistory(url);

Tags: firefox, mozilla, programming, browsers
[ 20:28 May 09, 2011    More tech/web | permalink to this entry | ]

Overview of MeeGo Development Tools

Over the past week I've been playing with the MeeGo ExoPC tablet, experimenting with the various options for building programs.

One advantage MeeGo has over Android or iOS is that there are quite a lot of language and toolkit options. If you have an existing program, especially if it runs on Linux, you can probably find a way to port it to MeeGo fairly without much extra coding.

But since MeeGo is still quite new, not all of the options work quite as well as you might hope -- yet. I'm sure they'll get better, but here's what the development climate looks like now.

C++, Qt and Meego SDK

C++ and Qt are the primary supported environment on MeeGo, via the MeeGo SDK, run in an IDE called QtCreator.

The documentation turns out to be somewhat incomplete; I'll write up a howto as a separate article. But once you get it installed (much easier than installing, say, Eclipse for Android), it runs nicely. You can use normal Qt, not a specialized environment, so existing Qt programs should be quick to port, and the IDE takes care of packaging, copying the app to your remote device and starting it running. Very painless.

Testing apps locally isn't so easy. They're set up to use QEMU, which only works if your development machine has hardware virtualization. Supposedly there's a way to make this work in virtualbox (which doesn't require hardware virtualization), but the docs don't address how.

Still, testing on the target device is easy. Unfortunately, not many of my existing programs are C++ and Qt. I mostly write Python, C, and Javascript/HTML web apps.

Update: I should have mentioned that QtCreator also lets you program in QML, an XML-like language that lets you design Qt user interface and even write application code.

Web Apps under C++ and QWebView

So what about those web apps? Since the C++ SDK was so well supported, I figured, why not use a QWebView so I can run my HTML and Javascript code?

Unfortunately QWebView turns out to be tricky to use, has very few sample apps available, and so far I've been unable to get it to work under MeeGo at all.

Nokia Web RunTime

And anyway, for pure web apps, there's a kit explicitly for that: WRT, the Nokia Web RunTime. But it's pretty heavyweight: the official download and documentation is all based around Eclipse (for HTML/Javascript apps? Seriously?) and a required set of Nokia libraries that I had trouble installing.

But WRT apps are packaged according to the W3C Widget Packaging and Configuration specification -- so it's probably possible to roll your own without the big Nokia WRT package. More research required.

Python, Qt and PySide

I have a lot of Python apps, so I was very happy to see that Python comes already installed on MeeGo. (Perl is also supported; Ruby theoretically is but it's not installed by default.)

Since Qt is the officially blessed user interface, Python-Qt seems like the way to go. Except -- Python in MeeGo has no Qt bindings installed. The package that provides them is called PySide, but depending on where you look you'll either be steered toward a lengthy source compile on the MeeGo device, or a zypper install from someone's personal repository.

None of that is difficult for developers, but you can't expect users to enable experimental repositories or compile from source. Is PySide going to be a standard part of MeeGo by the time it ships to users? Nobody's saying. It makes me leery of putting much energy into Python-Qt development until I hear more.

Python-GTK

A little frustrated with the options so far, I was idly poking around an interactive Python session and typed

>>> import gtk

And it worked! Even though Qt is the supported UI toolkit and nobody talks about GTK, it's installed on MeeGo -- complete with Python bindings. It gave two deprecation warnings, but what's a little deprecation among friends?

A simple test program ran just fine. So I whipped up a starter page for PyTopo, made a .desktop file and icon for it, copied the three files over with scp -- and everything worked great.

There's no standard way yet to make a MeeGo RPM from a Python program, so that will require a little hand-fiddling, but only a little. And sadly, python-webkit-gtk isn't included, so this isn't a good solution for porting web apps.

I'll be keeping an eye on PySide and continuing to experiment with PySide, WRT and C++. But in the meantime, it's great that it's so easy to port all my existing Python-GTK programs to MeeGo.

Tags: meego, programming
[ 11:33 May 07, 2011    More programming | permalink to this entry | ]