Shallow Thoughts : tags : open source

Akkana's Musings on Open Source Computing, Science, and Nature.

Mon, 16 May 2011

How to make a patch that might get accepted into Ubuntu, using bzr

Update and warning: My bzr diff was not accepted. It turns out this particular package doesn't accept that format. Apparently different packages within Ubuntu require different types of patches, and there's no good way to find out besides submitting one type of patch and seeing if it's rejected or ignored. In the end, I did get a patch accepted, and will write up separately how that patch was generated.

The process of submitting bugs and patches to Ubuntu can be deeply frustrating. Even if you figure out how to fix a bug and attach a patch, the patch can sit in Launchpad for years with no attention, as this ubuntu-devel-discuss thread attests.

The problem is that there are a lot of bugs and not enough people qualified to review patches and check them in. To make things easier for the packagers, sometimes people are told to "make a debdiff" or "make a ppa". But it's tough to find good instructions on how to do these things. There are partial instructions at Contributing and on the Packaging Guide -- but both pages are aimed at people who want to become regular packagers of new apps, not someone who just has one patch for a specific bug, and they're both missing crucial steps. Apparently there's a new and better packaging guide being written, but it's not publically available yet.

These days, Bazaar (bzr), not debdiff, is considered the best way to make a patch easy for Ubuntu developers to review. With a lot of help from #ubuntu-women, and particularly Maco (THANKS!), I worked through the steps to submit a patch I'd posted to bug 370735 two years ago for gmemusage. Here's what I needed to do.

Set up the tools

First, install some build tools you'll need, if you don't already have them: 

sudo apt-get install bzr bzr-builddeb pbuilder

You will also need a Launchpad account:

and connect bzr to your Launchpad account: 
bzr whoami "Firstname Lastname <yourname@example.com>"
bzr launchpad-login your-acct

Check out the code

Create a directory where you'll do the work: 

mkdir pkgname
cd pkgname

Check out the source from bzr: 

bzr branch lp:ubuntu/pkgname pkgname

Make a bzr branch for your fixes. It's probably a good idea to include the bug number or other specifics in the branch name: 

bzr branch pkgname pkgname-fix-bugnum
cd pkgname-fix-bugnum

Now you can apply the patch, e.g. patch <../mypatch.diff, or edit source files directly.

Make a package you can test

Making a package from a bzr directory requires several steps.

Making a source package is easy: 

bzr bd -S -- -uc -us
This will show up as ../pkgname_version.dsc.

But if you want something you can install and test, you need a binary package. That's quite a bit more trouble to generate. You'll be using pbuilder to create a minimal install of Ubuntu in a chroot environment, so the build isn't polluted by any local changes you have on your own machine.

First create the chroot: this takes a while, maybe 10 minutes or so, or a lot longer if you have a slow network connection. You'll also need some disk space: on my machine it used 168M in /var/cache (plus more for the next step). Since it uses /var/cache, it needs sudo to write there: 

sudo pbuilder --create natty

Now build a .deb binary package from your .dsc source package: 

sudo pbuilder --build ../pkgname_version.dsc
pbuilder will install a bunch of additional packages, like X and other libraries that are needed to build your package but weren't included in the minimal pbuilder setup.

And then once it's done with the build, it removes them all again. Apparently there's a way to make it cache them so you'll have them if you need to build again, but I'm not sure how.

pbuilder --build gives lots of output, but none of that output tells you where it's actually creating the .deb. Look in /var/cache/pbuilder/result for it.

And now you can finally try installing it: 

sudo  dpkg -i /var/cache/pbuilder/result/pkgname_blahblah.deb

You can now test your fix, and make sure you fixed the problem and didn't break anything else.

Check in your bzr branch

Once you're confident your fix is good. it's time to check it in.

Make a new changelog entry: 

dch -i
This will open your editor of choice, where you should explain briefly what you changed and why. If it's a fix for a Launchpad bug, list the bug number like this: (LP: #370735).

If you're proposing a fix for an Ubuntu that's already released, you also need to add -proposed to the release name in the top line in the changelog, e.g.: 

pkgname (0.2-11ubuntu1) natty-proposed; urgency=low

Also, pay attention to that ubuntu1 part of the version string if the entry prior to yours doesn't include "ubuntu" in the version. If you're proposing a change to a stable release, change that to ubuntu0.1; if it's for the current development release, it's okay to leave it at ubuntu1 (more details on this Packaging page).

Finally, you can check it in to your local repository: 

debcommit
and push it to Launchpad: 
bzr push lp:~yourname/ubuntu/natty/pkgname/pkgname-fix-bugnum

Notify possible sponsors

You'll want to make sure your patch gets on the sponsorship queue, so someone can review it and check in the fix. 

bzr lp-open
(For me, this opened chromium even though firefox is my preferred browser. To use Firefox, I had to: sudo update-alternatives --config x-www-browser first. Boo chromium for making itself default without asking me.)

You should now have a launchpad page open in your browser. Click on "Propose for merging into another branch" and include a description of your change and why it should be merged. This, I'm told, notifies potential sponsors who can review your patch and approve it for check-in.

Whew! That's a lot of steps. You could argue that it's harder to prepare a patch for Ubuntu than it was to fix the bug in the first place. Stay tuned ... I'll let you know when and if my patch actually gets approved.

Tags: , ,
[ 14:38 May 16, 2011    More linux | permalink to this entry | comments ]

Sat, 06 Feb 2010

Making "Citizen Science" compelling

I had the opportunity to participate in a focus group on NASA's new "citizen science" project, called Moon Zoo, with a bunch of other fellow lunatics, amateur astronomers and lunar enthusiasts.

Moon Zoo sounds really interesting. Ordinary people will analyze high-resolution photos of the lunar surface: find out how many boulders and craters are there. I hope it will also include more details like crater type and size, rilles and so forth, though that wasn't mentioned. These are all tasks that are easy for a human and hard for a computer: perfect for crowdsourcing. Think Galaxy Zoo for the moon. The resulting data will be used for planning future lunar missions as well as for general lunar science.

It sounds like a great project and I'm excited about it. But I'm not going to write about Moon Zoo today -- it doesn't exist yet (current estimate is mid-March), though there is a preliminary PDF. Instead, I want to talk about some of the great ideas that came out of the focus group.

The primary question: How do we get people -- both amateur astronomers and the general public, people of all ages -- interested in contributing to a citizen science project like Moon Zoo?

Here are some of the key ideas:

Make the data public

This was the most important point, echoed by a lot of participants. Some people felt that many of the existing "citizen science" projects project the attitude "We want something from you, but we're not going to give you anything in return." If you use crowdsourcing to create a dataset, make it available to the crowd.

Opening the data has a lot of advantages:

Projects like Wikipedia and Open Street Map, as well as Linux and the rest of the open source movement, show how much an open data model can inspire contributions.

Give credit to individuals and teams

People cited the example of SETI@Home, where teams of contributors can compete to see who's contributed the most. Show rankings for both individuals and groups, so they can track their progress and maybe get a bit competitive with other groups. Highlight groups and individuals who contribute a lot -- maybe even make it a formal competition and offer inexpensive prizes like T-shirts or mugs.

A teenaged panel member had the great suggestion of making buttons that said "I'm a Moon Zookeeper." Little rewards like that don't cost much but can really motivate people.

Offer an offline version

They wanted to hear ideas for publicizing Moon Zoo to groups like our local astronomy clubs.

I mentioned that I've often wanted to spread the word about Galaxy Zoo, but it's entirely a web-based application and when I give talks to clubs or school groups, web access is never an option. (Ironically, the person leading the focus group had planned to demonstrate Galaxy Zoo to us but couldn't get connected to the wi-fi at the Lawrence Hall of Science.)

Projects are so much easier to evangelize if you can download an offline demo.

And not just a demo, either. There should be a way to download a real version, including a small data set. Imagine if you could grab a Moon Zoo pack and do a little classifying whenever you got a few spare minutes -- on the airplane or train, or in a hotel room while traveling.

Important note: this does not mean you should write a separate Windows app for people to download. Keep it HTML, Javascript and cross platform so everyone can run it. Then let people download a local copy of the same web app they run on your site.

Make sure it works on phones and game consoles

Lots of people use smartphones more than they use a desktop computer these days. Make sure the app runs on all the popular smartphones. And lots of kids have access to handheld web-enabled game consoles: you can reach a whole new set of kids by supporting these platforms.

Offer levels of accomplishment, like a game

Lots of people are competitive by nature, and like to feel they're getting better at what they're doing. Play to that: let users advance as they get more experienced, and give them the option of doing harder projects. "I'm up to level 7 in Moon Zoo!"

Use social networking

Facebook. Twitter. Nuff said.

Don't keep results a secret

Quite a few scientific publications have arisen out of Galaxy Zoo -- yet although most of us were familiar with Galaxy Zoo, few of us knew that. Why so secretive? They should be trumpeting achievements like that.

How many times have you volunteered for a survey or study, then wondered for years afterward how the results came out? Researchers never contact the volunteers when the paper is finally published. It's frustrating and demotivating; it makes you not want to volunteer again. Lots of us sign up because we're curious about the science -- but that means we're also curious about the results.

With citizen science projects, this is particularly easy. Set up a mailing list or forum (or both) to discuss results and announce when papers are published. Set up a Twitter account and a Facebook group to announce new papers to anyone who wants to follow. This is the age of Web 2.0, folks -- there's no excuse for not communicating.

I don't know if NASA will listen to our ideas. But I hope they do. Moon Zoo promises to be a terrific project ... and the more of these principles they follow, the more dedicated volunteers they'll get and that will make the project even better.

Tags: , , ,
[ 19:25 Feb 06, 2010    More science/astro | permalink to this entry | comments ]

Sun, 06 Sep 2009

Using apt-file to track down build errors

Someone was asking for help building XEphem on the XEphem mailing list. It was a simple case of a missing include file, where the only trick is to find out what package you need to install to get that file. (This is complicated on Ubuntu, which the poster was using, by the way they fragment the X developement headers into a maze of a xillion tiny packages.)

The solution -- apt-file -- is so simple and easy to use, and yet a lot of people don't know about it. So here's how it works.

The poster reported getting these compiler errors: 

ar rc libz.a adler32.o compress.o crc32.o uncompr.o deflate.o trees.o zutil.o inflate.o inftrees.o inffast.o
ranlib libz.a
make[1]: Leaving directory `/home/gregs/xephem-3.7.4/libz'
gcc -I../../libastro -I../../libip -I../../liblilxml -I../../libjpegd -I../../libpng -I../../libz -g -O2 -Wall -I../../libXm/linux86 -I/usr/X11R6/include   -c -o aavso.o aavso.c
In file included from aavso.c:12:
../../libXm/linux86/Xm/Xm.h:56:27: error: X11/Intrinsic.h: No such file or directory
../../libXm/linux86/Xm/Xm.h:57:23: error: X11/Shell.h: No such file or directory
../../libXm/linux86/Xm/Xm.h:58:23: error: X11/Xatom.h: No such file or directory
../../libXm/linux86/Xm/Xm.h:59:34: error: X11/extensions/Print.h: No such file or directory
In file included from ../../libXm/linux86/Xm/Xm.h:60,
                 from aavso.c:12:
../../libXm/linux86/Xm/XmStrDefs.h:1373: error: expected `=', `,', `;', `asm' or `__attribute__' before `char'
In file included from ../../libXm/linux86/Xm/Xm.h:60,
                 from aavso.c:12:
../../libXm/linux86/Xm/XmStrDefs.h:5439:28: error: X11/StringDefs.h: No such file or directory
In file included from ../../libXm/linux86/Xm/Xm.h:61,
                 from aavso.c:12:
../../libXm/linux86/Xm/VirtKeys.h:108: error: expected `)' before `*' token
In file included from ../../libXm/linux86/Xm/Display.h:49,
                 from ../../libXm/linux86/Xm/DragC.h:48,
                 from ../../libXm/linux86/Xm/Transfer.h:44,
                 from ../../libXm/linux86/Xm/Xm.h:62,
                 from aavso.c:12:
../../libXm/linux86/Xm/DropSMgr.h:88: error: expected specifier-qualifier-list before `XEvent'
../../libXm/linux86/Xm/DropSMgr.h:100: error: expected specifier-qualifier-list before `XEvent'
How do you go about figuring this out?

When interpreting compiler errors, usually what matters is the *first* error. So try to find that. In the transcript above, the first line saying "error:" is this one: 

../../libXm/linux86/Xm/Xm.h:56:27: error: X11/Intrinsic.h: No such file or directory

So the first problem is that the compiler is trying to find a file called Intrinsic.h that isn't installed.

On Debian-based systems, there's a great program you can use to find files available for install: apt-file. It's not installed by default, so install it, then update it, like this (the update will take a long time): 

$ sudo apt-get install apt-file
$ sudo apt-file update
Once it's updated, you can now find out what package would install a file like this: 
$  apt-file search Intrinsic.h
libxt-dev: /usr/include/X11/Intrinsic.h
tendra: /usr/lib/TenDRA/lib/include/x5/t.api/X11/Intrinsic.h

In this case two two packages could install a file by that name. You can usually figure out from looking which one is the "real" one (usually the one with the shorter name, or the one where the package name sounds related to what you're trying to do). If you're stil not sure, try something like apt-cache show libxt-dev tendra to find out more about the packages involved.

In this case, it's pretty clear that tendra is a red herring, and the problem is likely that the libxt-dev package is missing. So apt-get install libxt-dev and try the build again.

Repeat the process until you have everything you need for the build.

Remember apt-file if you're not already using it. It's tremendously useful in tracking down build dependencies.

Tags: , , , ,
[ 10:25 Sep 06, 2009    More linux | permalink to this entry | comments ]

Fri, 12 Jun 2009

A Table of Closed versus Open Formats

My last Toastmasters speech was on open formats: why you should use open formats rather than closed/proprietary ones and the risks of closed formats.

To make it clearer, I wanted to print out handouts people could take home summarizing some of the most common closed formats, along with open alternatives.

Surely there are lots of such tables on the web, I thought. I'll just find one and customize it a little for this specific audience.

To my surprise, I couldn't find a single one. Even openformats.org didn't have very much.

So I started one: Open vs. Closed Formats. It's far from complete, so I hope I'll continue to get contributions to flesh it out more.

And the talk? It went over very well, and people appreciated the handout. There's a limit to how much information you can get across in under ten minutes, but I think I got the point across. The talk itself, such as it is, is here: Open up!

Tags: , , ,
[ 10:37 Jun 12, 2009    More tech | permalink to this entry | comments ]

Sat, 03 Jan 2009

OpenStreetMap mapping parties

Latest obsession: mapping with OpenStreetMap.

Last month, OpenStreetMap and its benefactor company CloudMade held a "mapping party" in Palo Alto. I love maps and mapping (I wrote my own little topographic map viewer when I couldn't find one ready-made) and I've been wanting to know more about the state of open source mapping. A mapping party sounded perfect.

The party was a loosely organized affair. We met at a coffeehouse and discussed basics of mapping and openstreetmap. The hosts tried to show us newbies how OSM works, but that was complicated by the coffeehouse's wireless net being down. No big deal -- turns out the point of a mapping party is to hand out GPSes to anyone who doesn't already have one and send us out to do some mapping.

I attached myself to a couple of CloudMade folks who had some experience already and we headed north on a pedestrian path. We spent a couple of hours walking urban trails and marking waypoints. Then we all converged on a tea shop (whose wireless worked a little better than the one at the coffeehouse, but still not very reliably) for lunch and transfer of track and waypoint files.

This part didn't work all that well. It turned out the units we were using (Garmin Legend HCx) can transfer files in two modes, USB mass storage (the easy way, just move files as if from an external disk) or USB Garmin protocol (the hard way: you have to use software like gpsbabel, or the Garmin software if you're on Windows). And in mass storage mode, you get a file but the waypoints aren't there.

The folks running the event all had Macs, and there were several Linux users there as well, but no Windows laptops. By the time the Macs both had gpsbabel downloaded over the tea shop's flaky net, it was past time for me to leave, so I never did get to see our waypoint files. Still, I could see it was possible (and one of the Linux attendees assured me that he had no trouble with any of the software; in fact, he found it easier than what the Mac people at the party were going through).

But I was still pretty jazzed about how easy OpenStreetMap is to use. You can contribute to the maps even without a GPS. Once you've registered on the site, you just click on the Edit tab on any map, and you see a flash application called "Potlatch" that lets you mark trails, roads or other features based on satellite images or the existing map. I was able to change a couple of mismarked roads near where I live, as well as adding a new trail and correcting the info on an existing one for one of the nearby parks.

If you prefer (as, I admit, I do) to work offline or don't like flash, you can use a Java app, JOSM, or a native app, merkaartor. Very cool! Merkaartor is my favorite so far (because it's faster and works better in standalone mode) though it's still fairly rough around the edges. They're all described on the OSM Map Editing page.

Of course, all this left me lusting after a GPS. But that's another story, to be told separately.

Tags: , ,
[ 12:00 Jan 03, 2009    More mapping | permalink to this entry | comments ]

Fri, 04 Jul 2008

Learning about Firefox 3 extensions

Oops! Right after I posted that last entry, I discovered that my little kitfox extension wasn't working as well as I'd thought. And the more I hacked it, the less well it worked, and the more I discovered was missing, like a chrome.manifest file (which firefox 2 hadn't seemed to need).

Eventually some very helpful folks on #extdev pointed me to Ted Mielczarek's excellent Extension Wizard. Give it some details about your extension (its name and version, your name, and a couple things you might want like a toolbar button, a prefs panel and a context menu) and it generates a zipped directory containing a bare bones extension, even including niceties like internationalized strings.

Even better, your new extension skeleton includes a readme that tells you how to leave the extension expanded while you work on it. That's quite a bit easier than building the XPI file and installing it each time.

So kitfox has a 0.3 version (in the unlikely event that anybody besides me wants it).

There's a project called fizzypop to develop and extend useful Mozilla dev tools like the Extension Wizard ... watch that space for more details.

Tags: , ,
[ 20:12 Jul 04, 2008    More tech/web | permalink to this entry | comments ]