Shallow Thoughts : : Nov

Bitlbee tips: hide timestamps, and share configurations

I recently set up bitlbee on a new machine. Things worked fine, mostly -- but here are a couple of tweaks that should speed things up when moving a bitlbee configuration to another machine.

Sharing configuration files

I get so tired of re-authenticating with Twitter every time I move to a new machine, disk, or distro. And it turns out you don't have to!

Your configuration is in /var/lib/bitlbee/yournick.xml, and you can copy that file to other machines and it will work just fine -- with one caveat.

Assuming you have bitlbee set up to run as a user named "bitlbee", rather than as root (the default is bitlbee), you'll need to make sure the /var/lib/bitlbee/yournick.xml file is owned by the bitlbee user. If you just copy it as root, you'll get an error like "The nick is (probably) not registered". You can fix it with chown bitlbee /var/lib/bitlbee/yournick.xml

Hiding timestamps

On the new machine, every new tweet had a timestamp added. Timestamps look like this:

<NatGeo> [20:26:24] Elusive marbled cat filmed: http://t.co/oOo3Xa81 
<OliverSacks> [20:28:09] Happy Thanksgiving week! Check out Dr. Sacks's new blog post about Gabby Giffords and what he is reading now: http://t.co/kZCTx53h 

These timestamps add clutter and make the lines too long. But googling for bitlbee timestamps only gets a lot of people who couldn't figure out how to suppress them and ended up writing scripts to hide them in various IRC clients.

Turns out bitlbee has a perfectly straightforward way to hide them. Go to your &bitlbee tab -- you know, the one that always opens first that you have to close manually every time after it finally opens the #twitter tab (I wish I could find a way to auto-close it!) and type:

set display_timestamps 'false'

That's it! Timestamps-b-gone.

You can see more bitlbee variables by typing set in the &bitlbee tab, or get help by typing help there.

Configuring extlinux's auto-update on Debian

A few days ago, I wrote about how to set up and configure extlinux (syslinux) as a bootloader. But on Debian or Ubuntu, if you make changes to files like /boot/extlinux/extlinux.conf directly, they'll be overwritten.

The configuration files are regenerated by a program called extlinux-update, which runs automatically every time you update your kernel. (Specifically, it runs from the postinst script of the linux-base package: you can see it in /var/lib/dpkg/info/linux-base.postinst.)

So what's a Debian user to do if she wants to customize the menus, add a splash image or boot other operating systems?

First, if you decide you really don't want Debian overwriting your configuration files, you can change disable updates by editing /etc/default/extlinux. Just be aware you won't get your boot menu updated when you install new kernels -- you'll have to remember to update them by hand.

It might be worth it: the automatic update is nearly as annoying as the grub2 updater: it creates two automatic entries for every kernel you have installed. So if you have several distros installed, each with a kernel or two in your shared /boot, you'll get an entry to boot Debian Squeeze with the Ubuntu Oneiric kernel, one for Squeeze with the Natty kernel, one for Squeeze with the Fedora 16 kernel ... as well as entries for every kernel you have that's actually owned by Debian. And then for each of these, you'll also get a second entry, to boot in recovery mode. If you have several distros installed, it makes for a very long and confusing boot menu!

It's a shame that the auto-updater doesn't restrict itself to kernels managed by the packaging system, which would be easy enough to do. (Wonder if they would accept a patch?) You might be able to fudge something that works right by setting up symlinks so that the only readable kernels actually live on the root partition, so Debian can't read the kernels from the other distros. Sounds a bit complicated and I haven't tried it. For now, I've turned off automatic updating on my system.

But if your setup is simpler -- perhaps just one Debian or one Ubuntu partition plus some non-Linux entries such as BSD or Windows -- here's how to set up Debian-style automatic updating and still keep all your non-Linux boot entries and your nice menu customizations.

Debian automatic updates and themes

First, take a quick look at /etc/default/extlinux and customize anything there you might need, like the names of the kernels, kernel boot parameters or timeout. See man extlinux-update for details.

For configuring menu colors, image backgrounds and such, you'll need to make a theme. You can see a sample theme by installing the package syslinux-themes-debian -- but watch out. If you haven't configured apt not to pull in suggested packages, that may bring back grub or grub-legacy, which you probably don't want.

You can make a theme without needing that package, though. Create a directory /usr/share/syslinux/themes/mythemename (the extlinux-update man page claims you can put a theme anywhere and specify it by its full path, but it lies). Create a directory called extlinux inside it, and make a file with everything you want from extlinux.conf. For example:

default 0
prompt 1
timeout 50

ui vesamenu.c32
menu title Welcome to my Linux machine!
menu background mysplash.png
menu color title 1;36 #ffff8888 #00000000 std
menu color unsel 0    #ffffffff #00000000 none
menu color sel   7    #ff000000 #ffffff00 none

include linux.cfg
menu separator
include themes/mythemename/other.cfg

Note that last line: you can include other files from your theme. For instance, you can create a file called other.cfg with entries for other partitions you want to boot:

label oneiric
menu label Ubuntu Oneiric Ocelot
kernel /vmlinuz-3.0.0-12-generic
append initrd=/initrd.img-3.0.0-12-generic root=UUID=c332b3e2-5c38-4c50-982a-680af82c00ab ro quiet

label fedora
menu label Fedora 16
kernel /vmlinuz-3.1.0-7.fc16.i686
append initrd=/initramfs-3.1.0-7.fc16.i686.img root=UUID=47f6b1fa-eb5d-4254-9fe0-79c8b106f0d9 ro quiet

menu separator

LABEL Windows
KERNEL chain.c32
APPEND hd0 1

Of course, you could have a debian.cfg, an ubuntu.cfg, a fedora.cfg etc. if you wanted to have multiple distros all keeping their kernels up-to-date. Or you can keep the whole thing in one file, theme.cfg. You can make a theme as complex or as simple as you like.

How to install extlinux (syslinux) as a bootloader

When my new netbook arrived, I chose Debian Squeeze as the first Linux distro to install, because I was under the impression it still used grub1, and I wanted to avoid grub2. I was wrong -- Squeeze uses grub2. Uninstalling grub2, installing grub-legacy and running grub-install and update-grub didn't help; it turns out even in Debian's grub-legacy package, those programs come from grub2's grub-common package.

What a hassle! But maybe it was a blessing in disguise -- I'd been looking for an excuse to explore extlinux as a bootloader as a way out of the grub mess.

Extlinux is one of the many spinoffs of syslinux -- the bootloader used for live CDs and many other applications. It's not as commonly used as a bootloader for desktops and laptops, but it's perfectly capable of that. It's simple, well tested and has been around for years. And it supports the few things I want out of a bootloader: it has a simple configuration file that lives on the /boot partition; it can chain-load Windows, on machines with a Windows partition; it even offers pretty graphical menus with image backgrounds.

Since there isn't much written about how to use extlinux, I wrote up my experiences along with some tips for configuring it. It came out too long for a blog article, so instead I've made it its own page: How to install extlinux (syslinux) as a bootloader.

New trails, and new PyTopo 1.1 release

A new trail opened up above Alum Rock park! Actually a whole new open space preserve, called Sierra Vista -- with an extensive set of trails that go all sorts of interesting places.

Dave and I visit Alum Rock frequently -- we were married there -- so having so much new trail mileage is exciting. We tried to explore it on foot, but quickly realized the mileage was more suited to mountain bikes. Even with bikes, we'll be exploring this area for a while (mostly due to not having biked in far too long, so it'll take us a while to work up to that much riding ... a combination of health problems and family issues have conspired to keep us off the bikes).

Of course, part of the fun of discovering a new trail system is poring over maps trying to figure out where the trails will take us, then taking GPS track logs to study later to see where we actually went.

And as usual when uploading GPS track logs and viewing them in pytopo, I found some things that weren't working quite the way I wanted, so the session ended up being less about studying maps and more about hacking Python.

In the end, I fixed quite a few little bugs, improved some features, and got saved sites with saved zoom levels working far better.

Now, PyTopo 1.0 happened quite a while ago -- but there were two of us hacking madly on it at the time, and pinning down the exact time when it should be called 1.0 wasn't easy. In fact, we never actually did it. I know that sounds silly -- of all releases to not get around to, finally reaching 1.0? Nevertheless, that's what happened.

I thought about cheating and calling this one 1.0, but we've had 1.0 beta RPMs floating around for so long (and for a much earlier release) that that didn't seem right.

So I've called the new release PyTopo 1.1. It seems to be working pretty solidly. It's certainly been very helpful to me in exploring the new trails. It's great for cross-checking with Google Earth: the OpenCycleMap database has much better trail data than Google does, and pytopo has easy track log loading and will work offline, while Google has the 3-D projection aerial imagery that shows where trails and roads were historically (which may or may not correspond to where they decide to put the new trails). It's great to have both.

Anyway, here's the new PyTopo.

Robot/Arduino Hackathon in Redwood City

Yesterday Dave and I attended a "Robot Hackathon" in Redwood City, part of a "nerd new year" 11/11/11 celebration.

What a fun event! O'Reilly/Make generously sponsored hardware, so everybody got an Arduino Uno as well as a Grid Kit, a couple of sheets of cardboard pre-scored in a grid to encourage cutting and bending into various robot shapes, and a couple of motors. Tools were provided -- there were big bins of wire, soldering irons, glue guns, box cutters and other odds and ends.

People of all ages were there having fun -- lots of kids there with their parents, as well as adults of all ages and experience levels. The adults were mostly fiddling with the Arduinos; the younger kids mostly eschewed the electronics and concentrated on building cool monsters and vehicles with the cardboard kits. I saw some great models -- penguins, squid, tanks, cherrypickers, many-legged bugs. Wish I'd thought to take a camera along.

No instructions were provided, but I didn't see many people looking lost; there were enough people there with experience in Arduino, soldering and the other tools who were happy to help others. I was able to help some folks with their Arduino projects while I worked on copying a grid penguin model from a nearby table. There were lots of friendly volunteers (I think they were from Robotics for Fun) wandering around offering advice, in between building a cardboard city out of GridKits. There was even pizza, from host Pizza & Pipes.

I had to leave before finishing my penguin, but it does have me inspired to do more with Arduinos and motors. I had a blast, both fiddling with my own projects and helping other people get started with Arduinos, and I'm pretty sure everybody else in the room was having an equally good time. Thanks, sponsors O'Reilly/Make, Robotics for Fun, The Grid Kit, Mozilla, MS and Andreessen Horowitz!

Controlling motors from an Arduino

One point of confusion: everybody got their Arduino LEDs blinking quickly, but then how do you control a motor? I wasn't sure about that either, but one of the volunteers found a printout of sample code, and it turned out to be simplicity itself: just plug in to one of the digital outputs, and set it to HIGH when you want the motor to spin.

There was much discussion at my table over how to reverse a motor. I suggested you could plug the two motor leads into two digital pins, then set one HIGH and the other LOW; then to reverse the motor, just swap the HIGH and LOW pin. Nobody believed me, and there were a lot of fervent assertions that there was some magic difference between a pin being LOW and a real ground. I should have coded it up then to demonstrate ... I wish I had, rather than spending so much time hot-gluing penguin parts.

Now that I'm home and it's too late, here's an example of how to reverse a motor by plugging in to two digital outputs.

// Arduino basic motor control

#define DELAYTIME   1000      // milliseconds

int motorPins[2] = { 5, 6 };  // plug the motor leads into these pins
int direction = 0;            // toggle between 0 and 1

void setup()
{
    pinMode(motorPins[0], OUTPUT);
    digitalWrite(motorPins[0], LOW);
    pinMode(motorPins[1], OUTPUT);
    digitalWrite(motorPins[1], LOW);
}

// Alternate between two directions and motionless.
// Assume we start with both pins low, motor motionless.
void loop()
{
    delay(DELAYTIME);
    digitalWrite(motorPins[direction], HIGH);
    delay(DELAYTIME);
    digitalWrite(motorPins[direction], LOW);
    direction = !direction;
}

Incidentally, powering robot motors directly from an Arduino is generally a bad idea. It's okay for testing or for small servos, but if you're going to be driving a truck with the motors or otherwise pulling a lot of current, it's better to use a separate power supply for the motors rather than trying to power them from the Arduino. The easy way is to buy something like this Motor/Stepper/Servo Shield that plugs in to the top of your Arduino and has its own power supply.

Arduino Uno on the command line

As I've written before, I prefer to do my Arduino hacking from the command line ... but I didn't know the settings needed for an Uno, and avrdude is quite particular about settings and can't auto-configure anything. So I ended up using the standard Arduino IDE while I was at the event ... there was theoretically wifi at the site, but it wasn't working for me so I had to wait 'til I got home to search for solutions.

Now I've uploaded a new, more flexible version of my Arduino Makefile with presets for the Uno, Duemilanove and Diecimila models: Makefile-0022-v3.

Increase Your ... Pancakes?

This coupon showed up on a Safeway receipt.

Everyone I've showed it to has the same reaction as I did: stacks of pancakes! Oh, wait, the headline says ... oh, I see, I guess those are supposed to be coins.

I'm not sure what the lesson is ... maybe that you should show your ad to a few other people before publishing it.

Or maybe the program is actually for cafe owners looking to increase their breakfast sales ...

The Stanford online Machine Learning class

In case you haven't been following it, Stanford's computer science department began a grand experiment in online learning early this month: free, upper division college courses, given online and open to the whole world. There are three classes offered: Artificial Intelligence, Machine Learning and Introduction to Databases.

They've sparked an incredible response: exact numbers don't seem to be available, but rumor is that AI had about 130,000 enrolees, while ML had about 70,000. (Nobody seems to have published numbers for DB.) Update, a day later: @seemsArtless tweets that ML currently has 87,000 registered users.

Why so much interest? Surely there are lots of places to get free information (like wikipedia) and even course lectures (like MIT). And there are plenty of places to take classes for relatively low cost, like local junior colleges or ed2go.

What's different about the Stanford classes is that they cover advanced material, in far more depth than you'd find at a junior college or typical online site. They offer graded homework so you can see how you're doing, and there are other students taking the class at the same time, so if you get stuck, there are all sorts of discussion groups you can turn to. It's one thing to read a textbook or watch a video by yourself; I find a class much more helpful, and judging by the response to the Stanford classes, I'm not alone in that.

I agonized over whether to take AI or Machine Learning. They both sounded so interesting! Since I couldn't decide, I initially signed up for both, figuring I'd drop one if the load was too great. By the end of the second week, I'd settled on Machine Learning. I was starting to dread the AI class flash quizzes -- which didn't always work right, but made it hard to proceed until you'd answered the question right even if you couldn't see the question -- and to feel frustrated about the lectures, which clearly were meant as a jumping off point for students to go do their own outside reading.

On the other hand, I was really enjoying the Machine Learning lectures, and looking forward to each new one. And the real kicker: Machine Learning includes programming assignments, so students can implement the algorithms Professor Ng talks about in the lectures.

What's great about Machine Learning

Andrew Ng's video lectures are wonderfully clear, well paced and full of interesting content.

He uses a lot of graphs to help students visualize what's going on geometrically, rather than just relying on the equations. (Better yet, in the programming exercises he shows us how to create those graphs for ourselves.)

And he's great about flagging certain portions as possibly review (you can skip this lecture if you already know linear algebra) or advanced (this is some extra background for people who know calculus, but you can skip it and still do fine in the course).

The technology is simpler than that used in the AI course. If you have a slow net connection or travel a lot, you can download the lectures as mp4 files and watch them offline. You can download lecture slides as a PDF or PPT. Review questions (graded) are handled with simple HTML forms. All very simple, well-tested technology, and it works great. I've had no problems accessing the servers, submitting homework or anything else -- very impressive!

But the heart of the course is the programming exercises. ML is taught in GNU octave, a framework and language for numerical computing and matrix operations. Students aren't absolutely required to use octave, but it's highly recommended: Professor Ng says he's found that students learn much faster that way. Sounds good to me, and octave looks like a useful skill, well worth acquiring. I'm having fun learning it.

The programming exercises come with a lot of scaffold code plus a few files with "Your code goes here". The actual amount of coding isn't large. But I'm finding that it does the job: it forces me to make sure I understand the matrix operations discussed in the lectures. And at the end, you come out with something that's actually useful! From the first few weeks, I have linear and logistic regression code that I could use to analyze and visualize all sorts of datasets. Now, at the end of week 4, we're halfway through writing a neural network to recognize handwritten numerals from image data. How cool is that?

Suggestions for improvement

The class is a huge success. Who would have thought that you could teach something this advanced on such a huge scale, so effectively?

I have only a couple of small suggestions -- ways the class could be even better next time.

• An errata page. In week 3, there was an error in the lecture and notes, a - instead of a +, that made one part of programming ex. 2 quite a bit trickier than it would otherwise have been. If I hadn't noticed that the slides used + in some places and - in others, I might never have gotten that part of the assignment working. Lots of other people found that too, and there were discussions in the Q&A forum ... but you wouldn't find it without coming up with clever search terms.
• The Q&A forum would be so much more useful if it was organized by topic, and/or by week. There are some great discussions there, but the only way of getting to them is by searching for the right terms. There's no way to browse discussions, see how people are doing on assignment 3, or look for errata and similar warnings. It would help make the class more of a community, more like a real in-person class.

Hope for future expansion

I mentioned my suggestions because I fervently hope there is a "next time". These classes are a great service, and I hope the huge response isn't putting too much burden on the instructors.

"Common wisdom" among providers of online classes seems to be that there's no demand outside of enrolled university students for hard courses, courses with prerequisites, and especially courses that involve (shudder) math. Just look at the offerings from any online courseware or adult ed program -- they're long on art appreciation and "Introduction to MS Word", short on physics and econometrics. Even the for-pay online degree mills concentrate on humanities and business, not technical subjects.

Stanford's experiment has proven that "common wisdom" is wrong -- that tens of thousands of students will jump at the chance to take highly technical, mathematical courses. I'd love to see the model expanded to other subjects, such as statistics, economics, physics, geology and climate science.

And, yes, there is money to be made here. If this many people will take a free class, wouldn't quite a few of them be willing to pay? Most couldn't afford $1000 like UC Extension classes -- but how about $100, comparable to other online education classes? Would people pay more if you offered college credit?

Online education providers, take note! There's a large, underserved market for scientific and technical classes out here in the long tail.