Shallow Thoughts

Powertop BOF at Linuxworld 2007

The best thing at Linuxworld was the Powertop BOF, despite the fact that it ended up stuck in a room with no projector. The presenter, Arjan van de Ven, coped well with the setback and managed just fine.

The main goal of Powertop is to find applications that are polling or otherwise waking the CPU unnecessarily, draining power when they don't need to. Most of the BOF focused on "stupid stuff": programs that wake up too often for no reason. Some examples he gave (many of these will be fixed in upcoming versions of the software):

• gnome-screensaver checked every 2 sec to see if the mouse moved (rather than using the X notification for mouse move);
• gnome volume checked 10 times a second whether the volume has changed;
• gnome-clock woke up once a second to see if the minute had rolled over, rather than checking once a minute;
• firefox in an ssl layer polled 10 times a second in case there was a notification;
• the gnome file monitor woke up 40 times a second to check a queue even if there was nothing in the queue;
• evolution woke up 10 times a second;
• the fedora desktop checked 10 times a second for a smartcard;
• gksu used a 10000x/sec loop (he figures someone mistook milliseconds/microseconds: this alone used up 45 min on one battery test run)
• Adobe's closed-source flash browser plugin woke up 2.5 times a second, and acroread had similar problems (this has been reported to Adobe but it's not clear if a fix is coming any time soon).

And that's all just the desktop stuff, without getting into other polling culprits like hal and the kernel's USB system. The kernel itself is often a significant culprit: until recently, kernels woke up once a millisecond whether they needed to or not. With the recent "tickless" option that appeared in the most recent kernel, 2.6.22, the CPU won't wake up unless it needs to.

A KDE user asked if the KDE desktop was similarly bad. The answer was yes, with a caveat: Arjan said he gave a presentation a while back to a group of KDE developers, and halfway through, one of the developers interrupted him when he pointed out a problem to say "That's not true any more -- I just checked in a fix while you were talking." It's nice to hear that at least some developers care about this stuff! Arjan said most developers responded very well to patches he'd contributed to fix the polling issues.

(Of course, those of us who use lightweight window managers like openbox or fvwm have already cut out most of these gnome and kde power-suckers. The browser issues were the only ones that applied to me, and I certainly do notice firefox' polling: when the laptop gets slow, firefox is almost always the culprit, and killing it usually brings performance back.)

As for hardware, he mentioned that some linux LCD drivers don't really dim the backlight when you reduce brightness -- they just make all the pixels darker. (I've been making a point of dimming my screen when running off batteries; time to use that Kill-A-Watt and find out if it actually matters!) Wireless cards like the ipw100 use a lot of power even when not transmitting -- sometimes even more than when they're transmitting -- so turning them off can be a big help. Using a USB mouse can cut as much as half an hour off a battery. The 2.6.23 kernel has lots of new USB power saving code, which should help. Many devices have activity every millisecond, so there's lots of room to improve.

Another issue is that even if you get rid of the 10x/sec misbehavers, some applications really do need to wake up every second or so. That's not so bad by itself, but if you have lots of daemons all waking up at different times, you end up with a CPU that never gets to sleep.

The solution is to synchronize them by rounding the wakeup times to the nearest second, so that they all wake up at about the same time, and the CPU can deal with them all then go back to sleep. But there's a trick: each machine has to round to a different value. You don't want every networking application on every machine across the internet all waking up at once -- that's a good way to flood your network servers. Arjan's phrase: "You don't want to round the entire internet" [to the same value].

The solution is a new routine in glib: timeout_add_seconds. It takes a hash of the hostname (and maybe other values) and uses that to decide where to round timeouts for the current machine. If you write programs that wake up on a regular basis, check it out. In the kernel, round_jiffies does something similar.

After all the theory, we were treated to a demo of powertop in action. Not surprisingly, it looks a bit like top. High on the screen is summary information telling you how much time your CPU is spending in the various sleep states. Getting into the deeper sleep states is generally best, but it's not quite that simple: if you're only getting there for short periods, it takes longer and uses more power to get back to a running state than it would from higher sleep states.

Below that is the list of culprits: who is waking your CPU up most often? This updates every few seconds, much like the top program. Some of it's clear (names of programs or library routines); other lines are more obscure if you're not a kernel hacker, but I'm sure they can all be tracked down.

At the bottom of the screen is a geat feature: a short hint telling you how you could eliminate the current top offender (e.g. kill the process that's polling). Not only that, but in many cases powertop will do it for you at the touch of a key. Very nice! You can try disabling things and see right away whether it helped.

Arjan stepped through killing several processes and showing the power saving benefits of each one. (I couldn't help but notice, when he was done, that the remaining top offender, right above nautilus, was gnome-power-manager. Oh, the irony!)

It's all very nifty and I'm looking forward to trying it myself. Unfortunately, I can't do that on the laptop where I really care about battery life. Powertop requires a kernel API that went in with the "tickless" option, meaning it's in 2.6.22 (and I believe it's available as a patch for 2.6.21). My laptop is stuck back on 2.6.18 because of an IRQ handling bug (bug 7264). Powertop also requires ACPI, which I have to disable because of an infinite loop in kacpid (bug 8274, Ubuntu bug 75174). It's frustrating to have great performance tools like powertop available, yet not be able to use them because of kernel regressions. But at least I can experiment with it on my desktop machine.

Tags: linux, conferences, linuxworld, laptops, gnome
[ 13:06 Aug 12, 2007    More linux | permalink to this entry ]

Desktop Suspend!

My laptop has always been able to sleep (suspend to RAM), one way or another, but I had never managed it on a desktop machine. Every time I tried running something like apm -s, apm -S, echo 3 >/sys/power/state, or Ubuntu's /etc/acpi/sleep.sh, the machine would sleep nicely, then when I resumed it would come up partway then hang, or would simply boot rather than resuming.

Dave was annoyed by it too: his Mac G4 sleeps just fine, but none of his Linux desktops could. And finally he got annoyed enough to spend half a day playing with different options. With what he learned, both he and I now have desktops that can suspend to RAM (his under Debian Sarge, mine under Ubuntu Edgy).

One step was to install hibernate (available as a deb package in both Sarge and Edgy, but distros which don't offer it can probably get it from somewhere on suspend2.net). The hibernate program suspends to disk by default (which is what its parent project, suspend2, is all about) but it can also suspend to RAM, with the following set of arcane arguments:

hibernate -v 4 -F /etc/hibernate/ram.conf

(the -v 4 adds a lot of debugging output; remove it once you have things working).

Though actually, in retrospect I suspect I didn't need to install hibernate at all, and Ubuntu's /etc/acpi/sleep.sh script would have done just as well, once I'd finished the other step:

Fiddle with BIOS options. Most BIOSes have a submenu named something like "Power Management Options", and they're almost always set wrong by default (if you want suspend to work). Which ones are wrong depends on your BIOS, of course. On Dave's old PIII system, the key was to change "Sleep States" to include S3 (S3 is the ACPI suspend-to-RAM state). He also enabled APM sleep, which was disabled by default but which works better with the older Linux kernels he uses under Sarge.

On my much newer AMD64 system, the key was an option to "Run VGABIOS if S3 Resume", which was turned off by default. So I guess it wasn't re-enabling the video when I resumed. (You might think this would mean the machine comes up but doesn't have video, but it's never as simple as that -- the machine came up with its disk light solid red and no network access, so it wasn't just the screen that was futzed.)

Such a simple fix! I should have fiddled with BIOS settings long ago. It's lovely to be able to suspend my machine when I go away for a while. Power consumption as measured on the Kill-a-Watt goes down to 5 watts, versus 3 when the machine is "off" (desktop machines never actually power off, they're always sitting there on standby waiting for you to press the power button) and about 75 watts when the machine is up and running.

Now I just have to tweak the suspend scripts so that it gives me a new desktop background when I resume, since I've been having so much fun with my random wallpaper script.

Later update: Alas, I was too optimistic. Turns out it actually only works about one time out of three. The other two times, it hangs after X comes up, or else it initially reboots instead of resuming. Bummer!

Tags: linux, laptops, suspend, ubuntu
[ 10:07 Apr 12, 2007    More linux | permalink to this entry ]

Ubuntu "Breezy Badger"

I installed the latest Ubuntu Linux, called "Breezy Badger", just before leaving to visit family over the holidays. My previous Ubuntu attempt on this machine had been rather unstable (probably not Ubuntu's fault -- 2.6 kernels and this laptop don't get along very well) but Ubuntu seems to have some very sharp kernel developers, so I was curious to see whether there'd been progress.

Installation: Didn't go well. I had most of the same problems I'd had installing Hoary to this laptop (mostly due to the installer assuming that a CDROM and network must remain connected throughout the install, something that's impossible on a laptop where both of those functions require sharing the single PCMCIA port). The Breezy installer has the additional "feature" that it tends to hang if you change things like the CDROM while the install is in progress, trashing everything and forcing you to restart from the beginning. (Filed bug 20443.)

Networking: But eventually I found a sequence that let me get a network-less Breezy onto the laptop, and I'm happy to report that Breezy's built-in networking tools were able to add networking after the first boot (something that hadn't worked in Hoary). Well, admittedly I did have to add a script, /etc/hotplug/pci/3c59x, to call ifup when my cardbus network card is plugged in; but every other distro needs that too, and Breezy is the first 2.6-based distro which correctly calls the script every time.

Suspend: Once up and running, Breezy shows impressive laptop savvy. Like Hoary, it can suspend either to disk or to RAM; unlike Hoary, it can do this without my needing to hack any config files except to uncomment the line enabling suspend to RAM in /etc/default/acpi-support. It does print various error messages on stdout when it resumes from sleep or hibernate, but that's a minor issue.

Not only that, but it restores both network and usb when resuming from suspend (on hoary I had to hack some of the suspend scripts to make that work).

(Kernel flakiness: Well, mostly it suspends fine. Unplugging a usb mouse at the wrong time still causes a kernel hang. That's a 2.6 bug, not an Ubuntu-specific problem. And the system also tends to hang and need to be power cycled about one time out of five when exiting X; perhaps it's an Xorg bug.)

Ironically, my "safe" partition on this laptop (a much- modified Debian sarge) mysteriously stopped seeing PCMCIA on the first day away from home, so I ended up using Breezy for the whole trip and giving it a good workout.

Hal: One problem Breezy shares with Hoary is that every few seconds, the hald daemon makes the hard drive beep and whir. Unlike Hoary, which had an easy solution, Breezy ignores the storage_media_check_enabled and storage_automount_enabled hints. The only way I found to disable the beeping was to kill hald entirely by renaming /usr/sbin/hald (it's not called from /etc/init.d, and I never did find out who was starting it so I could disable it). Removing hald seems to have caused no ill effects; at least, hotplug of pcmcia and usb still works, as do udev rules. (Filed bug 21238.

Udev: Oh, about those udev rules! Regular readers may recall that I had some trouble with Hoary regarding udev choking on multiple flash card readers which I solved on my desktop machine with a udev rule that renames the four fixed, always present devices. But with a laptop, I don't have fixed devices; I wanted a setup that would work regardless of what I plugged in. That required a new udev rule. Here's the rule that worked for me: in /etc/udev/permissions.rules, change

BUS=="scsi", KERNEL=="sd[a-z]*", PROGRAM="/etc/udev/scripts/removable.sh %k 'usb ieee1394'", RESULT="1", MODE="0640", GROUP="plugdev"

to

BUS=="scsi", KERNEL=="sd[a-z]*", NAME{all_partitions}="%k", MODE="0640", GROUP="plugdev"

Note that this means that whatever scripts/removable.sh does, it's not happening any more. That doesn't seem to have caused any problem, though. (Filed bug 21662 on that problem.)

Conclusion: Overall, Breezy is quite impressive and required very little tweaking before it was usable. It was my primary distro for two weeks while travelling; I may switch to it on the desktop once I find a workaround for bug 352358 in GTK 2.8 (which has been fixed in gnome cvs, but that doesn't make it any less maddening when using the buggy version).

Tags: linux, ubuntu, laptops
[ 21:43 Jan 04, 2006    More linux | permalink to this entry ]

Ubuntu "Hoary Hedgehog" on a Vaio SR17 Laptop

I've been experimenting with Ubuntu's second release, "Hoary Hedgehog" off and on since just before it was released.

Overall, I'm very impressed. It's quite usable on a desktop machine; but more important, I'm blown away by the fact that Ubuntu's kernel team has made a 2.6 acpi kernel that actually works on my aging but still beloved little Vaio SR17 laptop. It can suspend to RAM (if I uncomment ACPI_SLEEP in /etc/defaults/acpi-support), it can suspend to disk, it gets power button events (which are easily customizable: by default it shuts the machine down, but if I replace powerbtn.sh with a single line calling sleep.sh, it suspends), it can read the CPU temperature. Very cool.

One thing didn't work: USB stopped working when resuming after a suspend to RAM. It turned out this was a hotplug problem, not a kernel problem: the solution was to add calls to /etc/init.d/hotplug stop and /etc/init.d/hotplug start in the /etc/acpi/sleep.sh script. Problem solved (except now resuming takes forever, as does booting; I need to tune that hotplug startup script and get rid of whatever is taking so long).

Sonypi (the jogdial driver) also works. It isn't automatically loaded (I've added it to /etc/modules), and it disables the power button (so much for changing the script to call sleep.sh), a minor annoyance. But when loaded, it automatically creates /dev/sonypi, so I don't have to play the usual guessing game about which minor number it wanted this time.

Oh, did I mention that the Hoary live CD also works on the Vaio? It's the first live linux CD which has ever worked on this machine (all the others, including rescue disks like the Bootable Business Card and SuperRescue, have problems with the Sony PCMCIA-emulating-IDE CD drive). It's too slow to use for real work, but the fact that it works at all is amazing.

I have to balance this by saying that Ubuntu's not perfect. The installer, which is apparently the Debian Sarge installer dumbed down to reduce the number of choices, is inconsistent, difficult, and can't deal with a networkless install (which, on a laptop which can't have a CD drive and networking at the same time because they both use the single PCMCIA slot, makes installation quite tricky). The only way I found was to boot into expert mode, skip the network installation step, then, after the system was up and running (and I'd several times dismissed irritating warnings about how it couldn't find the network, therefore "some things" in gnome wouldn't work properly, and did I want to log in anyway?) I manually edited /etc/network/interfaces to configure my card (none of Ubuntu's built-in hardware or network configuration tools would let me configure my vanilla 3Com card; presumably they depend on something that would have been done at install time if I'd been allowed to configure networking then). (Bug 2835.)

About that expert mode: I needed that even for the desktop, because hoary's normal installer doesn't offer an option for a static IP address. But on both desktop and laptop this causes a problem. You see, hoary's normal mode of operation is to add the first-created user to the sudoers list, and then not create a root account at all. All of their system administration tools depend on the user being in the sudoers file. Fine. But someone at ubuntu apparently decided that anyone installing in expert mode probably wants a root account (no argument so far) and therefore doesn't need to be in the sudoers file. Which means that after the install, none of the admin tools work; they just pop up variants on a permission denied dialog. The solution is to use visudo to add yourself to /etc/sudoers. (Bugs 7636 and 9832.)

Expert mode also has some other bugs, like prompting over and over for additional kernel modules (bug 5999).

Okay, so nothing's perfect. I'm not very impressed with Hoary's installer, though most of its problems are inherited from Sarge. But once it's on the machine, Hoary works great. It's a modern Debian-based Linux that gets security upgrades (something Debian hasn't been able to do, though they keep making noises about finally releasing Sarge). And there's that amazing kernel. Now that I have the hotplug-on-resume problem fixed, I'm going to try using it as the primary OS on the laptop for a while, and see how it goes.

Tags: linux, ubuntu, laptops, vaio
[ 16:29 Jun 03, 2005    More linux | permalink to this entry ]

Good article on laptop Linux

Linux Magazine has a good article by Jonathan A. Zdziarski on Linux on the laptop: Ten power tools for the mobile Linux user. He gives hints such as what services to turn off for better power management, and how to configure apmd to turn off those services automatically; finding modules to drive various types of wireless internet cards; various ways of minimizing disk activity; and even making data calls with a mobile phone.

Lots of good information in there!

Tags: linux, laptops
[ 19:51 Oct 07, 2004    More linux | permalink to this entry ]

swsusp working, sort of

I got swsusp working on blackbird. Re-reading the kernel documentation with a night's sleep behind me revealed that at the very end of swsusp.txt, there's code for a C program to suspend non-ACPI machines. Worked fine! swsusp in 2.6.7 doesn't quite resume X properly (I had to ctrl-alt-FN back and forth a few times before I saw my X screen) but it's progress, anyway.

Tags: linux, suspend, laptops
[ 20:00 Jun 27, 2004    More linux | permalink to this entry ]

PenLUG: Linux on Laptops

Great talk at PenLUG by Chander Kant of Linux Certified, on Linux on Laptops. It was much the same talk he gave a month ago at SVLUG, except at PenLUG he got to give his talk without being pestered by a bunch of irritating flamers, and it was possible for people to ask real questions and get answers.

The biggest revelation: "Suspend to Disk" in the 2.6 kernel is actually ACPI S4 suspend, and won't do anything on a machine that doesn't support ACPI S4. Why can't they say that in the help? Sheesh!

Tags: linux, laptops
[ 23:00 Jun 24, 2004    More linux | permalink to this entry ]