Shallow Thoughts : : Nov

Backing up a file system

I upgraded my system to the latest Ubuntu, "Gutsy Gibbon", recently. Of course, it's always best to make a backup before doing a major upgrade. In this case, the goal was to back up my root partition to another partition on the same disk and get it working as a bootable Ubuntu, which I could then upgrade, saving the old partition as a working backup. I'll describe here a couple of silly snags I hit, to save you from making the same mistakes.

Linux offers lots of ways to copy filesystems. I've used tar in the past, with a command like (starting in /gutsy): tar --one-file-system -cf - / | tar xvf - > /tmp/backup.out but cp seemed like an easier way, so I want to try it.

I mounted my freshly made backup partition as /gutsy and started a cp -ax /* /gutsy (-a does the right thing for permissions, owner and group, and file type; -x tells it to stay on the original filesystem). Count to ten, then check what's getting copied. Whoops! It clearly wasn't staying on the original filesystem.

It turned out my mistake was that /*. Pretty obvious in hindsight what cp was doing: for each entry in / it did a cp -ax, staying on the filesystem for that entry, not on the filesystem for /. So /home, /boot, /proc, etc. all got copied. The solution was to remove the *: cp -ax / /gutsy.

But it wasn't quite that simple. It looked like it was working -- a long wait, then cp finished and I had what looked like a nice filesystem backup. I adjusted /gutsy/etc/fstab so that it would point to the right root, set up a grub entry, and rebooted. Disaster! The boot hung right after Attached scsi generic sg0 type 0 with no indication of what was wrong.

Rebooting into the old partition told me that what's supposed to happen next is: * Setting preliminary keymap... But the crucial error message was actually several lines earlier: Warning: unable to open an initial console. It hadn't been able to open /dev/console.

Now, in the newly copied filesystem, there was no /dev/console: in fact, /dev was empty. Nothing had been copied because /dev is a virtual file system, created by udev.

But it turns out that the boot process needs some static devices in /dev, before udev has created anything. Of course, once udev's virtual filesystem has been mounted on /dev, you can no longer read whatever was in /dev on the root partition in order to copy it somewhere else. But udev nicely gives you access to it, in /dev/.static/dev. So what I needed to do to get my new partition booting was: cp -ax /dev/.static/dev/ /gutsy/dev/ With that done, I was able to boot into my new filesystem and upgrade to Gutsy.

Tags: linux, ubuntu, backups
[ 23:48 Nov 30, 2007    More linux | permalink to this entry | ]

Typing accented characters (for ignorant 'murricans)

Happy Thanksgiving, everyone! Today's holiday tip involves how to type international characters.

For the online Spanish class I've been taking, so far I've been able to manage without having to type characters like ñ or á. Usually, if I need one I can find it in one of the class examples, copy it, and paste it wherever I need it. But obviously that would be tedious if I needed to type much.

I hacked up a quickie workaround: a python script that shows a set of buttons, one for each accented character I'm likely to need. Clicking a button copies that character to the clipboard, so I can now paste via mouse middleclick or ctrl-V. (I'm sure that sounds pathetic to those of you who type accented characters every day, but it's not something most US English speakers need to do. And besides, now I know how to access the X clipboard from Python-GTK -- hooray for learning new things from procrastination projects!)

Anyway, Mikael Magnusson took pity on me and explained in simple language how to use the X "Multi key" to type these characters the right way (well, a right way, anyway). Since all the online instructions I've seen have been rather complicated, here are the simple instructions for any of my fellow US monolingists who'd like to expand their horizons:

First, choose a key for the "Multi key" that you're not using for anything else. A lot of people use one of the Alt or Windows keys, but I use both of those already. What I don't use is the Menu key (that little key down by the right Ctrl key, at least on my keyboard) since not many Linux apps support it anyway.

Find the keycode for that key, by firing up xev and typing the key. For my Menu key, the keycode is 117.

Now type:

xmodmap -e "keycode 117 = Multi_key"

Now you're ready to type a sequence like: [Menu] ~ n to type an n-tilde, [Menu] ' a for an accented a, or [menu] ? ? for the upside-down question mark, in any app that supports those characters.

Of course, you don't want to type that xmodmap command every time you log in, so to make it permanent, put this in your .Xmodmap (you're on your own for figuring out whether your X environment reads .Xmodmap automatically or whether you need to tell it to run xmodmap .Xmodmap when X starts up):

keycode 117 = Multi_key

I have one final useful international input tidbit to offer: how to type Unicode characters by number. Hold ctrl+shift+U, then release U but keep holding the other two while you type a numeric sequence. (This may only work in gtk apps.) For instance, try this: hold down ctrl and shift, then type: u 2 6 6 c. Cool, huh? You can use the "gucharmap" program to find other neat sequences (hint: View->By Unicode Block otherwise you'll never find anything).

Now it's time to check the turkey. Have a good day, everyone!

Tags: linux, i18n, keyboard
[ 17:03 Nov 22, 2007    More linux | permalink to this entry | ]

A newt in the hand is worth two in the creek

We found a tiny baby newt struggling its way across the Zinfandel trail at Stevens Creek.

Really across the trail -- I didn't see it and might have stepped on it, but luck was with both of us. Dave spotted it after I passed. We stopped to admire, handle and photograph it, then set it gently off the trail so it could continue to struggle its way up the hill.

(Then rinsed our hands thoroughly -- rough-skinned newts and their cousins the California newts secrete a strong neurotoxin through their skin. It's only dangerous if you eat it. They have an interesting defensive posture -- which I've only seen in books and on the web -- showing bright colors to let an attacker know they're poisonous. Garter snakes are the only predator resistant to the toxin.)

I don't know what's at the top of the hill that's so attractive for a young newt, but evidently it's worth some effort. I hope this little one makes it there.

Tags: nature
[ 16:03 Nov 17, 2007    More nature | permalink to this entry | ]

An ironic juxtaposition

I can't stop thinking about the woman in the Chinese restaurant the other night.

It was one of those conversations you try not to overhear, but they're so loud and distracting that you just can't avoid it.

In the middle of a long declamation on conspiracy theories and politics, the man made a comment about how we're in the middle east shooting Iraqis who never hurt anyone. (I didn't say his politics were all wrong, just loud).

The woman, who had been relatively quiet up to now, interrupted, "But they hurt us in 9/11!"

In the next booth, facing away from them, my mouth dropped open. The man quickly countered that Iraq had nothing to do with 9/11, but then was off onto other topics, sharing with the room his theories on war in the middle east in general, Israel, and people trying to wipe out the Jews. This caught the woman's interest -- "They already tried that, Hitler." After a pause, she added thoughtfully, "You know, the strangest thing about that is how people there just went along with it."

That came barely a minute after the 9/11 comment. She clearly had no idea of the irony of juxtaposing the two. I wanted to turn around and say, "Perhaps they went along for the same reason that you're going along with killing hundreds of thousands of Iraqi civilians, when even the president who started the war admits that Iraq had nothing to do with 9/11?"

Tags: politics
[ 22:24 Nov 16, 2007    More politics | permalink to this entry | ]

Proposal: A Simpler Linux Installer

I spent a couple of fruitless hours today trying to install PCLinuxOS, a well-reviewed new Linux distro, on my Vaio. I got lots of help from the nice folks on the IRC channel, and tried lots of different approaches, but no dice -- their Live CD won't boot because it doesn't grok PCMCIA CDROM drives, and they have no other installation method besides using the live CD.

Which brings me to today's question: Why do Linux distros have installers at all?

That probably sounds like a silly question. Of course you need an installer to get the system onto your disk ... don't you?

Well, yes and no. You could make it a lot simpler than anyone currently does.

What if you distributed a Linux distro as a filesystem image? Make it tar, zip, CD iso or whatever format you like -- but provide the user with a tree of files that, when copied onto a hard drive, can boot as a running Linux.

Set up this minimal installation filesystem so that when you boot into it, you get a commandline (X hasn't been configured yet) and a set of scripts that make it easy to go the rest of the way. From your running minimal system, you can configure X, set up networking, install more programs from the distro repositories (or even from a CD image), and do all the rest of the machine-specific configuration that an installer does.

The key is that this is all happening from a running system, not from some cobbled-together installer kernel or live CD.

If you have a problem with any step, you still have a basic running system, and tools to fix the problem. You avoid the usual loop:

• Run installer
• Spend 20 minutes answering questions
• Spend 45 minutes waiting for installer
• Discover it failed
• Start over with slightly different parameters

If your X configuration fails, try running X configuration again -- no need to do another install from the beginning. If it doesn't see your network card -- ditto. Since this debugging all happens from a normal running Linux, you can use the normal Linux tools you're used to, not some busybox-based installer.

This model would be much more hardware agnostic than current installers:

• You can install on systems with weirdo graphics cards;
• You can install on systems that need special drivers for the network card or other hardware;
• You can install on systems with no CDROM or an external CDROM;
• You can install even if you don't have access to a CD burner.

Another advantage is that it makes it very easy to make a customized version of your distro: just take the basic system image, change the part that needs changing and tar it up again.

Some distros have gone a little way with this, with an installer that gives you a starter system, then scripts to download the rest -- but I've never seen one that made the minimal system available as a filesystem image, with easy instructions on going to the next step.

What about the people who aren't already running Linux or aren't comfortable writing a filesystem image to a partition?

No problem. They get a CD image with a very simple installer -- it handles the partitioning, copies the minimal install to the partition, and updates grub. It's as prone to hardware compatibility issues as any installer (though far less so than a live CD is) but it's still better than the current model, because it won't be trying to configure hardware until the user reboots into a working minimal system.

Of course, Live CDs are still cool -- on machines where they actually work -- for showing Linux to people not ready to commit to an install. But don't tie your installer to that. Give people a simpler way to install, one that's fast and straightforward and hardware agnostic and easy to understand or customize.

The tarball installer. An idea whose time has come. Now if I could just persuade the distros to offer it.

Update: a couple of people told me about Dynebolic, a distro that apparently does just that -- you install by copying a directory on the CD onto your partition, or rsyncing from their site. Nice!

Tags: linux
[ 23:59 Nov 14, 2007    More linux | permalink to this entry | ]

Little Orphan Annie

Something rustled madly in the star jasmine when I walked past. Probably just a sparrow, I thought. Ever since the sparrows discovered the squirrel nuts, there's been a gang camped out in the guava tree just outside the office door at all times.

I put it out of my mind until an hour later, when Dave reported, "There's an orphan squirrel in the star jasmine. It looks too small to be out on its own. Where is its mother?"

We put a few pieces of walnut out by the bush and watched. After a little while the youngster came out to investigate, moving very slowly and awkwardly, and sat next to the walnut pieces. It didn't sit normally: its weight was back on its tail, with hind legs stuck out in front and crossed, like a tiny squirrel Buddha.

The tiny youngster took a piece of walnut in its front paws and stared at it blankly as if wondering what to do with it. But ten minutes later we saw that it was nibbling, slowly and tentatively. It took a long time, but the orphan eventually made it through three pieces of walnut.

We provided more walnut (the fearful youngster scurried back under the jasmine) and a little dish of water and waited, but the orphan didn't reappear. An hour later, we saw a small young squirrel climbing a tree in the front yard. Could it be the same one? The baby we'd seen didn't look capable of climbing anything. Could it have been merely weak from hunger and fear, and a few nuts revived it?

The next morning, a new squirrel appeared at our feeding area in the backyard. A young female, small but confident. She was able to move both up and down fenceposts and leap from the fence to the oak tree, usually difficult maneuvers for a squirrel trainee. Surely this couldn't be the same tiny, shivering orphan we'd seen the day before?

But after finding a nut I'd left on the fence, this youngster sat in the same odd Buddha fashion to eat it.

Little orphan Annie turned out to be smart as well as agile. She caught on to the nut shelf early -- she was hanging out in the guava (whose springy branches make a great playground for a light little squirreling) when a mouse made a rare appearance, darting out from under the deck to the nut shelf to grab a nut and run back to its hole. I could see Annie's head move as she watched the mouse; I could almost imagine her eyes widening. No need to tell her twice! She was down the guava and over to the nut shelf like a flash to pick up a piece for herself.

Annie hung around for about a week after that (getting chased by Ringtail a few times) but then she stopped visiting. Life is tough for young squirrels. I hope Annie's all right, and just moved on to find a nuttier place to live.

Tags: nature, squirrels, urban wildlife
[ 12:39 Nov 12, 2007    More nature/squirrels | permalink to this entry | ]

Tried bash, went back to tcsh

I've been a tcsh user for many years. Back in the day, there were lots of reasons for preferring csh to sh, mostly having to do with command history. Most of those reasons are long gone -- modern bash and tcsh are vastly improved from those early shells, and borrow from each other, so the differences are fairly minor.

Back in July, I solved the last blocker that had been keeping me off bash, so I put some effort into migrating all my .cshrc settings into a .bashrc so I could give bash a fair shot. It almost won; but after four months, I've switched back to tcsh, due mostly to a single niggling bash bug that I just can't seem to solve. After all these years, the crucial difference is still history. Specifically, history amnesia: bash has an annoying habit of forgetting history commands just when I most want them back.

Say I type some longish command. After it runs, I hit return a couple of times, wait a while, do a couple of other things, then decide I want to call that command back from history so I can run something similar, maybe with the filename changed or a different flag. I ctrl-P or up-arrow ... and the command isn't there!

If I type history at this point, I'll see most of my command history ... with an empty line in place of the line I was hoping to repeat. The command is gone. My only option is to remember what I typed, and type it all again.

Nobody seems to know why this happens, and it's sporadic, doesn't happen every time. Friends have been able to reproduce it, so it's not just me or my weird settings. It drives me batty. It wouldn't be so bad except it always seems to happen on the tricky commands that I really didn't want to retype.

It's too bad, because otherwise I had bash nicely whipped into shape, and it does have some advantages over tcsh. Some of the tradeoffs:

tcsh wins

• Totally reliable history: commands never disappear.
• History tab completion: typing !a<TAB> expands to the last command that started with a. In bash, I have to type !a:p to see the command, then !! to execute it.
• When I tab-complete a file and there are multiple matches, tcsh shows them, or at least beeps (depending on configuration). In bash I have to hit a second tab just in case there might be matches.
• When I tab-complete a directory, tcsh adds the / automatically. (Arguable. I find I want the / roughly 3/4 of the time.)
• tcsh doesn't drop remote connections if I suspend (with ~^Z). bash drops me within a minute or two, regardless of settings like $TMOUT. Bash users tell me I could solve this by using screen, but that seems like a heavyweight workaround when tcsh "just works".
• tcsh sees $REMOTEHOST and $DISPLAY automatically when I ssh. bash doesn't: ssh -X helps, but I still need some tricky code in .bash_profile.
• aliases can have arguments, e.g. alias llth 'ls -laFt \!* | head' In bash these have to be functions, which means they don't show up typing "which" or "alias".
• Prompt settings are more flexible -- there are options like %B for bold. In bash you have to get the terminal type and use the ansi color escape sequances, which don't include bold.
• Easier command editing setup -- behaviors like getting word-erase to stop at punctuation don't involve chasing through multiple semi-documented programs, and the answer doesn't vary with version.
• Documentation -- tcsh's is mostly in man tcsh, bash's is scattered all over man pages for various helper programs. And it's hard to google for bash help because "bash" as a keyword gets you lots of stuff not related to shells.

Of course, you bash users, set me straight if I missed out on some bash options that would have solved some of these problems. And especially if you have a clue about the evil disappearing history commands!

bash wins

• You don't have to rehash every time you add a program or change your path. That's a real annoyance of tcsh, and I could understand a person used to bash rejecting tcsh on this alone.
  Update: Holger Weiß has written a tcsh patch to fix this, and it has been accepted as of November 2009. Looking forward to the next version!
• History remembers entire multi-line commands, and shows them with semicolons when you arrow back through history. Very nice. tcsh only remembers the first line and you have to retype the rest.
• Functions: although I don't like having to use them instead of aliases, they're certainly powerful and nice to have.

Of course, bash and tcsh aren't the only shells around. From what I hear, zsh blends the good features of bash and tcsh. One of these days I'll try it and see.

Tags: linux, shell, CLI, bash, csh
[ 22:58 Nov 07, 2007    More linux | permalink to this entry | ]