Shallow Thoughts : tags : bash

A nifty shell redirection trick: process substitution

I love shell pipelines, and flatter myself that I'm pretty good at them. But a discussion last week on the Linuxchix Techtalk mailing list on finding added lines in a file turned up a terrific bash/zsh shell redirection trick I'd never seen before:

join -v 2 <(sort A.txt) <(sort B.txt)

I've used backquotes, and their cognate $(), plenty. For instance, you can do things like PS1=$(hostname): or PS1=`hostname`: to set your prompt to the current hostname: the shell runs the hostname command, takes its output, and substitutes that output in place of the backquoted or parenthesized expression.

But I'd never seen that <(...) trick before, and immediately saw how useful it was. Backquotes or $() let you replace arguments to a command with a program's output -- they're great for generating short strings for programs that take all their arguments on the command line. But they're no good for programs that need to read a file, or several files. <(...) lets you take the output of a command and pass it to a program as though it was the contents of a file. And if you can do it more than once in the same command -- as in Little Girl's example -- that could be tremendously useful.

Playing with it to see if it really did what it looked like it did, and what other useful things I could do with it, I tried this (and it worked just fine):

$ diff <(echo hello; echo there) <(echo hello; echo world)
2c2
< there
---
> world

It acts as though I had two files, which each have "hello" as their first line; but one has "there" as the second line, while the other has "world". And diff shows the difference. I don't think there's any way of doing anything like that with backquotes; you'd need to use temp files.

Of course, I wanted to read more about it -- how have I gone all these years without knowing about this? -- and it looks like I'm not the only one who didn't know about it. In fact, none of the pages I found on shell pipeline tricks even mentioned it.

It turns out it's called "process substitution" and I found it documented in Chapter 23 of the Advanced Bash-Scripting Guide.

I tweeted it, and a friend who is a zsh master gave me some similar cool tricks. For instance, in zsh echo hi > >(cat) > >(cat -n) lets you pipe the output of a command to more than one other command.

That's zsh, but in bash (or zsh too, of course), you can use >() and tee to do the same thing: echo hi | tee >(cat) | cat -n

If you want a temp file to be created automatically, one you can both read and write, you can use =(foo) (zsh only?)

Great stuff! Some other pages that discuss some of these tricks:

• IO Redirection
• Bash Redirection Cheat Sheet
• Bash One-Liners Explained, Part 3

Tags: shell, cmdline, zsh, bash
[ 19:23 Aug 24, 2013    More linux/cmdline | permalink to this entry | ]

Tip: Bash remembering history across sessions

Two separate friends just had this problem, one of them a fairly experienced Linux user:

You're in bash, history works, but it's not remembered across sessions. Why?

Maybe the size of the history file somehow got set to zero?

$ echo $HISTFILESIZE
500

Nope -- that's not it.

Maybe it's using the wrong file. In bash you can set $HISTFILE to point to different places; for instance, you can use that to maintain different histories per window, or per machine.

$ echo $HISTFILE
/home/username/.bash_history

Nope, that's not it either.

The problem, for both people, turned out to be really simple:

$ ls -l $HISTFILE
-rw------- 1 root root 92 2007-08-20 14:03 /home/user/.bash_history

I'm not sure how it happens, but sometimes the .bash_history file becomes owned by root, and then as a normal user you can't update your history any more.

So a simple

$ rm $HISTFILE

and you're all set -- history across sessions should start working again.

Tags: shell, bash, tips
[ 14:42 Nov 27, 2009    More linux/cmdline | permalink to this entry | ]

Getting rid of .sudo_as_admin_successful

Ever been annoyed by the file in your home directory, .sudo_as_admin_successful? You know, the one file with the name so long that it alone is responsible for making ls print out your home directory in two columns rather than three or four? And if you remove it, it comes right back after the next time you run sudo?

Here's what's creating it (credit goes to Dave North for figuring out most of this).

It's there because you're in the group admin, and it's there to turn off a silly bash warning. It's specific to Ubuntu (at least, Fedora doesn't do it). Whenever you log in under bash, if bash sees that you're in the admin group in /etc/groups, it prints this warning:

To run a command as administrator (user "root"), use "sudo ".
See "man sudo_root" for details.

Once you sudo to root, if you're in the admin group, sudo creates an empty file named .sudo_as_admin_successful in your home directory. That tells bash, the next time you log in, not to print the stupid warning any more. Sudo creates the file even if your login shell isn't bash and so you would never have seen the stupid warning. Hey, you might some day go back to bash, right?

If you want to reclaim your ls columns and get rid of the file forever, it's easy: just edit /etc/group and remove yourself from the admin group. If you were doing anything that required being in the admin group, substitute another group with a different name.

Tags: linux, bash, sudo, annoyances, ubuntu
[ 18:33 Oct 09, 2008    More linux | permalink to this entry | ]

Tried bash, went back to tcsh

I've been a tcsh user for many years. Back in the day, there were lots of reasons for preferring csh to sh, mostly having to do with command history. Most of those reasons are long gone -- modern bash and tcsh are vastly improved from those early shells, and borrow from each other, so the differences are fairly minor.

Back in July, I solved the last blocker that had been keeping me off bash, so I put some effort into migrating all my .cshrc settings into a .bashrc so I could give bash a fair shot. It almost won; but after four months, I've switched back to tcsh, due mostly to a single niggling bash bug that I just can't seem to solve. After all these years, the crucial difference is still history. Specifically, history amnesia: bash has an annoying habit of forgetting history commands just when I most want them back.

Say I type some longish command. After it runs, I hit return a couple of times, wait a while, do a couple of other things, then decide I want to call that command back from history so I can run something similar, maybe with the filename changed or a different flag. I ctrl-P or up-arrow ... and the command isn't there!

If I type history at this point, I'll see most of my command history ... with an empty line in place of the line I was hoping to repeat. The command is gone. My only option is to remember what I typed, and type it all again.

Nobody seems to know why this happens, and it's sporadic, doesn't happen every time. Friends have been able to reproduce it, so it's not just me or my weird settings. It drives me batty. It wouldn't be so bad except it always seems to happen on the tricky commands that I really didn't want to retype.

It's too bad, because otherwise I had bash nicely whipped into shape, and it does have some advantages over tcsh. Some of the tradeoffs:

tcsh wins

• Totally reliable history: commands never disappear.
• History tab completion: typing !a<TAB> expands to the last command that started with a. In bash, I have to type !a:p to see the command, then !! to execute it.
• When I tab-complete a file and there are multiple matches, tcsh shows them, or at least beeps (depending on configuration). In bash I have to hit a second tab just in case there might be matches.
• When I tab-complete a directory, tcsh adds the / automatically. (Arguable. I find I want the / roughly 3/4 of the time.)
• tcsh doesn't drop remote connections if I suspend (with ~^Z). bash drops me within a minute or two, regardless of settings like $TMOUT. Bash users tell me I could solve this by using screen, but that seems like a heavyweight workaround when tcsh "just works".
• tcsh sees $REMOTEHOST and $DISPLAY automatically when I ssh. bash doesn't: ssh -X helps, but I still need some tricky code in .bash_profile.
• aliases can have arguments, e.g. alias llth 'ls -laFt \!* | head' In bash these have to be functions, which means they don't show up typing "which" or "alias".
• Prompt settings are more flexible -- there are options like %B for bold. In bash you have to get the terminal type and use the ansi color escape sequances, which don't include bold.
• Easier command editing setup -- behaviors like getting word-erase to stop at punctuation don't involve chasing through multiple semi-documented programs, and the answer doesn't vary with version.
• Documentation -- tcsh's is mostly in man tcsh, bash's is scattered all over man pages for various helper programs. And it's hard to google for bash help because "bash" as a keyword gets you lots of stuff not related to shells.

Of course, you bash users, set me straight if I missed out on some bash options that would have solved some of these problems. And especially if you have a clue about the evil disappearing history commands!

bash wins

• You don't have to rehash every time you add a program or change your path. That's a real annoyance of tcsh, and I could understand a person used to bash rejecting tcsh on this alone.
  Update: Holger Weiß has written a tcsh patch to fix this, and it has been accepted as of November 2009. Looking forward to the next version!
• History remembers entire multi-line commands, and shows them with semicolons when you arrow back through history. Very nice. tcsh only remembers the first line and you have to retype the rest.
• Functions: although I don't like having to use them instead of aliases, they're certainly powerful and nice to have.

Of course, bash and tcsh aren't the only shells around. From what I hear, zsh blends the good features of bash and tcsh. One of these days I'll try it and see.

Tags: linux, shell, CLI, bash, csh
[ 22:58 Nov 07, 2007    More linux | permalink to this entry | ]

Adventures with bash's word erase

I've been a happy csh/tcsh user for decades. But every now and then I bow to pressure and try to join the normal bash-using Linux world.

But I always come up against one problem right away: word erase (Control-W). For those who don't use ^W, suppose I type something like:

% ls /backups/images/trips/arizona/2007

Then I suddenly realize I want utah in 2007, not arizona. In csh, I can hit ^W twice and it erases the last two words, and I'm ready to type u<tab>. In bash, ^W erases the whole path leaving only "ls", so it's no help here. It may seem like a small thing, but I use word erase hundreds of times a day and it's hard to give it up. Google was no help, except to tell me I wasn't the only one asking.

Then the other day I was chatting about this issue with a friend who uses zsh for that reason (zsh is much more flexible at defining key bindings) and someone asked, "Is that like Meta-Delete?"

It turned out that Alt-Backspace (like many Linux applications, bash calls the Alt key "Meta", and Linux often confuses Delete and Backspace) did exactly what I wanted. Very promising!

But Alt-Backspace is not easy to type, since it's not reachable from the "home" typing position. What I needed, now that I knew bash and readline had the function, was a way to bind it to ^W.

Bash's binding syntax is documented, though the functions available don't seem to be. But bind -p | grep word gave me some useful information. It seems that \C-w was bound to "unix-word-rubout" (that was the one I didn't want) whereas "\e\C-?" was bound to "backward-kill-word". ("\e\C-?" is an obscure way of saying Meta-DEL: \e is escape, and apparently bash, like emacs, treats ESC followed by a key as the same as pressing Alt and the key simultaneously. And Control-question-mark is the Delete character in ASCII.)

So my task was to bind \C-w to backward-kill-word. It looked like this ought to work:

bind '\C-w:backward-kill-word'

... Except it didn't. bind -p | grep w showed that C-W was still bound to "unix-word-rubout".

It turned out that it was the terminal (stty) settings causing the problem: when the terminal's werase (word erase) character is set, readline hardwires that character to do unix-word-rubout and ignores any attempts to change it.

I found the answer in a bash bug report. The stty business was introduced in readline 5.0, but due to complaints, 5.1 was slated to add a way to override the stty settings. And happily, I had 5.2! So what was this new way override method? The posting gave no hint, but eventually I found it.

Put in your .inputrc:

set bind-tty-special-chars Off

And finally my word erase worked properly and I could use bash!

Tags: linux, CLI, shell, bash
[ 16:22 Jul 17, 2007    More linux | permalink to this entry | ]