Shallow Thoughts : tags : cmdline

Find out what processes are making network connections

A thread on the Ubuntu-devel-discuss mailing list last month asked about how to find out what processes are making outgoing network connectsion on a Linux machine. It referenced Ubuntu bug 820895: Log File Viewer does not log "Process Name", which is specific to Ubuntu's iptables logging of apps that are already blocked in iptables ... but the question goes deeper.

Several years ago, my job required me to use a program -- never mind which one -- from a prominent closed-source company. This program was doing various annoying things in addition to its primary task -- operations that got around the window manager and left artifacts all over my screen, operations that potentially opened files other than the ones I asked it to open -- but in addition, I noticed that when I ran the program, the lights on the DSL modem started going crazy. It looked like the program was making network connections, when it had no reason to do that. Was it really doing that?

Unfortunately, at the time I couldn't find any Linux command that would tell me the answer. As mentioned in the above Ubuntu thread, there are programs for Mac and even Windows to tell you this sort of information, but there's no obvious way to find out on Linux.

The discussion ensuing in the ubuntu-devel-discuss thread tossed around suggestions like apparmor and selinux -- massive, complex ways of putting up fortifications your whole system. But nobody seemed to have a simple answer to how to find information about what apps are making network connections.

Well, it turns out there are a a couple ofsimple way to get that list. First, you can use ss:

$ ss -tp
State      Recv-Q Send-Q      Local Address:Port          Peer Address:Port   
ESTAB      0      0                     ::1:58466                  ::1:ircd     users:(("xchat",1063,43))
ESTAB      0      0             192.168.1.6:57526       140.211.166.64:ircd     users:(("xchat",1063,36))
ESTAB      0      0                     ::1:ircd                   ::1:58466    users:(("bitlbee",1076,10))
ESTAB      0      0             192.168.1.6:54253       94.125.182.252:ircd     users:(("xchat",1063,24))
ESTAB      0      0             192.168.1.6:52167       184.72.217.144:https
users:(("firefox-bin",1097,47))

-t shows only TCP connections (so you won't see all the interprocess communication among programs running on your machine). -p prints the process associated with each connection.

ss can do some other useful things, too, like show all the programs connected to your X server right now, or show all your ssh connections. See man ss for examples.

Or you can use netstat:

$ netstat -A inet -p
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 imbrium.timochari:51800 linuxchix.osuosl.o:ircd ESTABLISHED 1063/xchat      
tcp        0      0 imbrium.timochari:59011 ec2-107-21-74-122.:ircd ESTABLISHED 1063/xchat      
tcp        0      0 imbrium.timochari:54253 adams.freenode.net:ircd ESTABLISHED 1063/xchat      
tcp        0      0 imbrium.timochari:58158 s3-1-w.amazonaws.:https ESTABLISHED
1097/firefox-bin

In both cases, the input is a bit crowded and hard to read. If all you want is a list of processes making connections, that's easy enough to do with the usual Unix utilities like grep and sed:

$ ss -tp | grep -v Recv-Q | sed -e 's/.*users:(("//' -e 's/".*$//' | sort | uniq
$ netstat -A inet -p | grep '^tcp' | grep '/' | sed 's_.*/__' | sort | uniq

Finally, you can keep an eye on what's going on by using watch to run one of these commands repeatedly:

watch ss -tp

Using watch with one of the pipelines to print only process names is possible, but harder since you have to escape a lot of quotation marks. If you want to do that, I recommend writing a script.

And back to the concerns expressed on the Ubuntu thread, you could also write a script to keep logs of which processes made connections over the course of a day. That's definitely a tool I'll keep in my arsenal.

Open the X selection in a browser window, from any desktop

Like most Linux users, I use virtual desktops. Normally my browser window is on a desktop of its own.

Naturally, it often happens that I encounter a link I'd like to visit while I'm on a desktop where the browser isn't visible. From some apps, I can click on the link and have it show up. But sometimes, the link is just text, and I have to select it, change to the browser desktop, paste the link into firefox, then change desktops again to do something else while the link loads.

So I set up a way to load whatever's in the X selection in firefox no matter what desktop I'm on.

In most browsers, including firefox, you can tell your existing browser window to open a new link from the command line: firefox http://example.com/ opens that link in your existing browser window if you already have one up, rather than starting another browser. So the trick is to get the text you've selected.

At first, I used a program called xclip. You can run this command: firefox `xclip -o` to open the selection. That worked okay at first -- until I hit my first URL in weechat that was so long that it was wrapped to the next line. It turns out xclip does odd things with multi-line output; depending on whether it thinks the output is a terminal or not, it may replace the newline with a space, or delete whatever follows the newline. In any case, I couldn't find a way to make it work reliably when pasted into firefox.

After futzing with xclip for a little too long, trying to reverse-engineer its undocumented newline behavior, I decided it would be easier just to write my own X clipboard app in Python. I already knew how to do that, and it's super easy once you know the trick:

mport gtk
primary = gtk.clipboard_get(gtk.gdk.SELECTION_PRIMARY)
if primary.wait_is_text_available() :
    print primary.wait_for_text()

That just prints it directly, including any newlines or spaces. But as long as I was writing my own app, why not handle that too?

It's not entirely necessary on Firefox: on Linux, Firefox has some special code to deal with pasting multi-line URLs, so you can copy a URL that spans multiple lines, middleclick in the content area and things will work. On other platforms, that's disabled, and some Linux distros disable it as well; you can enable it by going to about:config and searching for single, then setting the preference editor.singlelinepaste.pasteNewlines to 2.

However, it was easy enough to make my Python clipboard app do the right thing so it would work in any browser. I used Python's re (regular expressions) module:

#!/usr/bin/env python

import gtk
import re

primary = gtk.clipboard_get(gtk.gdk.SELECTION_PRIMARY)

if not primary.wait_is_text_available() :
    sys.exit(0)
s = primary.wait_for_text()

# eliminate newlines, and any spaces immediately following a newline:
print re.sub(r'[\r\n]+ *', '', s)

That seemed to work fine, even on long URLs pasted from weechat with newlines and spaces, like that looked like

http://example.com/long-
    url.html

All that was left was binding it so I could access it from anywhere. Of course, that varies depending on your desktop/window manager. In Openbox, I added two items to my desktop menu in menu.xml:

  <item label="open selection in Firefox">
    <action name="Execute"><execute>sh -c 'firefox `xclip -o`'</execute></action>
  </item>
  <item label="open selection in new tab">
    <action name="Execute"><execute>sh -c 'firefox -new-tab `xclip -o`'</execute></action>
  </item>

I also added some code in rc.xml inside <context name="Desktop">, so I can middle-click or control-middle-click on the desktop to open a link in the browser:

      <mousebind button="Middle" action="Press">
        <action name="Execute">
          <execute>sh -c 'firefox `pyclip`'</execute>
        </action>
      </mousebind>
      <mousebind button="C-Middle" action="Press">
        <action name="Execute">
          <execute>sh -c -new-tab 'firefox `pyclip`'</execute>
        </action>
      </mousebind>

I set this up maybe two hours ago and I've probably used it ten or fifteen times already. This is something I should have done long ago!

Convert patterns in only some lines to title case

A friend had a fun problem: she had some XML files she needed to import into GNUcash, but the program that produced them left names in all-caps and she wanted them more readable. So she'd have a file like this:

<STMTTRN>
   <TRNTYPE>DEBIT
   <DTPOSTED>20111125000000[-5:EST]
   <TRNAMT>-22.71
   <FITID>****

   <NAME>SOME    COMPANY
   <MEMO>SOME COMPANY    ANY TOWN   CA 11-25-11 330346
</STMTTRN>

The obvious candidate to do string substitutes is sed. But there are several components to the problem.

Addresses

First, how do you ensure the replacement only happens on lines with NAME and MEMO?

sed lets you specify address ranges for just that purpose. If you say sed 's/xxx/yyy/' sed will change all xxx's to yyy; but if you say sed '/NAME/s/xxx/yyy/' then sed will only do that substitution on lines containing NAME.

But we need this to happen on lines that contain either NAME or MEMO. How do you do that? With \|, like this: sed '/\(NAME\|MEMO\)/s/xxx/yyy/'

Converting to title case

Next, how do you convert upper case to lower case? There's a sed command for that: \L. Run sed 's/.*/\L&/' and type some upper and lower case characters, and they'll all be converted to lower-case.

But here we want title case -- we want most of each word converted to lowercase, but the first letter should stay uppercase. That means we need to detect a word and figure out which is the first letter.

In the strings we're considering, a word is a set of letters A through Z with one of the following characteristics:

1. It's preceded by a space
2. It's preceded by a close-angle-bracket, >

So the pattern /[ >][A-Z]*/ will match anything we consider a word that might need conversion.

But we need to separate the first letter and the rest of the word, so we can treat them separately. sed's \( \) operators will let us do that. The pattern \([ >][A-Z]\) finds the first letter of a word (including the space or > preceding it), and saves that as its first matched pattern, \1. Then \([A-Z]*\) right after it will save the rest of the word as \2.

So, taking our \L case converter, we can convert to title case like this: sed 's/\([ >][A-Z]\)\([A-Z]*\)/\1\L\2/g

Starting to look long and scary, right? But it's not so bad if you build it up gradually from components. I added a g on the end to tell sed this is a global replace: do the operation on every word it finds in the line, otherwise it will only make the substitution once, on the first word it sees, then quit.

Putting it together

So we know how to seek out specific lines, and how to convert to title case. Put the two together, and you get the final command:

sed '/\(NAME\|MEMO\)/s/\([ >][A-Z]\)\([A-Z]*\)/\1\L\2/g'

I ran it on the test input, and it worked just fine.

For more information on sed, a good place to start is the sed regular expressions manual.

List only directories

Fairly often, I want a list of subdirectories inside a particular directory. For instance, when posting blog entries, I may need to decide whether an entry belongs under "linux" or some sub-category, like "linux/cmdline" -- so I need to remind myself what categories I have under linux.

But strangely, Linux offers no straightforward way to ask that question. The ls command lists directories -- along with the files. There's no way to list just the directories. You can list the directories first, with the --group-directories-first option. Or you can flag the directories specially: ls -F appends a slash to each directory name, so instead of linux you'd see linux/. But you still have to pick the directories out of a long list of files. You can do that with grep, of course:

ls -1F ~/web/blog/linux | grep /

Or you can use the find program, which has an option -type d that lists only directories. Perfect, right?

find ~/web/blog/linux -maxdepth 1 -type d

Except that lists everything with full pathnames: /home/akkana/web/blog/linux, /home/akkana/web/blog/linux/editors, /home/akkana/web/blog/linux/cmdline and so forth. Way too much noise to read quickly.

What I'd really like is to have just a list of directory names -- no slashes, no newlines. How do we get from ls or find output to that? Either we can start with find and strip off all the path information, either in a loop with basename or with a sed command; or start with ls -F, pick only the lines with slashes, then strip off those slashes. The latter sounds easier.

So let's go back to that ls -1F ~/web/blog/linux | grep / command. To strip off the slashes, you can use sed's s (substitute) command. Normally the syntax is sed 's/oldpat/newpat/'. But since slashes are the pattern we're substituting, it's better to use something else as the separator character. I'll use an underscore.

The old pattern, the one I want to replace, is / -- but I only want to replace the last slash on the line, so I'll add a $ after it, representing end-of-line. The new pattern I want instead of the slash is -- nothing.

So my sed argument is 's_/$__' and the command becomes:

ls -1F ~/web/blog/linux | grep / | sed 's_/$__'

That does what I want. If I don't want them listed one per line, I can fudge that using backquotes to pass the output of the whole command to the shell's echo command:

echo `ls -1F ~/web/blog/linux | grep / | sed 's_/$__'`

If you have a lot of directories to list and you want ls's nice columnar format, that's a little harder. You can ls the list of directories (the names inside the backquotes), ls `your long command` -- except that now that you've stripped off the path information, ls won't know where to find the files. So you'd have to change directory first:

cd ~/web/blog/linux; ls -d `ls -1F | grep / | sed 's_/$__'`

That's not so good, though, because now you've changed directories from wherever you were before. To get around that, use parentheses to run the commands inside a subshell:

(cd ~/web/blog/linux; ls -d `ls -1F | grep / | sed 's_/$__'`)

Now the cd only applies within the subshell, and when the command finishes, your own shell will still be wherever you started.

Finally, I don't want to have to go through this discovery process every time I want a list of directories. So I turned it into a couple of shell functions, where $* represents all the arguments I pass to the command, and $1 is just the first argument.

lsdirs() { 
  (cd $1; /bin/ls -d `/bin/ls -1F | grep / | sed 's_/$__'`)
}

lsdirs2() { 
  echo `/bin/ls -1F $* | grep / | sed 's_/$__'` 
}

Now I can type lsdirs ~/web/blog/linux and get a nice list of directories.

echo ~/web/blog/linux/*(/:t)

Command-line Arduino development

I've been doing more Arduino development lately. But I don't use the Arduino Java development environment -- programming is so much easier when you have a real editor, like emacs or vim, and key bindings to speed everything up.

I've found very little documentation on how to do command-line Arduino development, and most of the Makefiles out there are old and no longer work. So I've written up a tutorial. It ended up too long for a blog post, so I've made it a separate article:

Command-line Arduino development.

Using grep to solve another Cartalk puzzler

Last week's puzzler was A Seven-Letter Vacation Curiosity. Basically, one couple hiking in Northern California and another couple carousing in Florida both see something described by a seven-letter word containing all five vowels -- but the two things they saw were very different. What's the word?

That's an easy one to solve using basic Linux command-line skills -- assuming the word is in the standard dictionary. If it's some esoteric word, all bets are off. But let's try it and see. It's a good beginning exercise in regular expressions and how to use the command line.

There's a handy word list in /usr/share/dict/words, one word per line. Depending on what packages you have installed, you may have bigger dictionaries handy, but you can usually count on /usr/share/dict/words being there on any Linux system. Some older Unix systems may have it in /usr/dict/words instead.

We need a way to choose all seven letter words. That's easy. In a regular expression, . (a dot) matches one letter. So ....... (seven dots) matches any seven letters.

(There's a more direct way to do that: the expression .\{7\} will also match 7 letters, and is really a better way. But personally, I find it harder both to remember and to type than the seven dots. Still, if you ever need to match 43 characters, or 114, it's good to know the "right" syntax.)

Fine, but if you grep ....... /usr/share/dict/words you get a list of words with seven or more letters. See why? It's because grep prints any line where it finds a match -- and a word with nine letters certainly contains seven letters within it.

The pattern you need to search for is '^.......$' -- the up-caret ^ matches the beginning of a line, and the dollar sign $ matches the end. Put single quotes around the pattern so the shell won't try to interpret the caret or dollar sign as special characters. (When in doubt, it's always safest to put single quotes around grep patterns.)

So now we can view all seven-letter words: grep '^.......$' /usr/share/dict/words
How do we choose only the ones that contain all the letters a e i o and u?

That's easy enough to build up using pipelines, using the pipe character | to pipe the output of one grep into a different grep. grep '^.......$' /usr/share/dict/words | grep a sends that list of 7-letter words through another grep command to make sure you only see words containing an a.

Now tack a grep for each of the other letters on the end, the same way:
grep '^.......$' /usr/share/dict/words | grep a | grep e | grep i | grep o | grep u

Voilà! I won't spoil the puzzler, but there are two words that match, and one of them is obviously the answer.

The power of the Unix command line to the rescue!

"Who am I?" Maybe nobody!

He was using the command who am i -- reasonable, since it's been a staple since the early days of Unix. For those not familiar with the command, /usr/bin/who, if given two arguments, regardless of what those arguments are, will print information about the current logged-in user. It also offers a -m option to do the same thing. So who am i, who a b, and who -m should all print a line like:

$ who am i
akkana   pts/1        2010-09-29 09:33 (:0.0)

Except they don't. For me, they printed nothing at all -- which broke my colleague's install script.

A quick poll among friends on IRC showed that who am i worked for some people, failed for others, with no obvious logic to it.

It's the terminal

It took some digging to find out what was going on, but the difference turned out to be the terminal being used. The who program -- with or without -m -- gets its info from /var/run/utmp, a file that maintains a record of who's logged in to the system. And it turns out some terminals create a utmp entry, while others don't. So:

Program	Creates utmp entry?
gnome-terminal	yes
konsole	yes
xterm	no
xfterm4	yes
terminator	no
rxvt	no
roxterm	yes

I use xterm myself. Xterm is documented (in its man page) to modify the utmp entry, and it has a command-line flat, +ut, plus two X resources, ptyHandshake and utmpInhibit. None of the three work: setting

XTerm*ptyHandshake: true
XTerm*utmpInhibit: false

How do you get the real user?

Okay, so who am i clearly isn't a reliable way of getting the user ID. What can you use instead?

Several people suggested the id program. It has a -r option which supposedly prints the real UID. Unfortunately, what it really does is print:

$ id -r
id: cannot print only names or real IDs in default format

Update: people keep suggesting id -ru to me. Evidently I wasn't very clear in this article: the goal is to get the real id of the login user. In other words, if you're logged in as mary and using sudo, you want mary, not root.

Alas, adding -u to id's flags gets only the effective user id: -u wins over -r. This is very easy to test: sudo id -ru prints 0, as does id -ru inside su.

But elly on Freenode had a great suggestion:

stat -c '%U' `readlink /proc/self/fd/0`

/proc/self is a symlink to /proc/pid, a directory where you can find out all sorts of information about a process.

One of the things you can find out about a process is open file descriptors: in particular, standard input, output and error. So /proc/self/fd/0 corresponds to standard input of the current process -- which in the example above is readlink.

What is readlink? Well, /proc/self/fd/0, in the normal case, is actually a symlink to the terminal controlling the process. readlink prints the file to which that link points -- for instance, /dev/pts/1. That's the terminal being used.

Now that we know the name of the terminal, all we need to do is find out who owns it. (This is the information who am i would have gotten from utmp, had there been a utmp entry.) ls -l /dev/pts/1 will show you that it's you, even if you run it as sudo ls -l /dev/pts/1. You could take that and strip off fields to get the username, but stat, as elly suggested, is a much better way of doing that.

Put it all together, and stat -c '%U' `readlink /proc/self/fd/0 gets standard input for the current process, follows the link to get the controlling terminal, then finds out who owns that terminal.

That's you!

A similar but slightly shorter solution suggested by Mikachu: stat -c %u `tty`

Use "date" to show time abroad

But I suggested he could do the same thing with the TZ environment variable. It's not documented well anywhere I could find, but if you set TZ to the name of a time zone, date will print out the time for that zone rather than your current one.

So, for bash:

$ TZ=Europe/Paris date  # time in Paris
$ TZ=GB date            # time in Great Britain
$ TZ=GMT-02 date        # time two timezones east of GMT

% ( setenv TZ Europe/Paris; date)
% ( setenv TZ GB; date)
% ( setenv TZ GMT-02; date)

That's all very well. But when I tried

% ( setenv TZ UK; date)
% ( setenv TZ FR; date)

The trick seems to be that setting TZ only works for abbreviations in /usr/share/zoneinfo/, or maybe in /usr/share/zoneinfo/posix/. If you give an abbreviation, like UK or FR or America/San_Francisco, it won't give you an error, it'll just print GMT as if that was what you had asked for.

So this trick is useful for printing times abroad -- but if you want to be safe, either stick to syntaxes like GMT-2, or make a script that checks whether your abbreviation exists in the directory before calling date, and warns you rather than just printing the wrong time.