Shallow Thoughts : : linux

Akkana's Musings on Open Source Computing, Science, and Nature.

Sat, 12 May 2012

Downloading Adobe-protected books to a Nook using Linux

University of Chicago Press has a Carl Zimmer book, A Planet of Viruses, as their free monthly e-book.

I know Zimmer is a good writer. but the ebook, despite being free, is encumbered with Adobe's version of DRM, which unlocks via a Windows or Mac program. I use Linux, and wanted to read the book on a Nook. Was I out of luck?

Happily, the instruction page they sent when I signed up for the book helpfully included a section for Linux users. Hooray, U. Chicago! It said Adobe Digital Editions will run under Wine, the Windows emulator. I'd been meaning to try that anyway, and a Carl Zimmer book seemed like the perfect excuse.

And overall, it worked pretty well, with only a few snags. Here are the steps I had to follow:

Authorizing a book using Adobe Digital Editions in Linux on Wine

Install wine (on Ubuntu, I used apt-get install wine).

Download the Adobe Digital Editions setup.exe

Run: wine setup.exe (this should install ADE inside your .wine directory)

Copy the file, e.g. URLLink.acsm, into .wine/drive_c/My\ Documents/ Don't bother trying to open it with ADE -- the program won't open anything except PDF and epub. Curiously, the only ways to open the file from ADE are to drag the file onto the ADE window or to pass it as a commandline argument:
wine start .wine/drive_c/My\ Documents/URLLink.acsm

Now ADE should download your book and display it. You can read it there, if you want. But you won't want to -- it's not a good reading interface.

Authorizing a device with Adobe Digital Editions under Wine

Now how do you get it into your ebook reader? ADE running under Wine doesn't recognize devices such as ebook readers. so nothing will be copied automatically. But you can copy it manually.

In theory, the drive letter should stay mapped, so you should be able to use it for opening future books. Just remember to mount your device to the same location before running ADE under wine.

Tags: , ,
[ 10:03 May 12, 2012    More linux | permalink to this entry | comments ]

Wed, 18 Apr 2012

Mounting a Samsung Galaxy Player on Linux

My new toy: a Samsung Galaxy Player 5.0!

So far I love it. It's everything my old Archos 5 wanted to be when it grew up, except the Archos never grew up. It's the same size, a little lighter weight, reliable hardware (no random reboots), great battery life, fast GPS, modern Android 2.3, and the camera is even not too bad (though it certainly wouldn't tempt me away from my Canon).

For the record, Dave got a Galaxy Player 4.0, and it's very nifty too, and choosing between them was a tough choice -- the 4-inch is light and so easy to hold, and it uses replaceable batteries, while the 5-inch's screen is better for reading and maps.

USB-storage devices don't register

I love the Galaxy ... but there's one thing that bugs me about it. When I plug it in to Linux, dmesg reports two new storage devices, one for main storage and one for the SD card. Just like most Android devices, so far.

The difference is that these Samsung devices aren't fully there. They don't show up in /proc/partitions or in /dev/disk/by-uuid, dmesg doesn't show sizes for them, and, most important, they can't be mounted by UUID from an fstab entry, like 

UUID=62B0-C667   /droidsd    vfat   user,noauto,exec,fmask=111,shortname=lower 0 0
That meant I couldn't mount it as myself -- I had to become root, figure out where it happened to show up this time (/dev/sdf or wherever, depending on what else might be plugged in), mount it, then do all my file transfers as root.

I found that if I mounted it explicitly using the device pathname -- mount /dev/sdf /mnt -- then subsequently the device shows up normally, even after I unmount it. So I could check dmesg to find the device name, mount it as root, unmount as root, then mount it again as myself using my fstab entry. What a pain!

A kernel expert I asked thought it looked like the Samsung is pretending to be a removable device, but only "plugging in" when the system actually tries to access it. Annoying. So how do you get Linux to "access" it?

Udev: still an exercise in frustration

The obvious solution is a udev rule. Some scrutiny of /lib/udev/rules.d/60-persistent-storage.rules found some rules that did this intriguing thing: IMPORT{program}="ata_id --export $tempnode".

Naturally, this mysterious ata_id is undocumented. It's hidden in /lib/udev/ata_id, and I found this tiny ata_id man page online since there's none available in Ubuntu. Running ata_id /dev/sdf seemed to do what I needed: it made the device show up in /proc/partitions and /dev/disk/by-uuid, and after that, I could mount it without being root.

I created a file named /etc/udev/rules.d/59-galaxy.rules, with the rule: 

KERNEL=="sd[b-g]", SUBSYSTEMS=="usb", ATTRS{idVendor}=="04e8", SYMLINK+="galaxy-%k-%n", IMPORT{program}="ata_id --export $tempnode"

When I tested it with udevadm test /block/sdf, not only did the rule trigger correctly, but it actually ran ata_id and the device became visible -- even though udevadm test states clearly no programs will be run. How do I love udev -- let me count the ways.

But a reboot revealed that udev was not actually running the rule when I actually plugged the Galaxy in -- the devices did not become visible. Did I mention how much I love udev?

Much simpler: a shell alias

But one thing I'd noticed in all this: side by side with /dev/disk/by-uuid is a directory called /dev/disk/by-id. And the Samsung devices did show up there, with names like usb-Android_UMS_Composite_c197022a2b41c1ae-0:0.

Faced with the prospect of spending the rest of the day trying random udev rules, rebooting each time since that's the only way to test udev, I decided to cheat and find another way. I made a shell alias: 

alias galaxy='sudo sh -c "for d in /dev/disk/by-id/usb-Android*; do /lib/udev/ata_id --export \$d; done"'

Typing galaxy will now cause the Samsung to register both devices; and from then on I can mount and unmount them without needing root, using my normal fstab entries.

Update: This works for the Nook's main storage, too -- just add x/dev/disk/by-id/usb-B_N_Ebook_Disk* to the list -- but it doesn't work reliably for the Nook's SD card. The SD card does show up in /dev/disk/by-id along with main storage; but running ata_id on it doesn't make its UUID appear. I may just change my fstab entry to refer to the /dev/disk/by-id device directly.

Tags: , ,
[ 12:43 Apr 18, 2012    More linux/kernel | permalink to this entry | comments ]

Sat, 24 Mar 2012

Find out what processes are making network connections

A thread on the Ubuntu-devel-discuss mailing list last month asked about how to find out what processes are making outgoing network connectsion on a Linux machine. It referenced Ubuntu bug 820895: Log File Viewer does not log "Process Name", which is specific to Ubuntu's iptables logging of apps that are already blocked in iptables ... but the question goes deeper.

Several years ago, my job required me to use a program -- never mind which one -- from a prominent closed-source company. This program was doing various annoying things in addition to its primary task -- operations that got around the window manager and left artifacts all over my screen, operations that potentially opened files other than the ones I asked it to open -- but in addition, I noticed that when I ran the program, the lights on the DSL modem started going crazy. It looked like the program was making network connections, when it had no reason to do that. Was it really doing that?

Unfortunately, at the time I couldn't find any Linux command that would tell me the answer. As mentioned in the above Ubuntu thread, there are programs for Mac and even Windows to tell you this sort of information, but there's no obvious way to find out on Linux.

The discussion ensuing in the ubuntu-devel-discuss thread tossed around suggestions like apparmor and selinux -- massive, complex ways of putting up fortifications your whole system. But nobody seemed to have a simple answer to how to find information about what apps are making network connections.

Well, it turns out there are a a couple ofsimple way to get that list. First, you can use ss: 

$ ss -tp
State      Recv-Q Send-Q      Local Address:Port          Peer Address:Port   
ESTAB      0      0                     ::1:58466                  ::1:ircd     users:(("xchat",1063,43))
ESTAB      0      0             192.168.1.6:57526       140.211.166.64:ircd     users:(("xchat",1063,36))
ESTAB      0      0                     ::1:ircd                   ::1:58466    users:(("bitlbee",1076,10))
ESTAB      0      0             192.168.1.6:54253       94.125.182.252:ircd     users:(("xchat",1063,24))
ESTAB      0      0             192.168.1.6:52167       184.72.217.144:https
users:(("firefox-bin",1097,47))

-t shows only TCP connections (so you won't see all the interprocess communication among programs running on your machine). -p prints the process associated with each connection.

ss can do some other useful things, too, like show all the programs connected to your X server right now, or show all your ssh connections. See man ss for examples.

Or you can use netstat: 

$ netstat -A inet -p
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 imbrium.timochari:51800 linuxchix.osuosl.o:ircd ESTABLISHED 1063/xchat      
tcp        0      0 imbrium.timochari:59011 ec2-107-21-74-122.:ircd ESTABLISHED 1063/xchat      
tcp        0      0 imbrium.timochari:54253 adams.freenode.net:ircd ESTABLISHED 1063/xchat      
tcp        0      0 imbrium.timochari:58158 s3-1-w.amazonaws.:https ESTABLISHED
1097/firefox-bin

In both cases, the input is a bit crowded and hard to read. If all you want is a list of processes making connections, that's easy enough to do with the usual Unix utilities like grep and sed: 

$ ss -tp | grep -v Recv-Q | sed -e 's/.*users:(("//' -e 's/".*$//' | sort | uniq
$ netstat -A inet -p | grep '^tcp' | grep '/' | sed 's_.*/__' | sort | uniq

Finally, you can keep an eye on what's going on by using watch to run one of these commands repeatedly: 

watch ss -tp

Using watch with one of the pipelines to print only process names is possible, but harder since you have to escape a lot of quotation marks. If you want to do that, I recommend writing a script.

And back to the concerns expressed on the Ubuntu thread, you could also write a script to keep logs of which processes made connections over the course of a day. That's definitely a tool I'll keep in my arsenal.

Tags: , , ,
[ 11:28 Mar 24, 2012    More linux | permalink to this entry | comments ]

Tue, 03 Jan 2012

Open the X selection in a browser window, from any desktop

Like most Linux users, I use virtual desktops. Normally my browser window is on a desktop of its own.

Naturally, it often happens that I encounter a link I'd like to visit while I'm on a desktop where the browser isn't visible. From some apps, I can click on the link and have it show up. But sometimes, the link is just text, and I have to select it, change to the browser desktop, paste the link into firefox, then change desktops again to do something else while the link loads.

So I set up a way to load whatever's in the X selection in firefox no matter what desktop I'm on.

In most browsers, including firefox, you can tell your existing browser window to open a new link from the command line: firefox http://example.com/ opens that link in your existing browser window if you already have one up, rather than starting another browser. So the trick is to get the text you've selected.

At first, I used a program called xclip. You can run this command: firefox `xclip -o` to open the selection. That worked okay at first -- until I hit my first URL in weechat that was so long that it was wrapped to the next line. It turns out xclip does odd things with multi-line output; depending on whether it thinks the output is a terminal or not, it may replace the newline with a space, or delete whatever follows the newline. In any case, I couldn't find a way to make it work reliably when pasted into firefox.

After futzing with xclip for a little too long, trying to reverse-engineer its undocumented newline behavior, I decided it would be easier just to write my own X clipboard app in Python. I already knew how to do that, and it's super easy once you know the trick: 

mport gtk
primary = gtk.clipboard_get(gtk.gdk.SELECTION_PRIMARY)
if primary.wait_is_text_available() :
    print primary.wait_for_text()

That just prints it directly, including any newlines or spaces. But as long as I was writing my own app, why not handle that too?

It's not entirely necessary on Firefox: on Linux, Firefox has some special code to deal with pasting multi-line URLs, so you can copy a URL that spans multiple lines, middleclick in the content area and things will work. On other platforms, that's disabled, and some Linux distros disable it as well; you can enable it by going to about:config and searching for single, then setting the preference editor.singlelinepaste.pasteNewlines to 2.

However, it was easy enough to make my Python clipboard app do the right thing so it would work in any browser. I used Python's re (regular expressions) module: 

#!/usr/bin/env python

import gtk
import re

primary = gtk.clipboard_get(gtk.gdk.SELECTION_PRIMARY)

if not primary.wait_is_text_available() :
    sys.exit(0)
s = primary.wait_for_text()

# eliminate newlines, and any spaces immediately following a newline:
print re.sub(r'[\r\n]+ *', '', s)

That seemed to work fine, even on long URLs pasted from weechat with newlines and spaces, like that looked like 

http://example.com/long-
    url.html

All that was left was binding it so I could access it from anywhere. Of course, that varies depending on your desktop/window manager. In Openbox, I added two items to my desktop menu in menu.xml: 

  <item label="open selection in Firefox">
    <action name="Execute"><execute>sh -c 'firefox `xclip -o`'</execute></action>
  </item>
  <item label="open selection in new tab">
    <action name="Execute"><execute>sh -c 'firefox -new-tab `xclip -o`'</execute></action>
  </item>

I also added some code in rc.xml inside <context name="Desktop">, so I can middle-click or control-middle-click on the desktop to open a link in the browser: 

      <mousebind button="Middle" action="Press">
        <action name="Execute">
          <execute>sh -c 'firefox `pyclip`'</execute>
        </action>
      </mousebind>
      <mousebind button="C-Middle" action="Press">
        <action name="Execute">
          <execute>sh -c -new-tab 'firefox `pyclip`'</execute>
        </action>
      </mousebind>

I set this up maybe two hours ago and I've probably used it ten or fifteen times already. This is something I should have done long ago!

Tags: , , ,
[ 21:37 Jan 03, 2012    More linux | permalink to this entry | comments ]

Sun, 18 Dec 2011

Convert patterns in only some lines to title case

A friend had a fun problem: she had some XML files she needed to import into GNUcash, but the program that produced them left names in all-caps and she wanted them more readable. So she'd have a file like this: 

<STMTTRN>
   <TRNTYPE>DEBIT
   <DTPOSTED>20111125000000[-5:EST]
   <TRNAMT>-22.71
   <FITID>****

   <NAME>SOME    COMPANY
   <MEMO>SOME COMPANY    ANY TOWN   CA 11-25-11 330346
</STMTTRN>
and wanted to change the NAME and MEMO lines to read Some Company and Any Town. However, the tags, like <NAME>, all had to remain upper case, and presumably so did strings like DEBIT. How do you change just the NAME and MEMO lines from upper case to title case?

The obvious candidate to do string substitutes is sed. But there are several components to the problem.

Addresses

First, how do you ensure the replacement only happens on lines with NAME and MEMO?

sed lets you specify address ranges for just that purpose. If you say sed 's/xxx/yyy/' sed will change all xxx's to yyy; but if you say sed '/NAME/s/xxx/yyy/' then sed will only do that substitution on lines containing NAME.

But we need this to happen on lines that contain either NAME or MEMO. How do you do that? With \|, like this: sed '/\(NAME\|MEMO\)/s/xxx/yyy/'

Converting to title case

Next, how do you convert upper case to lower case? There's a sed command for that: \L. Run sed 's/.*/\L&/' and type some upper and lower case characters, and they'll all be converted to lower-case.

But here we want title case -- we want most of each word converted to lowercase, but the first letter should stay uppercase. That means we need to detect a word and figure out which is the first letter.

In the strings we're considering, a word is a set of letters A through Z with one of the following characteristics:

  1. It's preceded by a space
  2. It's preceded by a close-angle-bracket, >

So the pattern /[ >][A-Z]*/ will match anything we consider a word that might need conversion.

But we need to separate the first letter and the rest of the word, so we can treat them separately. sed's \( \) operators will let us do that. The pattern \([ >][A-Z]\) finds the first letter of a word (including the space or > preceding it), and saves that as its first matched pattern, \1. Then \([A-Z]*\) right after it will save the rest of the word as \2.

So, taking our \L case converter, we can convert to title case like this: sed 's/\([ >][A-Z]\)\([A-Z]*\)/\1\L\2/g

Starting to look long and scary, right? But it's not so bad if you build it up gradually from components. I added a g on the end to tell sed this is a global replace: do the operation on every word it finds in the line, otherwise it will only make the substitution once, on the first word it sees, then quit.

Putting it together

So we know how to seek out specific lines, and how to convert to title case. Put the two together, and you get the final command: 

sed '/\(NAME\|MEMO\)/s/\([ >][A-Z]\)\([A-Z]*\)/\1\L\2/g'

I ran it on the test input, and it worked just fine.

For more information on sed, a good place to start is the sed regular expressions manual.

Tags: , ,
[ 13:13 Dec 18, 2011    More linux/cmdline | permalink to this entry | comments ]

Tue, 13 Dec 2011

Taming the loud system beep

One of the distros I'm trying on my Dell Latitude 2120 laptop is Arch Linux. I like a lot of things about Arch; I had to stop using it on my previous laptop because something broke in the wi-fi drivers, but I always regretted giving it up. And it seems to work quite well on the Dell, with one exception: the system beep (which other distros don't support at all) was ear-shatteringly loud, far too loud to consider being able to use this laptop in a public space. Clearly that needed to be fixed.

The usual approach to system beeps, unloading or blacklisting the pcspkr module, had no effect. xset b off turned the beep off in X; I could also set its pitch and duration to change it to a nice quiet click, though I wasn't able to change the volume that way. I actually do like having a system beep, as long as it's fairly quiet and won't disturb people nearby. Unfortunately, xset b only affects the bell while in X; it didn't have any effect on the deafening sound Arch gave upon shutdown or reboot.

It turns out that on some laptops, including this Dell, the system beep goes not through the old-style pcspkr driver, but through the normal sound card. And the sound card has a separate channel for the system beep, so even if you have your volume turned down, the beep may still be at 100%. All I needed to do was run alsamixer and find out what the channel was called: "Beep".

Given that, I could use the amixer program to ensure the beep volume will be sane when I log in. I added the following to .zlogin (for zsh; obviously, adjust for your own shell): 

amixer -q -c 0 set Beep 5

That gave me a nice quiet beep. If I need to turn it off completely, amixer can do that too: amixer -q -c 0 set Beep mute (curiously, amixer -q -c 0 set Beep 0 doesn't actually set the volume to zero, just sets it very low).

That volume setting applies to the shutdown beep, too, fortunately. Though what I'd really like is to have quiet beeps while I'm running, but no shutdown beep at all. I don't understand the purpose of the shutdown beep; obviously I know when I've told my machine to shut down or reboot, so why do I need an audible reminder? But I've been unable to find anything explaining what's causing this beep. I tried adding a amixer -q -c 0 set Beep mute to /etc/rc.local.shutdown, but it didn't help; apparently the shutdown beep is called before that file is run. Which strongly suggests it is being run by Arch Linux, not by something in the BIOS. But nobody I've asked had any suggestions as to its source, or how to change it. Another enduring Linux mystery ...

Update: I mentioned xset b as a way to adjust beeps inside X -- in addition to turning beeps totally off, you can also set pitch, duration, and sometimes volume though the volume part didn't work for me. But outside X, you can make similar adjustments with setterm, e.g. setterm -bfreq 400 -blength 50. Thanks to Mikachu for the tip!

Tags: , ,
[ 11:05 Dec 13, 2011    More linux | permalink to this entry | comments ]

Sun, 11 Dec 2011

Set Ubuntu's system clock to use localtime, not UTC

Need your Ubuntu clock to stay in sync with a dual-boot Windows install? It seems to have changed over the years, and google wasn't finding any pages for me offering suggestions that still worked.

Turns out, in Ubuntu Oneiric Ocelot, it's controlled by the file /etc/default/rcS: set UTC=no.

Apparently it's also possible to get Windows to understand a UTC system clock using a registry tweak.

Ironically, that page, which I found by searching for windows system clock utc, also has the answer for setting Ubuntu to local time. So if I'd searched for Windows in the first place, I wouldn't have had to puzzle out the Ubuntu solution myself. Go figure!

[ 12:56 Dec 11, 2011    More linux | permalink to this entry | comments ]

Thu, 24 Nov 2011

Configuring extlinux's auto-update on Debian

A few days ago, I wrote about how to set up and configure extlinux (syslinux) as a bootloader. But on Debian or Ubuntu, if you make changes to files like /boot/extlinux/extlinux.conf directly, they'll be overwritten.

The configuration files are regenerated by a program called extlinux-update, which runs automatically every time you update your kernel. (Specifically, it runs from the postinst script of the linux-base package: you can see it in /var/lib/dpkg/info/linux-base.postinst.)

So what's a Debian user to do if she wants to customize the menus, add a splash image or boot other operating systems?

First, if you decide you really don't want Debian overwriting your configuration files, you can change disable updates by editing /etc/default/extlinux. Just be aware you won't get your boot menu updated when you install new kernels -- you'll have to remember to update them by hand.

It might be worth it: the automatic update is nearly as annoying as the grub2 updater: it creates two automatic entries for every kernel you have installed. So if you have several distros installed, each with a kernel or two in your shared /boot, you'll get an entry to boot Debian Squeeze with the Ubuntu Oneiric kernel, one for Squeeze with the Natty kernel, one for Squeeze with the Fedora 16 kernel ... as well as entries for every kernel you have that's actually owned by Debian. And then for each of these, you'll also get a second entry, to boot in recovery mode. If you have several distros installed, it makes for a very long and confusing boot menu!

It's a shame that the auto-updater doesn't restrict itself to kernels managed by the packaging system, which would be easy enough to do. (Wonder if they would accept a patch?) You might be able to fudge something that works right by setting up symlinks so that the only readable kernels actually live on the root partition, so Debian can't read the kernels from the other distros. Sounds a bit complicated and I haven't tried it. For now, I've turned off automatic updating on my system.

But if your setup is simpler -- perhaps just one Debian or one Ubuntu partition plus some non-Linux entries such as BSD or Windows -- here's how to set up Debian-style automatic updating and still keep all your non-Linux boot entries and your nice menu customizations.

Debian automatic updates and themes

First, take a quick look at /etc/default/extlinux and customize anything there you might need, like the names of the kernels, kernel boot parameters or timeout. See man extlinux-update for details.

For configuring menu colors, image backgrounds and such, you'll need to make a theme. You can see a sample theme by installing the package syslinux-themes-debian -- but watch out. If you haven't configured apt not to pull in suggested packages, that may bring back grub or grub-legacy, which you probably don't want.

You can make a theme without needing that package, though. Create a directory /usr/share/syslinux/themes/mythemename (the extlinux-update man page claims you can put a theme anywhere and specify it by its full path, but it lies). Create a directory called extlinux inside it, and make a file with everything you want from extlinux.conf. For example: 

default 0
prompt 1
timeout 50

ui vesamenu.c32
menu title Welcome to my Linux machine!
menu background mysplash.png
menu color title 1;36 #ffff8888 #00000000 std
menu color unsel 0    #ffffffff #00000000 none
menu color sel   7    #ff000000 #ffffff00 none

include linux.cfg
menu separator
include themes/mythemename/other.cfg

Note that last line: you can include other files from your theme. For instance, you can create a file called other.cfg with entries for other partitions you want to boot: 

label oneiric
menu label Ubuntu Oneiric Ocelot
kernel /vmlinuz-3.0.0-12-generic
append initrd=/initrd.img-3.0.0-12-generic root=UUID=c332b3e2-5c38-4c50-982a-680af82c00ab ro quiet

label fedora
menu label Fedora 16
kernel /vmlinuz-3.1.0-7.fc16.i686
append initrd=/initramfs-3.1.0-7.fc16.i686.img root=UUID=47f6b1fa-eb5d-4254-9fe0-79c8b106f0d9 ro quiet

menu separator

LABEL Windows
KERNEL chain.c32
APPEND hd0 1

Of course, you could have a debian.cfg, an ubuntu.cfg, a fedora.cfg etc. if you wanted to have multiple distros all keeping their kernels up-to-date. Or you can keep the whole thing in one file, theme.cfg. You can make a theme as complex or as simple as you like.

Tags: , , , , ,
[ 11:26 Nov 24, 2011    More linux/install | permalink to this entry | comments ]

Syndicated on:
• LinuxChix Live
• Ubuntu Women
• Women in Free Software
• Graphics Planet
• DevChix
• Ubuntu California
• Planet Openbox
• Devchix
• Planet LCA2009

Friends' Blogs:
• Ups & Downs
• DailyBBG
• Long Live the Village Green
• Dan Heller
• Morris "Mojo" Jones
• Jane Houston Jones

Other Blogs:
• DevChix
• Scott Adams
• Dave Barry
• BoingBoing (Cory Doctorow)
• Young Female Scientist

Powered by PyBlosxom.