Shallow Thoughts : : linux

Akkana's Musings on Open Source Computing and Technology, Science, and Nature.

Fri, 30 Oct 2015

HDMI presentation setup on Linux, Part II: problems and tips

In Part I of HDMI Presentation Setup on Linux, I covered the basics of getting video and audio working over HDMI. Now I want to cover some finer-grained details: some problems I had, and ways to make it easier to enable HDMI when you need it.

Testing follies, delays, and screen blinking/flashing woes

While I was initially trying to get this working, I was using my own short sound clip (one of the alerts I use for IRC) and it wasn't working. Then I tried the test I showed in part I, $ aplay -D plughw:0,3 /usr/share/sounds/alsa/Front_Center.wav and that worked fine. Tried my sound clip again -- nothing. I noticed that my clip was mono and 8-bit while the ALSA sample was stereo and 16-bit, and I wasted a lot of time in web searches on why HDMI would play one and not the other.

Eventually I figured out that the reason my short clip wasn't playing was that there's a delay when switching on HDMI sound, and the first second two two of any audio may be skipped. I found lots of complaints about people missing the first few seconds of sound over HDMI, so this problem is quite common, and I haven't found a solution.

So if you're giving a talk where you need to play short clips -- for instance, a talk on bird calls -- be aware of this. I'm probably going to make a clip of a few seconds of silence, so I can play silence before every short clip to make sure I'm fully switched over to HDMI before the clip starts: aplay -D plughw:0,3 silence.wav osprey.wav

Another problem, probably related, when first starting an audio file: the screen blinks brieftly off then on again, then blinks again a little while after the clip ends. ("Flicker" turns out to be a better term to use when web searching, though I just see a single blink, not continued flickering). It's possible this is something about my home TV, and I will have to try it with another monitor somewhere to see if it's universal. It sounds like kernel bug 51421: Enabling HDMI sound makes HDMI video flicker, but that bug was marked resolved in 2012 and I'm seeing this in 2015 on Debian Jessie.

Making HDMI the sound default

What a pain, to have to remember to add -D plughw:0,3 every time you play a sound. And what do you do for other programs that don't have that argument?

Fortunately, you can make HDMI your default sound output. Create a file in your home directory called .asoundrc with this in it (you may be able to edit this down -- I didn't try) and then all audio will go to HDMI:

pcm.dmixer {
  type dmix
  ipc_key 1024
  ipc_key_add_uid false
  ipc_perm 0660
  slave {
    pcm "hw:0,3"
    rate 48000
    channels 2
    period_time 0
    period_size 1024
    buffer_time 0
    buffer_size 4096
pcm. !default {
  type plug
  slave.pcm "dmixer"

Great! But what about after you disconnect? Audio will still be going to HDMI ... in other words, nowhere. So rename that file:

$ mv .asoundrc asoundrc-hdmi
Then when you connect to HDMI, you can copy it back:
$ cp asoundrc-hdmi .asoundrc 

What a pain, you say again! This should happen automatically!

That's possible, but tricky: you have to set up udev rules and scripts. See this Arch Linux discussion on HDMI audio output switching automatically for the gory details. I haven't bothered, since this is something I'll do only rarely, when I want to give one of those multimedia presentations I sometimes contemplate but never actually give. So for me, it's not worth fighting with udev when, by the time I actually need HDMI audio, the udev syntax probably will have changed again.

Aliases to make switching easy

But when I finally do break down and design a multimedia presentation, I'm not going to be wanting to do all this fiddling in the presentation room right before the talk. I want to set up aliases to make it easy.

There are two things that need to be done in that case: make HDMI output the default, and make sure it's unmuted.

Muting can be done automatically with amixer. First run amixer with no arguments to find out the channel name (it gives a lot of output, but look through the "Simple mixer control" lines, or speed that up with amixer | grep control.

Once you know the channel name (IEC958 on my laptop), you can run: amixer sset IEC958 unmute The rest of the alias is just shell hackery to create a file called .asoundrc with the right stuff in it, and saving .asoundrc before overwriting it. My alias in .zshrc is set up so that I can say hdmisound on or hdmisound off (with no arguments, it assumes on), and it looks like this:

# Send all audio output to HDMI.
# Usage: hdmisound [on|off], default is on.
hdmisound() {
    if [[ $1 == 'off' ]]; then
        if [[ -f ~/.asoundrc ]]; then
            mv ~/.asoundrc ~/.asoundrc.hdmi
        amixer sset IEC958 mmute
        if [[ -f ~/.asoundrc ]]; then
            mv ~/.asoundrc ~/.asoundrc.nohdmi
        cat >> ~/.asoundrc <<EOF
pcm.dmixer {
  type dmix
  ipc_key 1024
  ipc_key_add_uid false
  ipc_perm 0660
  slave {
    pcm "hw:0,3"
    rate 48000
    channels 2
    period_time 0
    period_size 1024
    buffer_time 0
    buffer_size 4096
pcm. !default {
  type plug
  slave.pcm "dmixer"
        amixer sset IEC958 unmute

Of course, I could put all that .asoundrc content into a file and just copy/rename it each time. But then I have another file I need to make sure is in place on every laptop; I decided I'd rather make the alias self-contained in my .zshrc.

Tags: , ,
[ 11:57 Oct 30, 2015    More linux/laptop | permalink to this entry | comments ]

Tue, 27 Oct 2015

HDMI presentation setup on Linux, video and audio: Part I

For about a decade now I've been happily connecting to projectors to give talks. xrandr --output VGA1 --mode 1024x768 switches on the laptop's external VGA port, and xrandr --auto turns it off again after I'm disconnected. No fuss.

But increasingly, local venues are eschewing video projectors and instead using big-screen TVs, some of which offer only an HDMI port, no VGA. I thought I'd better figure out how to present a talk over HDMI now, so I'll be ready when I need to know.

Fortunately, my newest laptop does have an HDMI port. But in case it ever goes on the fritz and I have to use an older laptop, I discovered you can buy VGA to HDMI adaptors rather cheaply (about $10) on ebay. I bought one of those, tested it on my TV at home and it at least worked there. Be careful when shopping: you want to make sure you're getting something that takes VGA in and outputs HDMI, rather than the reverse. Ebay descriptions aren't always 100% clear on that, but if you check the gender of the connector in the photo and make sure it's right to plug into the socket on your laptop, you should be all right.

Once you're plugged in (whether via an adaptor, or native HDMI built into your laptop), connecting is easy, just like connecting with VGA:

xrandr --output HDMI1 --mode 1024x768

Of course, you can modify the resolution as you see fit. I plan to continue to design my presentations for a 1024x768 resolution for the forseeable future. Since my laptop is 1366x1024, I can use the remaining 342-pixel-wide swath for my speaker notes and leave them invisible to the audience.

But for GIMP presentations, I'll probably want to use the full width of my laptop screen. --mode 1366x768 didn't work -- that resolution wasn't available -- but running xrandr with no arguments got me a list of available resolutions, which included 1360x768. That worked fine and is what I'll use for GIMP talks and other live demos where I want more screen space.

Sound over HDMI

My Toastmasters club had a tech session where a few of us tried out the new monitor in our meeting room to make sure we could use it. One person was playing a video with sound. I've never used sound in a talk, but I've always wanted to find an excuse to try it. Alas, it didn't "just work" -- xrandr's video settings have nothing to do with ALSA's audio settings. So I had to wait until I got home so I could do web searches and piece together the answer.

First, run aplay -l , which should show something like this:

$ aplay -l
**** List of PLAYBACK Hardware Devices ****
card 0: Intel [HDA Intel], device 0: STAC92xx Analog [STAC92xx Analog]
  Subdevices: 0/1
  Subdevice #0: subdevice #0
card 0: Intel [HDA Intel], device 3: HDMI 0 [HDMI 0]
  Subdevices: 1/1
  Subdevice #0: subdevice #0

Find the device number for the HDMI device, which I've highlighted here: in this case, it's 3 (which seems to be common on Intel chipsets).

Now you can run a test:

$ aplay -D plughw:0,3 /usr/share/sounds/alsa/Front_Center.wav
Playing WAVE '/usr/share/sounds/alsa/Front_Center.wav' : Signed 16 bit Little Endian, Rate 48000 Hz, Mono
If you don't hear anything, don't worry: the HDMI channel is probably muted if you've never used it before. Run either alsamixer or alsamixergui.

[alsamixergui with HDMI muted] [alsamixer] Now find the channel representing your HDMI connection. (HDMI must be plugged in for this to work.) In alsamixer, it's called S/PDIF; in alsamixergui, it's called IEC958. If you look up either of those terms, Wikipedia S/PDIF will tell you that S/PDIF is the Sony/Philips Digital Interconnect Format, a data protocol and a set of physical specifications. Those physical specifications appear to have nothing to do with video, and use connectors that are nothing like HDMI. So it doesn't make much sense. Just remember that if you see IEC958 or S/PDIF in ALSA, that's probably your HDMI channel.

In the alsamixergui screenshot, IEC958 is muted: you can tell because the little speaker icon at the top of the column is bright white. If it were unmuted, the speaker icon would be grey like most of the others. Yes, this seems backward. It's Linux audio: get used to obscure user interfaces.

In the alsamixer screenshot, the mutes are at the bottom of each column, and MM indicates a channel is muted (like the Beep channel in the screenshot). S/PDIF is not muted here, though it appears to be at zero volume. (The 00 doesn't tell you it's at zero volume; 00 means it's not muted. What did I say about Linux audio?) ALSA apparently doesn't let you adjust the volume of HDMI output: presumably they expect that your HDMI monitor will have its own volume control. If your S/PDIF is muted, you can use your right-arrow key to arrow over to the S/PDIF channel, then type m to toggle muting. You can exit alsamixer with Ctrl-C (Q and Ctrl-Q don't work).

Now try that aplay -D command again and see if it works. With any luck, it will (loudly).

A couple of other tests you might want to try:
speaker-test -t sine -f 440 -c 2 -s 1 -D hw:0,3
plays a sine wave. speaker-test -c 2 -r 48000 -D hw:0,3
runs a general speaker test sequence.

In Part II of Linux HDMI Presentations, I'll cover some problems I had, and how to write an alias to make it easy to turn HDMI audio on and off.

Tags: , ,
[ 14:36 Oct 27, 2015    More linux/laptop | permalink to this entry | comments ]

Thu, 15 Oct 2015

Viewer for email attachments in Office formats

I seem to have fallen into a nest of Mac users whose idea of email is a text part, an HTML part, plus two or three or seven attachments (no exaggeration!) in an unholy combination of .DOC, .DOCX, .PPT and other Microsoft Office formats, plus .PDF.

Converting to text in mutt

As a mutt user who generally reads all email as plaintext, normally my reaction to a mess like that would be "Thanks, but no thanks". But this is an organization that does a lot of good work despite their file format habits, and I want to help.

In mutt, HTML mail attachments are easy. This pair of entries in ~/.mailcap takes care of them:

text/html; firefox 'file://%s'; nametemplate=%s.html
text/html; lynx -dump %s; nametemplate=%s.html; copiousoutput
Then in .muttrc, I have
auto_view text/html
alternative_order text/plain text

If a message has a text/plain part, mutt shows that. If it has text/html but no text/plain, it looks for the "copiousoutput" mailcap entry, runs the HTML part through lynx (or I could use links or w3m) and displays that automatically. If, reading the message in lynx, it looks to me like the message has complex formatting that really needs a browser, I can go to mutt's attachments screen and display the attachment in firefox using the other mailcap entry.

Word attachments are not quite so easy, especially when there are a lot of them. The straightforward way is to save each one to a file, then run LibreOffice on each file, but that's slow and tedious and leaves a lot of temporary files behind. For simple documents, converting to plaintext is usually good enough to get the gist of the attachments. These .mailcap entries can do that:

application/msword; catdoc %s; copiousoutput
application/vnd.openxmlformats-officedocument.wordprocessingml.document; docx2txt %s -; copiousoutput
Alternatives to catdoc include wvText and antiword.

But none of them work so well when you're cross-referencing five different attachments, or for documents where color and formatting make a difference, like mail from someone who doesn't know how to get their mailer to include quoted text, and instead distinguishes their comments from the text they're replying to by making their new comments green (ugh!) For those, you really do need a graphical window.

I decided what I really wanted (aside from people not sending me these crazy emails in the first place!) was to view all the attachments as tabs in a new window. And the obvious way to do that is to convert them to formats Firefox can read.

Converting to HTML

I'd used wvHtml to convert .doc files to HTML, and it does a decent job and is fairly fast, but it can't handle .docx. (People who send Office formats seem to distribute their files fairly evenly between DOC and DOCX. You'd think they'd use the same format for everything they wrote, but apparently not.) It turns out LibreOffice has a command-line conversion program, unoconv, that can handle any format LibreOffice can handle. It's a lot slower than wvHtml but it does a pretty good job, and it can handle .ppt (PowerPoint) files too.

For PDF files, I tried using pdftohtml, but it doesn't always do so well, and it's hard to get it to produce a single HTML file rather than a directory of separate page files. And about three quarters of PDF files sent through email turn out to be PDF in name only: they're actually collections of images of single pages, wrapped together as a PDF file. (Mostly, when I see a PDF like that I just skip it and try to get the information elsewhere. But I wanted my program at least to be able to show what's in the document, and let the user choose whether to skip it.) In the end, I decided to open a firefox tab and let Firefox's built-in PDF reader show the file, though popping up separate mupdf windows is also an option.

I wanted to show the HTML part of the email, too. Sometimes there's formatting there (like the aforementioned people whose idea of quoting messages is to type their replies in a different color), but there can also be embedded images. Extracting the images and showing them in a browser window is a bit tricky, but it's a problem I'd already solved a couple of years ago: Viewing HTML mail messages from Mutt (or other command-line mailers).

Showing it all in a new Firefox window

So that accounted for all the formats I needed to handle. The final trick was the firefox window. Since some of these conversions, especially unoconv, are quite slow, I wanted to pop up a window right away with a "converting, please wait..." message. Initially, I used a javascript: URL, running the command:

firefox -new-window "javascript:document.writeln('<br><h1>Translating documents, please wait ...</h1>');"

I didn't want to rely on Javascript, though. A data: URL, which I hadn't used before, can do the same thing without javascript:

firefox -new-window "data:text/html,<br><br><h1>Translating documents, please wait ...</h1>"

But I wanted the first attachment to replace the contents of that same window as soon as it was ready, and then subsequent attachments open a new tab in that window. But it turned out that firefox is inconsistent about what -new-window and -new-tab do; there's no guarantee that -new-tab will show up in the same window you recently popped up with -new-window, and running just firefox URL might open in either the new window or the old, in a new tab or not, or might not open at all. And things got even more complicated after I decided that I should use -private-window to open these attachments in private browsing mode.

In the end, the only way firefox would behave in a repeatable, predictable way was to use -private-window for everything. The first call pops up the private window, and each new call opens a new tab in the private window. If you want two separate windows for two different mail messages, you're out of luck: you can't have two different private windows. I decided I could live with that; if it eventually starts to bother me, I can always give up on Firefox and write a little python-webkit wrapper to do what I need.

Using a file redirect instead

But that still left me with no way to replace the contents of the "Please wait..." window with useful content. Someone on #firefox came up with a clever idea: write the content to a page with a meta redirect.

So initially, I create a file pleasewait.html that includes the header:

<meta http-equiv="refresh" content="2;URL=pleasewait.html">
(other HTML, charset information, etc. as needed). The meta refresh means Firefox will reload the file every two seconds. When the first converted file is ready, I just change the header to redirect to URL=first_converted_file.html. Meanwhile, I can be opening the other documents in additional tabs.

Finally, I added the command to my .muttrc. When I'm viewing a message either in the index or pager screens, F10 will call the script and decode all the attachments.

macro index <F10> "<pipe-message>~/bin/viewmailattachments\n" "View all attachments in browser"
macro pager <F10> "<pipe-message>~/bin/viewmailattachments\n" "View all attachments in browser"

Whew! It was trickier than I thought it would be. But I find I'm using it quite a bit, and it takes a lot of the pain out of those attachment-full emails.

The script is available at: viewmailattachments on GitHub.

Tags: , , , , ,
[ 15:18 Oct 15, 2015    More linux | permalink to this entry | comments ]

Sun, 11 Oct 2015

How to get X output redirection back

X stopped working after my last Debian update.

Rather than run a login manager, I typically log in on the console. Then in my .zlogin file, I have:

if [[ $(tty) == /dev/tty1 ]]; then
  # do various things first, then:
  startx -- -dumbSched >& $HOME/.xsession-errors
Ignore -dumbSched for now; it's a fix for a timing problem openbox has when bringing up initial windows. The relevant part here is that I redirect both standard output and standard error to a file named .xsession-errors. That means that if I run GIMP or firefox or any other program from a menu, and later decide I need to see their output to look for error messages, all I have to do is check that file.

But as of my last update, that no longer works. Plain startx, without the output redirection, works fine. But with the redirect, X pauses for five or ten seconds, then exits, giving me my prompt back but with messed-up terminal settings, so I have to type reset before I do anything else.

Of course, I checked that .xsession file for errors, and also the ~/.local/share/xorg/Xorg.0.log file it referred me to (which is where X stores its log now that it's no longer running as root). It seems the problem is this:

Fatal server error:
(EE) xf86OpenConsole: VT_ACTIVATE failed: Operation not permitted
Which wasn't illuminating but at least gave me a useful search keyword.

I found a fair number of people on the web having the same problem. It's related to the recent Xorg change that makes it possible to run Xorg as a regular user, not root. Not that running as a user should have anything to do with capturing standard output and error. But apparently Xorg running as a user is dependent on what sort of virtual terminal it was run from; and the way it determines the controlling terminal, apparently, is by checking stderr (and maybe also stdout).

Here's a slightly longer description of what it's doing, from the ever useful Arch Linux forums.

I'm fairly sure there are better ways of determining a process's controlling terminal than using stderr. For instance, a casual web search turned up ctermid; or you could do checks on /dev/tty. There are probably other ways.

The Arch Linux thread linked above, and quite a few others, suggest adding the server option -keeptty when starting X. The Xorg manual isn't encouraging about this as a solution:

Prevent the server from detaching its initial controlling terminal. This option is only useful when debugging the server. Not all platforms support (or can use) this option.
But it does work.

I found several bugs filed already on the redirection problem. Freedesktop has a bug report on it, but it's more than a year old and has no comments or activity: Freedesktop bug 82732: rootless X doesn't start if stderr redirected.

Redhat has a bug report: Xorg without root rights breaks by streams redirection, and supposedly added a fix way back in January in their package version xorg-x11-xinit-1.3.4-3.fc21 ... though it looks like their fix is simply to enable -keeptty automatically, which is better than nothing but doesn't seem ideal. Still, it does suggest that it's probably not harmful to use that workaround and ignore what the Xorg man page says.

Debian didn't seem to have a bug filed on the problem yet (not terribly surprising, since they only enabled it in unstable a few days ago), so I used reportbug to attempt to file one. I would link to it here if Debian had an actual bug system that allowed searching for bugs (they do have a page entitled "BTS Search" but it gives "Internal Server Error", and the alternate google groups bug search doesn't find my bug), or if their bug reporting system acknowledged new bugs by emailing the submitter the bug number. In truth I strongly suspect that reportbug is actually a no-op and doesn't actually do anything with the emailed report.

But I'm not sure the Debian bug matters since the real bug is Xorg's, and it doesn't look like they're very interested in the problem. So anyone who wants to be able to access output of programs running under X probably needs to use -keeptty for the forseeable future.

Update: the bug acknowledgement came in six hours later. It's bug 801529.

Tags: , ,
[ 12:31 Oct 11, 2015    More linux | permalink to this entry | comments ]

Sun, 26 Jul 2015

Trackpad workarounds: using function keys as mouse buttons

I've had no end of trouble with my Asus 1015E's trackpad. A discussion of laptops on a mailing list -- in particular, someone's concerns that the nifty-looking Dell XPS 13, which is available preloaded with Linux, has had reviewers say that the trackpad doesn't work well -- reminded me that I'd never posted my final solution.

The Asus's trackpad has two problems. First, it's super sensitive to taps, so if any part of my hand gets anywhere near the trackpad while I'm typing, suddenly it sees a mouse click at some random point on the screen, and instead of typing into an emacs window suddenly I find I'm typing into a live IRC client. Or, worse, instead of typing my password into a password field, I'm typing it into IRC. That wouldn't have been so bad on the old style of trackpad, where I could just turn off taps altogether and use the hardware buttons; this is one of those new-style trackpads that doesn't have any actual buttons.

Second, two-finger taps don't work. Three-finger taps work just fine, but two-finger taps: well, I found when I wanted a right-click (which is what two-fingers was set up to do), I had to go TAP, TAP, TAP, TAP maybe ten or fifteen times before one of them would finally take. But by the time the menu came up, of course, I'd done another tap and that canceled the menu and I had to start over. Infuriating!

I struggled for many months with synclient's settings for tap sensitivity and right and left click emulation. I tried enabling syndaemon, which is supposed to disable clicks as long as you're typing then enable them again afterward, and spent months playing with its settings, but in order to get it to work at all, I had to set the timeout so long that there was an infuriating wait after I stopped typing before I could do anything.

I was on the verge of giving up on the Asus and going back to my Dell Latitude 2120, which had an excellent trackpad (with buttons) and the world's greatest 10" laptop keyboard. (What the Dell doesn't have is battery life, and I really hated to give up the Asus's light weight and 8-hour battery life.) As a final, desperate option, I decided to disable taps completely.

Disable taps? Then how do you do a mouse click?

I theorized, with all Linux's flexibility, there must be some way to get function keys to work like mouse buttons. And indeed there is. The easiest way seemed to be to use xmodmap (strange to find xmodmap being the simplest anything, but there you go). It turns out that a simple line like

  xmodmap -e "keysym F1 = Pointer_Button1"
is most of what you need. But to make it work, you need to enable "mouse keys":
  xkbset m

But for reasons unknown, mouse keys will expire after some set timeout unless you explicitly tell it not to. Do that like this:

  xkbset exp =m

Once that's all set up, you can disable single-finger taps with synclient:

  synclient TapButton1=0
Of course, you can disable 2-finger and 3-finger taps by setting them to 0 as well. I don't generally find them a problem (they don't work reliably, but they don't fire on their own either), so I left them enabled.

I tried it and it worked beautifully for left click. Since I was still having trouble with that two-finger tap for right click, I put that on a function key too, and added middle click while I was at it. I don't use function keys much, so devoting three function keys to mouse buttons wasn't really a problem.

In fact, it worked so well that I decided it would be handy to have an additional set of mouse keys over on the other side of the keyboard, to make it easy to do mouse clicks with either hand. So I defined F1, F2 and F3 as one set of mouse buttons, and F10, F11 and F12 as another.

And yes, this all probably sounds nutty as heck. But it really is a nice laptop aside from the trackpad from hell; and although I thought Fn-key mouse buttons would be highly inconvenient, it took surprisingly little time to get used to them.

So this is what I ended up putting in .config/openbox/autostart file. I wrap it in a test for hostname, since I like to be able to use the same configuration file on multiple machines, but I don't need this hack on any machine but the Asus.

if [ $(hostname) == iridum ]; then
  synclient TapButton1=0 TapButton2=3 TapButton3=2 HorizEdgeScroll=1

  xmodmap -e "keysym F1 = Pointer_Button1"
  xmodmap -e "keysym F2 = Pointer_Button2"
  xmodmap -e "keysym F3 = Pointer_Button3"

  xmodmap -e "keysym F10 = Pointer_Button1"
  xmodmap -e "keysym F11 = Pointer_Button2"
  xmodmap -e "keysym F12 = Pointer_Button3"

  xkbset m
  xkbset exp =m
  synclient TapButton1=1 TapButton2=3 TapButton3=2 HorizEdgeScroll=1

Tags: , , , , , ,
[ 20:54 Jul 26, 2015    More linux | permalink to this entry | comments ]

Fri, 15 May 2015

Of file modes, umasks and fmasks, and mounting FAT devices

I have a bunch of devices that use VFAT filesystems. MP3 players, camera SD cards, SD cards in my Android tablet. I mount them through /etc/fstab, and the files always look executable, so when I ls -f them, they all have asterisks after their names. I don't generally execute files on these devices; I'd prefer the files to have a mode that doesn't make them look executable.

I'd like the files to be mode 644 (or 0644 in most programming languages, since it's an octal, or base 8, number). 644 in binary is 110 100 100, or as the Unix ls command puts it, rw-r--r--.

There's a directive, fmask, that you can put in fstab entries to control the mode of files when the device is mounted. (Here's Wikipedia's long umask article.) But how do you get from the mode you want the files to be, 644, to the mask?

The mask (which corresponds to the umask command) represent the bits you don't want to have set. So, for instance, if you don't want the world-execute bit (1) set, you'd put 1 in the mask. If you don't want the world-write bit (2) set, as you likely don't, put 2 in the mask. So that's already a clue that I'm going to want the rightmost byte to be 3: I don't want files mounted from my MP3 player to be either world writable or executable.

But I also don't want to have to puzzle out the details of all nine bits every time I set an fmask. Isn't there some way I can take the mode I want the files to be -- 644 -- and turn them into the mask I'd need to put in /etc/fstab or set as a umask?

Fortunately, there is. It seemed like it ought to be straightforward, but it took a little fiddling to get it into a one-line command I can type. I made it a shell function in my .zshrc:

# What's the complement of a number, e.g. the fmask in fstab to get
# a given file mode for vfat files? Sample usage: invertmask 755
invertmask() {
    python -c "print '0%o' % (~(0777 & 0$1) & 0777)"

This takes whatever argument I give to it -- $1 -- and takes only the three rightmost bytes from it, (0777 & 0$1). It takes the bitwise NOT of that, ~. But the result of that is a negative number, and we only want the three rightmost bytes of the result, (result) & 0777, expressed as an octal number -- which we can do in python by printing it as %o. Whew!

Here's a shorter, cleaner looking alias that does the same thing, though it's not as clear about what it's doing:

invertmask1() {
    python -c "print '0%o' % (0777 - 0$1)"

So now, for my MP3 player I can put this in /etc/fstab:

UUID=0000-009E /mp3 vfat user,noauto,exec,fmask=133,shortname=lower 0 0

Tags: ,
[ 10:27 May 15, 2015    More linux/cmdline | permalink to this entry | comments ]

Mon, 23 Feb 2015

Tips for developing on a web host that offers only FTP

Generally, when I work on a website, I maintain a local copy of all the files. Ideally, I use version control (git, svn or whatever), but failing that, I use rsync over ssh to keep my files in sync with the web server's files.

But I'm helping with a local nonprofit's website, and the cheap web hosting plan they chose doesn't offer ssh, just ftp.

While I have to question the wisdom of an ISP that insists that its customers use insecure ftp rather than a secure encrypted protocol, that's their problem. My problem is how to keep my files in sync with theirs. And the other folks working on the website aren't developers and are very resistant to the idea of using any version control system, so I have to be careful to check for changed files before modifying anything.

In web searches, I haven't found much written about reasonable workflows on an ftp-only web host. I struggled a lot with scripts calling ncftp or lftp. But then I discovered curftpfs, which makes things much easier.

I put a line in /etc/fstab like this: /servername fuse rw,allow_other,noauto,user 0 0

Then all I have to do is type mount /servername and the ftp connection is made automagically. From then on, I can treat it like a (very slow and somewhat limited) filesystem.

For instance, if I want to rsync, I can

rsync -avn --size-only /servername/subdir/ ~/servername/subdir/
for any particular subdirectory I want to check. A few things to know about this:
  1. I have to use --size-only because timestamps aren't reliable. I'm not sure whether this is a problem with the ftp protocol, or whether this particular ISP's server has problems with its dates. I suspect it's a problem inherent in ftp, because if I ls -l, I see things like this:
    -rw-rw---- 1 root root 7651 Feb 23  2015 guide-geo.php
    -rw-rw---- 1 root root 1801 Feb 14 17:16 guide-header.php
    -rw-rw---- 1 root root 8738 Feb 23  2015 guide-table.php
    Note that a file modified a week ago shows a modification time, but files modified today show only a day and year, not a time. I'm not sure what to make of this.
  2. Note the -n flag. I don't automatically rsync from the server to my local directory, because if I have any local changes newer than what's on the server they'd be overwritten. So I check the diffs by hand with tkdiff or meld before copying.
  3. It's important to rsync only the specific directories you're working on. You really don't want to see how long it takes to get the full file tree of a web server recursively over ftp.

How do you change and update files? It is possible to edit the files on the curlftpfs filesystem directly. But at least with emacs, it's incredibly slow: emacs likes to check file modification dates whenever you change anything, and that requires an ftp round-trip so it could be ten or twenty seconds before anything you type actually makes it into the file, with even longer delays any time you save.

So instead, I edit my local copy, and when I'm ready to push to the server, I cp filename /servername/path/to/filename.

Of course, I have aliases and shell functions to make all of this easier to type, especially the long pathnames: I can't rely on autocompletion like I usually would, because autocompleting a file or directory name on /servername requires an ftp round-trip to ls the remote directory.

Oh, and version control? I use a local git repository. Just because the other people working on the website don't want version control is no reason I can't have a record of my own changes.

None of this is as satisfactory as a nice git or svn repository and a good ssh connection. But it's a lot better than struggling with ftp clients every time you need to test a file.

Tags: ,
[ 19:46 Feb 23, 2015    More linux | permalink to this entry | comments ]

Thu, 19 Feb 2015

Finding core dump files

Someone on the SVLUG list posted about a shell script he'd written to find core dumps.

It sounded like a simple task -- just locate core | grep -w core, right? I mean, any sensible packager avoids naming files or directories "core" for just that reason, don't they?

But not so: turns out in the modern world, insane numbers of software projects include directories called "core", including projects that are developed primarily on Linux so you'd think they would avoid it ... even the kernel. On my system, locate core | grep -w core | wc -l returned 13641 filenames.

Okay, so clearly that isn't working. I had to agree with the SVLUG poster that using "file" to find out which files were actual core dumps is now the only reliable way to do it. The output looks like this:

$ file core
core: ELF 32-bit LSB core file Intel 80386, version 1 (SYSV), too many program headers (375)

The poster was using a shell script, but I was fairly sure it could be done in a single shell pipeline. Let's see: you need to run locate to find any files with 'core" in the name.

Then you pipe it through grep to make sure the filename is actually core: since locate gives you a full pathname, like /lib/modules/3.14-2-686-pae/kernel/drivers/edac/edac_core.ko or /lib/modules/3.14-2-686-pae/kernel/drivers/memstick/core, you want lines where only the final component is core -- so core has a slash before it and an end-of-line (in grep that's denoted by a dollar sign, $) after it. So grep '/core$' should do it.

Then take the output of that locate | grep and run file on it, and pipe the output of that file command through grep to find the lines that include the phrase 'core file'.

That gives you lines like

/home/akkana/geology/NorCal/pinnaclesGIS/core: ELF 32-bit LSB core file Intel 80386, version 1 (SYSV), too many program headers (523)

But those lines are long and all you really need are the filenames; so pass it through sed to get rid of anything to the right of "core" followed by a colon.

Here's the final command:

file `locate core | grep '/core$'` | grep 'core file' | sed 's/core:.*//'

On my system that gave me 11 files, and they were all really core dumps. I deleted them all.

Tags: ,
[ 12:54 Feb 19, 2015    More linux | permalink to this entry | comments ]