Shallow Thoughts : tags : imaging

Zsh magic: remove all raw photos that don't have a corresponding JPEG

Lately, when shooting photos with my DSLR, I've been shooting raw mode but with a JPEG copy as well. When I triage and label my photos (with pho and metapho), I use only the JPEG files, since they load faster and there's no need to index both. But that means that sometimes I delete a .jpg file while the huge .cr2 raw file is still on my disk.

I wanted some way of removing these orphaned raw files: in other words, for every .cr2 file that doesn't have a corresponding .jpg file, delete the .cr2.

That's an easy enough shell function to write: loop over *.cr2, change the .cr2 extension to .jpg, check whether that file exists, and if it doesn't, delete the .cr2.

But as I started to write the shell function, it occurred to me: this is just the sort of magic trick zsh tends to have built in.

So I hopped on over to #zsh and asked, and in just a few minutes, I had an answer:

rm *.cr2(e:'[[ ! -e ${REPLY%.cr2}.jpg ]]':)

Yikes! And it works! But how does it work? It's cheating to rely on people in IRC channels without trying to understand the answer so I can solve the next similar problem on my own.

Most of the answer is in the zshexpn man page, but it still took some reading and jumping around to put the pieces together.

First, we take all files matching the initial wildcard, *.cr2. We're going to apply to them the filename generation code expression in parentheses after the wildcard. (I think you need EXTENDED_GLOB set to use that sort of parenthetical expression.)

The variable $REPLY is set to the filename the wildcard expression matched; so it will be set to each .cr2 filename, e.g. img001.cr2.

The expression ${REPLY%.cr2} removes the .cr2 extension. Then we tack on a .jpg: ${REPLY%.cr2}.jpg. So now we have img001.jpg.

[[ ! -e ${REPLY%.cr2}.jpg ]] checks for the existence of that jpg filename, just like in a shell script.

So that explains the quoted shell expression. The final, and hardest part, is how to use that quoted expression. That's in section 14.8.7 Glob Qualifiers. (estring) executes string as shell code, and the filename will be included in the list if and only if the code returns a zero status.

The colons -- after the e and before the closing parenthesis -- are just separator characters. Whatever character immediately follows the e will be taken as the separator, and anything from there to the next instance of that separator (the second colon, in this case) is taken as the string to execute. Colons seem to be the character to use by convention, but you could use anything. This is also the part of the expression responsible for setting $REPLY to the filename being tested.

So why the quotes inside the colons? They're because some of the substitutions being done would be evaluated too early without them: "Note that expansions must be quoted in the string to prevent them from being expanded before globbing is done. string is then executed as shell code."

Whew! Complicated, but awfully handy. I know I'll have lots of other uses for that.

One additional note: section 14.8.5, Approximate Matching, in that manual page caught my eye. zsh can do fuzzy matches! I can't think offhand what I need that for ... but I'm sure an idea will come to me.

Tags: zsh, shell, cmdline, imaging
[ 15:28 Oct 01, 2016    More linux/cmdline | permalink to this entry | ]

Image manipulation in Python

Someone asked me about determining whether an image was "portrait" or "landscape" mode from a script.

I've long had a script for automatically rescaling and rotating images, using ImageMagick under the hood and adjusting automatically for aspect ratio. But the scripts are kind of a mess -- I've been using them for over a decade, and they started life as a csh script back in the pnmscale days, gradually added ImageMagick and jpegtran support and eventually got translated to (not very good) Python.

I've had it in the back of my head that I should rewrite this stuff in cleaner Python using the ImageMagick bindings, rather than calling its commandline tools. So the question today spurred me to look into that. I found that ImageMagick isn't the way to go, but PIL would be a fine solution for most of what I need.

ImageMagick: undocumented and inconstant

Ubuntu has a python-pythonmagick package, which I installed. Unfortunately, it has no documentation, and there seems to be no web documentation either. If you search for it, you find a few other people asking where the documentation is.

Using things like help(PythonMagick) and help(PythonMagick.Image), you can ferret out a few details, like how to get an image's size:

import PythonMagick
filename = 'img001.jpg'
img = PythonMagick.Image(filename)
size = img.size()
print filename, "is", size.width(), "x", size.height()

Great. Now what if you want to rescale it to some other size? Web searching found examples of that, but it doesn't work, as illustrated here:

>>> img.scale('1024x768')
>>> img.size().height()
640

The built-in help was no help:

>>> help(img.scale)
Help on method scale:

scale(...) method of PythonMagick.Image instance
    scale( (Image)arg1, (Geometry)arg2) -> None :
    
        C++ signature :
            void scale(Magick::Image {lvalue},Magick::Geometry)

So what does it want for (Geometry)? Strings don't seem to work, 2-tuples don't work, and there's no Geometry object in PythonMagick. By this time I was tired of guesswork. Can the Python Imaging Library do better?

PIL -- the Python Imaging Library

PIL, happily, does have documentation. So it was easy to figure out how to get an image's size:

from PIL import Image
im = Image.open(filename)
w = im.size[0]
h = im.size[1]
print filename, "is", w, "x", h

It was equally easy to scale it to half its original size, then write it to a file:

newim = im.resize((w/2, h/2))
newim.save("small-" + filename)

Reading EXIF

Wow, that's great! How about EXIF -- can you read that? Yes, PIL has a module for that too:

import PIL.ExifTags

exif = im._getexif()
for tag, value in exif.items():
    decoded = PIL.ExifTags.TAGS.get(tag, tag)
    print decoded, '->', value

There are other ways to read exif -- pyexiv2 seems highly regarded. It has documentation, a tutorial, and apparently it can even write EXIF tags.

If neither PIL nor pyexiv2 meets your needs, here's a Stack Overflow thread on other Python EXIF solutions, and here's another discussion of Python EXIF. But since you probably already have PIL, it's certainly an easy way to get started.

What about the query that started all this: how to find out whether an image is portrait or landscape? Well, the most important thing is the image dimensions themselves -- whether img.size[0] > img.size[1]. But sometimes you want to know what the camera's orientation sensor thought. For that, you can use this code snippet:

for tag, value in exif.items():
    decoded = PIL.ExifTags.TAGS.get(tag, tag)
    if decoded == 'Orientation':
        print decoded, ":", value

Then compare the number you get to this Exif Orientation table. Normal landscape-mode photos will be 1.

Given all this, have I actually rewritten resizeall and rotateall using PIL? Why, no! I'll put it on my to-do list, honest. But since the scripts are actually working fine (just don't look at the code), I'll leave them be for now.

Tags: programming, python, imaging, imagemagick,
[ 15:33 Mar 16, 2012    More programming | permalink to this entry | ]

Increase Your ... Pancakes?

This coupon showed up on a Safeway receipt.

Everyone I've showed it to has the same reaction as I did: stacks of pancakes! Oh, wait, the headline says ... oh, I see, I guess those are supposed to be coins.

I'm not sure what the lesson is ... maybe that you should show your ad to a few other people before publishing it.

Or maybe the program is actually for cafe owners looking to increase their breakfast sales ...

Tags: humor, imaging, advertising
[ 12:26 Nov 08, 2011    More humor | permalink to this entry | ]

Getting a Wacom Tablet Working under Edgy

Another person popped into #gimp today trying to get a Wacom tablet working (this happens every few weeks). But this time it was someone using Ubuntu's new release, "Edgy Eft", and I just happened to have a shiny new Edgy install on my laptop (as well as a Wacom Graphire 2 gathering dust in the closet because I can never get it working under Linux), so I dug out the Graphire and did some experimenting.

And got it working! It sees pressure changes and everything. It actually wasn't that hard, but it did require some changes. Here's what I had to do:

1. Install wacom-tools and xinput
2. Edit /etc/X11/xorg.conf and comment out those ForceDevice lines that say "Tablet PC ONLY".
3. Reconcile the difference between udev creating /dev/input/wacom and xorg.conf using /dev/wacom: you can either change xorg.conf, change /etc/udev/rules.d/65-wacom.rules, or symlink /dev/input/wacom to /dev/wacom (that's what I did for testing, but it won't survive a reboot, so I'll have to pick a device name and make udev and X consistent).

A useful tool for testing is /usr/X11R6/bin/xinput list (part of the xinput package). That's a lot faster than going through GIMP's input device preference panel every time.

I added some comments to Ubuntu's bug 73160, where people had already described some of the errors but nobody had posted the steps to work around the problems.

While I was fiddling with GIMP on the laptop, I decided to install the packages I needed to build the latest CVS GIMP there. It needed a few small tweaks from the list I used on Dapper. I've updated the package list on my GIMP Building page accordingly.

Tags: linux, X11, imaging, gimp, ubuntu
[ 16:12 Dec 09, 2006    More linux | permalink to this entry | ]

Wacom Rides Again

Updating the blog again after taking time off for various reasons, including lack of time, homework, paying work, broken computer motherboard and other hardware problems, illness, a hand injury, and so on.

This afternoon, thanks to a very helpful Keir Mierle showing up on #gimp, I finally got all the pieces sorted and I now have a working tablet again. Hurrah!

I've put details of the setup that finally worked on my Linux and Wacom page.

Tags: linux, imaging, gimp
[ 19:08 May 08, 2005    More linux | permalink to this entry | ]

Scanner working under 2.6

My Epson 2400 Photo scanner is finally working again. It used to work beautifully under 2.4, but since the scanner.o module disappeared in 2.6 and sane started needing libusb, I haven't been able to get it to work. (sane-find-scanner would see the scanner, but scanimage -L would not, even as root so it wasn't a permissions problem.)

Working with someone on #sane tonight (who was also having problems with libusb and 2.6) I finally discovered the trick: I had an old version of /etc/sane.d/epson.conf which used a line:

usb /dev/usb/scanner0

but I was completely missing a new, important, section which includes a line that says simply

usb

preceeded by a couple of all important comment lines:

# For any system with libusb support
# (which is pretty much any
# recent Linux distribution) the
# following line is sufficient.

So I replaced the old libusbscanner script with the new one, commented out scsi, left /dev/usb/scanner0 commented out, and uncommented the standalone usb line. And voila, it worked!

<geeky_hotplug_details>
The old /etc/hotplug/usb/epson.scanner script (which I'd gotten from a SANE help page long ago) was no longer being called, since it's been replaced by libusbscanner. The main function of either of these scripts is to do a chown/chmod on the scanner device, so that non-root users can use it. An interesting variation on this is a bugzilla attachment which changes scanner ownership to the person who is currently logged in on the console. Might be worth doing on a multiuser system (not an issue for my own desktop).

I have a line for my scanner in /etc/hotplug/usb.usermap (and indeed that's the only line in that file):

libusbscanner 0x0003 0x04b8 0x011b 0x0000 0x0000 0x00 0x00 0x00 0x00 0x00 0x00 0x00000000

which is probably redundant with the 0x04b8 0x011b line in libsane.usermap (/etc/hotplug/usb.agent, which gets called whenever a USB hotplug event occurs, looks at usb.usermap and also usb/*.usermap)
</geeky_hotplug_details>

Tags: linux, imaging
[ 19:03 Nov 22, 2004    More linux | permalink to this entry | ]