Shallow Thoughts : : Jun

A Raspberry Pi Night Vision Camera

When I built my http://shallowsky.com/blog/hardware/raspberry-pi-motion-camera.html (and part 2), I always had the NoIR camera in the back of my mind. The NoIR is a version of the Pi camera module with the infra-red blocking filter removed, so you can shoot IR photos at night without disturbing nocturnal wildlife (or alerting nocturnal burglars, if that's your target).

After I got the daylight version of the camera working, I ordered a NoIR camera module and plugged it in to my RPi. I snapped some daylight photos with raspstill and verified that it was connected and working; then I waited for nightfall.

In the dark, I set up the camera and put my cup of hot chocolate in front of it. Nothing. I hadn't realized that although CCD cameras are sensitive in the near IR, the wavelengths only slightly longer than visible light, they aren't sensitive anywhere near the IR wavelengths that hot objects emit. For that, you need a special thermal camera. For a near-IR CCD camera like the Pi NoIR, you need an IR light source.

Knowing nothing about IR light sources, I did a search and came up with something called a "Infrared IR 12 Led Illuminator Board Plate for CCTV Security CCD Camera" for about $5. It seemed similar to the light sources used on a few pages I'd found for home-made night vision cameras, so I ordered it. Then I waited, because I stupidly didn't notice until a week and a half later that it was coming from China and wouldn't arrive for three weeks. Always check the shipping time when ordering hardware!

When it finally arrived, it had a tiny 2-pin connector that I couldn't match locally. In the end I bought a package of female-female SchmartBoard jumpers at Radio Shack which were small enough to make decent contact on the light's tiny-gauge power and ground pins. I soldered up a connector that would let me use a a universal power supply, taking a guess that it wanted 12 volts (most of the cheap LED rings for CCD cameras seem to be 12V, though this one came with no documentation at all). I was ready to test.

Testing the IR light

One problem with buying a cheap IR light with no documentation: how do you tell if your power supply is working? Since the light is completely invisible.

The only way to find out was to check on the Pi. I didn't want to have to run back and forth between the dark room where the camera was set up and the desktop where I was viewing raspistill images. So I started a video stream on the RPi:

$ raspivid -o - -t 9999999 -w 800 -h 600 | cvlc -vvv stream:///dev/stdin --sout '#rtp{sdp=rtsp://:8554/}' :demux=h264

Then, on the desktop: I ran vlc, and opened the network stream:
rtsp://pi:8554/
(I have a "pi" entry in /etc/hosts, but using an IP address also works).

Now I could fiddle with hardware in the dark room while looking through the doorway at the video output on my monitor.

It took some fiddling to get a good connection on that tiny connector ... but eventually I got a black-and-white view of my darkened room, just as I'd expect under IR illumination. I poked some holes in the milk carton and used twist-ties to seccure the light source next to the NoIR camera.

Lights, camera, action

Next problem: mute all the blinkenlights, so my camera wouldn't look like a christmas tree and scare off the nocturnal critters.

The Pi itself has a relatively dim red run light, and it's inside the milk carton so I wasn't too worried about it. But the Pi camera has quite a bright red light that goes on whenever the camera is being used. Even through the thick milk carton bottom, it was glaring and obvious. Fortunately, you can disable the Pi camera light: edit /boot/config.txt and add this line

disable_camera_led=1

My USB wi-fi dongle has a blue light that flickers as it gets traffic. Not super bright, but attention-grabbing. I addressed that issue with a triple thickness of duct tape.

The IR LEDs -- remember those invisible, impossible-to-test LEDs? Well, it turns out that in darkness, they emit a faint but still easily visible glow. Obviously there's nothing I can do about that -- I can't cover the camera's only light source! But it's quite dim, so with any luck it's not spooking away too many animals.

Results, and problems

For most of my daytime testing I'd used a threshold of 30 -- meaning a pixel was considered to have changed if its value differed by more than 30 from the previous photo. That didn't work at all in IR: changes are much more subtle since we're seeing essentially a black-and-white image, and I had to divide by three and use a sensitivity of 10 or 11 if I wanted the camera to trigger at all.

With that change, I did capture some nocturnal visitors, and some early morning ones too. Note the funny colors on the daylight shots: that's why cameras generally have IR-blocking filters if they're not specifically intended for night shots.

Here are more photos, and larger versions of those: Images from my night-vision camera tests.

But I'm not happy with the setup. For one thing, it has far too many false positives. Maybe one out of ten or fifteen images actually has an animal in it; the rest just triggered because the wind made the leaves blow, or because a shadow moved or the color of the light changed. A simple count of differing pixels is clearly not enough for this task.

Of course, the software could be smarter about things: it could try to identify large blobs that had changed, rather than small changes (blowing leaves) all over the image. I already know SimpleCV runs fine on the Raspberry Pi, and I could try using it to do object detection.

But there's another problem with detection purely through camera images: the Pi is incredibly slow to capture an image. It takes around 20 seconds per cycle; some of that is waiting for the network but I think most of it is the Pi talking to the camera. With quick-moving animals, the animal may well be gone by the time the system has noticed a change. I've caught several images of animal tails disappearing out of the frame, including a quail who visited yesterday morning. Adding smarts like SimpleCV will only make that problem worse.

So I'm going to try another solution: hooking up an infra-red motion detector. I'm already working on setting up tests for that, and should have a report soon. Meanwhile, pure image-based motion detection has been an interesting experiment.

Mirror a website using lftp

I'm helping an organization with some website work. But I'm not the only one working on the website, and there's no version control. I wanted an easy way to make sure all my files were up-to-date before I start to work on one ... a way to mirror the website, or at least specific directories, to my local disk.

Normally I use rsync -av over ssh to mirror directories, but this website is on a server that only offers ftp access. I've been using ncftp to copy files up one by one, but although ncftp's manual says it has a mirror mode and I found a few web references to that, I couldn't find anything telling me how to activate it.

Making matters worse, there are some large files that I don't need to mirror. The first time I tried to use get * in ncftp to get one directory, it spent 15 minutes trying to download a huge powerpoint file, then stalled and lost the connection. There are some big .doc and .docx files, too. And ncftp doesn't seem to have a way to exclude specific files.

Enter lftp. It has a mirror mode (with documentation, even!) which includes a -X to exclude files matching specified patterns.

lftp includes a -e to pass commands -- like "mirror" -- to it on the command line. But the documentation doesn't say whether you can use more than one command at a time. So it seemed safer to start up an lftp session and pass a series of commands to it.

And that works nicely. Just set up the list of directories you want to mirror, and you can write a nice shell function you can put in your. .zshrc or .bashrc:

sitemirror() {
commands=""
for dir in thisdir thatdir theotherdir
do
  commands="$commands
mirror --only-newer -vvv -X '*.ppt' -X '*.doc*' -X '*.pdf' htdocs/$dir $HOME/web/webmirror/$dir"
done

echo Commands to be run:
echo $commands
echo

lftp <<EOF
open -u 'user,password' ftp.example.com
$commands
bye
EOF
}

Super easy -- all I do is type sitemirror and wait a little. Now I don't have any excuse for not being up to date.

Fuzzy house finch chicks

The wind was strong a couple of days ago, but that didn't deter the local house finch family. With three hungry young mouths to feed, and considering how long it takes to crack sunflower seeds, poor dad -- two days after Father's Day -- was working overtime trying to keep them all fed. They emptied by sunflower seed feeder in no time and I had to refill it that evening.

The chicks had amusing fluffy "eyebrow" feathers sticking up over their heads, and one of them had an interesting habit of cocking its tail up like a wren, something I've never seen house finches do before.

More photos: House finch chicks.

Vim: Set wrapping and indentation according to file type

Although I use emacs for most of my coding, I use vim quite a lot too, for quick edits, mail messages, and anything I need to edit when logged onto a remote server. In particular, that means editing my procmail spam filter files on the mail server.

The spam rules are mostly lists of regular expression patterns, and they can include long lines, such as:
gift ?card .*(Visa|Walgreen|Applebee|Costco|Starbucks|Whitestrips|free|Wal.?mart|Arby)

My default vim settings for editing text, including line wrap, don't work if get a flood of messages offering McDonald's gift cards and decide I need to add a "|McDonald" on the end of that long line.

Of course, I can type ":set tw=0" to turn off wrapping, but who wants to have to do that every time? Surely vim has a way to adjust settings based on file type or location, like emacs has.

It didn't take long to find an example of Project specific settings on the vim wiki. Thank goodness for the example -- I definitely wouldn't have figured that syntax out just from reading manuals. From there, it was easy to make a few modifications and set textwidth=0 if I'm opening a file in my procmail directory:

" Set wrapping/textwidth according to file location and type
function! SetupEnvironment()
  let l:path = expand('%:p')
  if l:path =~ '/home/akkana/Procmail'
    " When editing spam filters, disable wrapping:
    setlocal textwidth=0
endfunction
autocmd! BufReadPost,BufNewFile * call SetupEnvironment()

Nice! But then I remembered other cases where I want to turn off wrapping. For instance, editing source code in cases where emacs doesn't work so well -- like remote logins over slow connections, or machines where emacs isn't even installed, or when I need to do a lot of global substitutes or repetitive operations. So I'd like to be able to turn off wrapping for source code.

I couldn't find any way to just say "all source code file types" in vim. But I can list the ones I use most often. While I was at it, I threw in a special wrap setting for mail files:

" Set wrapping/textwidth according to file location and type
function! SetupEnvironment()
  let l:path = expand('%:p')
  if l:path =~ '/home/akkana/Procmail'
    " When editing spam filters, disable wrapping:
    setlocal textwidth=0
  elseif (&ft == 'python' || &ft == 'c' || &ft == 'html' || &ft == 'php')
    setlocal textwidth=0
  elseif (&ft == 'mail')
    " Slightly narrower width for mail (and override mutt's override):
    setlocal textwidth=68
  else
    " default textwidth slightly narrower than the default
    setlocal textwidth=70
  endif
endfunction
autocmd! BufReadPost,BufNewFile * call SetupEnvironment()

As long as we're looking at language-specific settings, what about doing language-specific indentation like emacs does? I've always suspected vim must have a way to do that, but it doesn't enable it automatically like emacs does. You need to set three variables, assuming you prefer to use spaces rather than tabs:

" Indent specifically for the current filetype
filetype indent on
" Set indent level to 4, using spaces, not tabs
set expandtab shiftwidth=4

Then you can also use useful commands like << and >> for in- and out-denting blocks of code, or ==, for indenting to the right level. It turns out vim's language indenting isn't all that smart, at least for Python, and gets the wrong answer a lot of them time. You can't rely on it as a syntax checker the way you can with emacs. But it's a lot better than no language-specific indentation.

I will be a much happier vimmer now!

Comcast actually installed a cable! Or say they did.

The doorbell rings at 10:40. It's a Comcast contractor.

They want to dig across the driveway. They say the first installer didn't know anything, he was wrong about not being able to use the box that's already on this side of the road. They say they can run a cable from the other side of the road through an existing conduit to the box by the neighbor's driveway, then dig a trench across the driveway to run the cable to the old location next to the garage.

They don't need to dig across the road since there's an existing conduit; they don't even need to park in the road. So no need for a permit.

We warn them we're planning to have driveway work done, so the driveway is going to be dug up at some point, and they need to put it as deep as possible. We even admit that we've signed a contract with CenturyLink for DSL. No problem, they say, they're being paid by Comcast to run this cable, so they'll go ahead and do it.

We shrug and say fine, go for it. We figure we'll mark the trench across the driveway afterward, and when we finally have the driveway graded, we'll make sure the graders know about the buried cable. They do the job, which takes less than an hour.

If they're right that this setup works, that means, of course, that this could have been done back in February or any time since then. There was no need to wait for a permit, let alone a need to wait for someone to get around to applying for a permit.

So now, almost exactly 4 months after the first installer came out, we may have a working cable installed. No way to know for sure, since we've been happily using DSL for over a month. But perhaps we'll find out some day.

The back story, in case you missed it: Getting cable at the house: a Comcast Odyssey.

Santa Fe Highway Art, and the Digestive Deer

Santa Fe is a city that prides itself on its art. There are art galleries everywhere, glossy magazines scattered around town pointing visitors to the various art galleries and museums.

Why, then, is Santa Fe county public art so bad?

Like this mural near the courthouse. It has it all! It combines motifs of crucifixions, Indian dancing, Hermaphroditism, eagles, jaguars, astronomy, menorahs (or are they power pylons?), an angel, armed and armored, attempting to stab an unarmed angel, and a peace dove smashing its head into a baseball. All in one little mural!

But it's really the highway art north of Santa Fe that I wanted to talk about today.

Some of it isn't totally awful. The roadrunner and the horned toad are actually kind of cute, and the rattlesnake isn't too bad.

On the other hand, the rooster and turkey are pretty bad ...

and the rabbit is beyond belief.

As you get farther away from Santa Fe, you get whole overpasses decorated with names and symbols:

I think of this one near Pojoaque as the "happy dancing shuriken" -- it looks more like a Japanese throwing star, a shuriken, than anything else, though no doubt it has some deeper meaning to the Pojoaque pueblo people.

But my favorite is the overpass near Cuyamungue.

See those deer in the upper right and left corners?

Here it is in close-up. We've taken to calling it "the digestive deer".

I can't figure out what this is supposed to tell us about a deer's alimentary tract. Food goes in ... and then we don't want to dwell on what happens after that? Is there a lot of foliage near Cuyamungue that's particularly enticing to deer? A "land of plenty", at least for deer? Do they then go somewhere else to relieve themselves?

I don't know what it means. But as we drive past the Cuyamungue digestive deer on the way to Santa Fe ... it's hard to take the city's airs of being a great center of art and culture entirely seriously.

Cicada Rice Krispies

Late last week we started hearing a loud buzz in the evenings. Cicadas? We'd heard a noise like that last year, when we visited Prescott during cicada season while house-hunting, but we didn't know they had them here in New Mexico. The second evening, we saw one in the gravel off the front walk -- but we were meeting someone to carpool to a talk, so I didn't have time to race inside and get a camera.

A few days later they started singing both morning and evening. But yesterday there was an even stranger phenomenon.

"It sounds like Rice Krispies out in the yard. Snap, crackle, pop," said Dave. And he was right -- a constant, low-level crackling sound was coming from just about all the junipers.

Was that cicadas too? It was much quieter than their loud buzzing -- quiet enough to be a bit eerie, really. You had to stop what you were doing and really listen to notice it.

It was pretty clearly an animal of some kind: when we moved close to a tree, the crackling (and snapping and popping) coming from that tree would usually stop. If we moved very quietly, though, we could get close to a tree without the noise entirely stopping. It didn't do us much good, though: there was no motion at all that we could see, no obvious insects or anything else active.

Tonight the crackling was even louder when I went to take the recycling out. I stopped by a juniper where it was particularly noticeable, and must have disturbed one, because it buzzed its wings and moved enough that I actually saw where it was. It was black, maybe an inch long, with narrow orange stripes. I raced inside for my camera, but of course the bug was gone by the time I got back out.

So I went hunting. It almost seemed like the crackling was the cicadas sort of "tuning up", like an orchestra before the performance. They would snap and crackle and pop for a while, and then one of them would go snap snap snap-snap-snap-snapsnapsnapsnap and then break into its loud buzz -- but only for a few seconds, then it would go back to snapping again. Then another would speed up and break into a buzz for a bit, and so it went.

One juniper had a particularly active set of crackles and pops coming from it. I circled it and stared until finally I found the cicadas. Two of them, apparently mating, and a third about a foot away ... perhaps the rejected suitor?

Near that particular juniper was a section of ground completely riddled with holes. I don't remember those holes being there a few weeks ago. The place where the cicadas emerged?

So our Rice Krispies mystery was solved. And by the way, I don't recommend googling for combinations like cicada rice krispies ... unless you want to catch and eat cicadas.

Meanwhile, just a few feet away from the cicada action, a cactus had sprung into bloom. Here, have a gratuitous pretty flower. It has nothing whatever to do with cicadas.