Shallow Thoughts : : Mar

Dinosaur Eggs, Collared Doves and Wildflowers

Happy Easter! In keeping with the season, here's a dinosaur egg I spotted on a recent hike.

Okay, or maybe it's just a vaguely egg-shaped rock. But there's been a lot going on this spring now that the weather is turning.

First, we seem to have Eurasian collared doves nesting somewhere near our house. There's a dove up on the power pole, cooing, most of the day. I know I've heard lots of reports of collared doves around the south bay in past years, particularly down around Morgan Hill, but this is the first time I'd seen more than a glimpse of them here at home in San Jose. It's fun to see new species, though I hope these European interlopers don't push out the native mourning doves entirely.

In addition, the wildflowers have been great out on the trails, especially around the south end of Windy Hill OSP and Coal Mine Ridge. A hike up there last week revealed nearly every wildflower on my wildflower page that could be in flower now -- California poppy, wild cucumber (intriguingly also called manroot), giant trillium, hound's tongue, milkmaids, the most impressive profusion of Indian warrior I've seen, blue larkspur, miner's lettuce, Sierra suncup, vetch (it's pretty despite the unfortunate name), red maid, wild radish, wood sorrel, broom, and my favorite, shooting star.
Dave had to keep waiting for me while I argued with the camera over macro focus distances. So if you like wildflowers, get out there and take a look!

Tags: nature, birds, wildflowers
[ 17:22 Mar 31, 2013    More nature | permalink to this entry | ]

Pasting the Primary X Selection into Firefox

Sometimes I need to take a URL from some text app -- like a shell window, or an IRC comment -- and open it in Firefox.

If it's a standard http://, that's trivial: I highlight the URL with my house (often a doubleclick will do it), go to my Firefox window and middleclick somewhere in the content area, anywhere that's not a link, and Firefox goes to the URL.

That works because selecting anything, in X, copies the selection to the Primary selection buffer. The Primary selection is different from the Clipboard selection that's used with Windows and Mac style Ctrl-X/Ctrl-C/Ctrl-V copy and paste; it's faster and doesn't require switching between keyboard and mouse. Since your hand is already on the mouse (from selecting the text), you don't have to move to the keyboard to type Ctrl-C, then back to the mouse to go to the Firefox window, then back to the keyboard to type Ctrl-V.

But it fails in some cases. Like when someone says in IRC, "There's a great example of that at coolhacks.org/greatexample". You can highlight coolhacks.org/greatexample and middleclick in Firefox all you want, but Firefox doesn't recognize it as a URL and won't go there. Or if I want to highlight a couple of search terms and pass them into a Google search.

(Rant: middlemouse used to work for these cases, but it was disabled -- without even an option for getting it back -- due to a lot of whining in bugzilla by people coming from Windows backgrounds who didn't like middleclick paste because they found it unexpected, yet who weren't willing to turn it off with the middlemouse.contentLoadURL pref).

So in those cases, what I've been doing is:

• Highlight the URL or search terms
• Switch to the Firefox window
• Ctrl-L -- this focuses and highlights the URL bar without changing the X selection
• Backspace (or Ctrl-U, Ctrl-K, any key that clears the URLbar)
• Move the mouse to the URL bar (a small target) and middleclick
• Hit Enter

It works, but it's a lot more steps, and entails several switches between keyboard and mouse. Frustrating!

It would be a little less frustrating if I had a key binding in Firefox that said "Paste the current X primary selection." A web search shows that quite a few other people have been bothered by this problem -- for instance, here and here -- but without any solutions. Apparently in a lot of apps, Ctrl-Insert inserts the Primary X selection -- but in Firefox and a few others, it inserts the Clipboard instead, just like Ctrl-C.

I could write my own fix, by unzipping Firefox's omni.ja file and editing various .xul and .js files inside it. But if I were doing that, I could just as easily revert Firefox's original behavior of going to the link. Neither of these is difficult; the problem is that every time I Firefox updates (which is about twice a week these days), things break until I manually go in and unzip the jar and make my changes again. I used to do that, but I got tired of needing to do it so often. And I tried to do it via a Firefox extension, until Mozilla changed the Firefox extension API so that extensions couldn't modify key bindings any more.

Since Firefox changes so often, it's nicer to have a solution that's entirely outside of Firefox. And a comment in one of those discussion threads gave me an idea: make a key binding in my window manager that uses xset to copy the primary selection to the clipboard, then use my crikey program to insert a fake Ctrl-V that Firefox will see.

Here's a command to do that:

xsel -o -p | xsel -i -b; crikey -s 1 "^V"

xsel -o prints a current X selection, and -p specifies the Primary. xsel -i sets an X selection to whatever it gets on standard input (which in this case will be whatever was in the Primary selection), and -b tells it to set the Clipboard selection.

Then crikey -s 1 "^V" waits one second (I'll probably reduce this after more testing) and then generates an X event for a Ctrl-V.

I bound that command to Ctrl-Insert in my window manager, Openbox, like this:

<keybind key="C-Insert">
  <action name="Execute">
    <execute>/bin/sh -c 'xsel -o -p | xsel -i -b; crikey -s 1 "^V"'</execute>
  </action>
</keybind>

Openbox didn't seem happy with the pipe, so I wrapped the whole thing in a sh -c.

Now, whenever I type Ctrl-Insert, whatever program I'm in will do a Ctrl-V but insert the Primary selection rather than the Clipboard. It should work in other recalcitrant programs, like LibreOffice, as well. In Firefox, now, I just have to type Ctrl-L Ctrl-Insert Return.

Of course, it's easy enough to make a binding specific to Firefox that does the Ctrl-L and the Return automatically. I've bound that to Alt-Insert, and its execute line looks like this:

    <execute>/bin/sh -c 'xsel -o -p | xsel -i -b; crikey -s 1 "^L^V\\n"'</execute>

Fun with Linux! Now the only hard part will be remembering to use the bindings instead of doing things the hard way.

Tags: linux, firefox, X11, cmdline, crikey
[ 20:35 Mar 26, 2013    More linux/cmdline | permalink to this entry | ]

Letters not used in Python keywords

One of the closing lightning talks at PyCon this year concerned the answers to a list of Python programming puzzles given at some other point during the conference. I hadn't seen the questions (I'm still not sure where they are), but some of the problems looked fun.

One of them was: "What are the letters not used in Python keywords?" I hadn't known about Python's keyword module, which could come in handy some day:

>>> import keyword
>>> keyword.kwlist
['and', 'as', 'assert', 'break', 'class', 'continue', 'def', 'del', 'elif', 'else', 'except', 'exec', 'finally', 'for', 'from', 'global', 'if', 'import', 'in', 'is', 'lambda', 'not', 'or', 'pass', 'print', 'raise', 'return', 'try', 'while', 'with', 'yield']

So, given the list of keywords, what's the best way to find the list of unique letters?

Any time you want a list of unique anything, you want a set. For instance,

>>> set([1, 2, 3, 2, 2, 4, 5, 1, 5])
set([1, 2, 3, 4, 5])

But first you need a list of letters so can make a set out of it.

Split the list of words into a list of letters

My first idea was to use list comprehensions. You can split a single word into letters like this:

>>> [ x for x in 'hello' ]
['h', 'e', 'l', 'l', 'o']

It took a bit of fiddling to get the right syntax to apply that to every word in the list:

>>> [[c for c in w] for w in keyword.kwlist]
[['a', 'n', 'd'], ['a', 's'], ['a', 's', 's', 'e', 'r', 't'], ... ]

Update: Dave Foster points out that [list(w) for w in keyword.kwlist] is another way, simpler and cleaner way than the double list comprehension.

That's a list of lists, so it needs to be flattened into a single list of letters before we can turn it into a set.

Flatten the list of lists

There are lots of ways to flatten a list of lists. Here are four of them:

[item for sublist in [[c for c in w] for w in keyword.kwlist] for item in sublist]

reduce(lambda x,y: x+y, [[c for c in w] for w in keyword.kwlist])

import itertools
list(itertools.chain.from_iterable([[c for c in w] for w in keyword.kwlist]))

sum([[c for c in w] for w in keyword.kwlist], [])

That last one, using sum(), makes use of the fact that Python uses + for list concatenation -- in other words, that [1, 2, 3] + [4, 5, 6] is [1, 2, 3, 4, 5, 6]. But the first method (item for sublist in) is faster: see Making a flat list out of list of lists in Python on StackOverflow. And another StackOverflow thread has a nice script for plotting speed vs. list size of various flatteners.

A simpler way of making the set

But it turns out none of this list comprehension stuff is needed anyway. set('word') splits words into letters already:

>>> set('bubble')
set(['e', 'b', 'u', 'l'])

Ignore the order -- elements of a set often end up displaying in some strange order. The important thing is that it has all the letters and no repeats.

Now we have an easy way of making a set containing the letters in one word. But how do we apply that to a list of words?

Again I initially tried using list comprehensions, then realized there's an easier way. Given a list of strings, it's trivial to join them into a single string using ''.join(). And that gives us our set of letters within keywords:

>>> set(''.join(keyword.kwlist))
set(['a', 'c', 'b', 'e', 'd', 'g', 'f', 'i', 'h', 'k', 'm', 'l', 'o', 'n', 'p', 's', 'r', 'u', 't', 'w', 'y', 'x'])

What letters are not in the set?

Almost done! But the original problem was to find the letters not in keywords. We can do that by subtracting this set from the set of all letters from a to z. How do we get that? The string module will give us a list:

>>> string.lowercase
'abcdefghijklmnopqrstuvwxyz'

You could also use a list comprehension and ord and chr (alas, range won't give you a range of letters directly):

>>> [chr(i) for i in range(ord('a'), ord('z')+1)]
['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']

It's a bit longer, but doesn't require an import.

Now that you have your a-z set, just subtract the two sets:

>>> set(string.lowercase[:]) - set(''.join(keyword.kwlist))
set(['q', 'j', 'z', 'v'])

So the only letters not used in Python keywords are q, j, z and v.

Just a useless little ditty, really ... but I thought it was a fun exercise, so maybe you will too.

Tags: programming, python
[ 13:36 Mar 19, 2013    More programming | permalink to this entry | ]

SimpleCV on Raspberry Pi

I'm at PyCon, and I spent a lot of the afternoon in the Raspberry Pi lab.

Raspberry Pis are big at PyCon this year -- because everybody at the conference got a free RPi! To encourage everyone to play, they have a lab set up, well equipped with monitors, keyboards, power and ethernet cables, plus a collection of breadboards, wires, LEDs, switches and sensors.

I'm primarily interested in the RPi as a robotics controller, one powerful enough to run a camera and do some minimal image processing (which an Arduino can't do). And on Thursday, I attended a PyCon tutorial on the Python image processing library SimpleCV. It's a wrapper for OpenCV that makes it easy to access parts of images, do basic transforms like greyscale, monochrome, blur, flip and rotate, do edge and line detection, and even detect faces and other objects. Sounded like just the ticket, if I could get it to work on a Raspberry Pi.

SimpleCV can be a bit tricky to install on Mac and Windows, apparently. But the README on the SimpleCV git repository gives an easy 2-line install for Ubuntu. It doesn't run on Debian Squeeze (though it installs), because apparently it depends on a recent version of pygame and Squeeze's is too old; but Ubuntu Pangolin handled it just fine.

The question was, would it work on Raspbian Wheezy? Seemed like a perfect project to try out in the PyCon RPi lab. Once my RPi was set up and I'd run an apt-get update, I used used netsurf (the most modern of the lightweight browsers available on the RPi) to browse to the SimpleCV installation instructions. The first line,

sudo apt-get install ipython python-opencv python-scipy python-numpy python-pygame python-setuptools python-pip

was no problem. All those packages are available in the Raspbian repositories.

But the second line,

sudo pip install https://github.com/ingenuitas/SimpleCV/zipball/master

failed miserably. Seems that pip likes to put its large downloaded files in /tmp; and on Raspbian, running off an SD card, /tmp quite reasonably is a tmpfs, running in RAM. But that means it's quite small, and programs that expect to be able to use it to store large files are doomed to failure.

I tried a couple of simple Linux patches, with no success. You can't rename /tmp to replace it with a symlink to a directory on the SD card, because /tmp is always in use. And pip makes a new temp directory name each time it's run, so you can't just symlink the pip location to a place on the SD card.

I thought about rebooting after editing the tmpfs out of /etc/fstab, but it turns out it's not set up there, and it wasn't obvious how to disable the tmpfs. Searching later from home, the size is set in /etc/default/tmpfs. As for disabling the tmpfs and using the SD card instead, it's not clear. There's a block of code in /etc/init.d/mountkernfs.sh that makes that decision; it looks like symlinking /tmp to somewhere else might do it, or else commenting out the code that sets RAMTMP="yes". But I haven't tested that.

Instead of rebooting, I downloaded the file to the SD card:

wget https://github.com/ingenuitas/SimpleCV/master

But it turned out it's not so easy to pip install from a local file. After much fussing around I came up with this, which worked:

pip install http:///home/pi/master --download-cache /home/pi/tmp

That worked, and the resulting SimpleCV install worked nicely! I typed some simple tests into the simplecv shell, playing around with their built-in test image "lenna":

img = Image('lenna')
img.show()
img.binarize().show()
img.toGray().show()
img.edges().show()
img.invert().show()

And, for something a little harder, some face feature detection: let's find her eyes and outline them in yellow.

img.listHaarFeatures()
img.findHaarFeatures('eye.xml').draw(color=Color.YELLOW)

SimpleCV is lots of fun! And the edge detection was quite fast on the RPi -- this may well be usable by a robot, once I get the motors going.

Tags: raspberry pi, python, programming, hardware, linux, maker
[ 21:43 Mar 16, 2013    More linux/install | permalink to this entry | ]

The Europa Eclipse -- results

I wrote last week about an upcoming eclipse of Europa by Jupiter's shadow. One of the interesting things I'd found was how much the predicted times of Europa's appearance from behind Jupiter, and subsequent disappearance into Jupiter's shadow, varied depending on which program you were using. I had just recently managed to get my own Javascript Jupiter page showing eclipse events, and its times didn't agree with any of the other programs either. So I was burning with curiosity to know who was right.

The predicted times were:

	Europa appears	Europa disappears
XEphem	7:43	7:59
S&T Jupiter's Moons	7:40	7:44
my Javascript Jupiter	7:45	7:52
Stellarium	6:49	7:05

I was out of town on March 10. I brought along a travel scope, an Orion 80mm f/6 Orion Express. Not the perfect planetary scope, but certainly enough to see Europa. (The Galilean moons are even visible in binoculars, as long as you mount the binoculars on a tripod or otherwise hold them steady.)

I synchronized my watch and had the telescope set up by 7:35. Sure enough, there was no Europa there. But at 7:38 on the dot, I saw the first hint of Europa peeking out. No question about it. I watched, and timed, and by 7:41 the whole disk of Europa was visible and I could start to think I could see blackness between it and Jupiter. I'd been to a school star party a few days earlier and hadn't cleaned my eyepieces afterward -- oops! -- so the view was a little foggy and it was hard to tell for sure exactly when Europa's disk cleared Jupiter.

In fact, no matter which eyepiece I used, the fogginess seemed to get worse and worse. I had a hard time seeing Europa at all. Finally I realized that I was looking through a tree branch, and moved the scope. But by the time I got it moved again, Europa had gotten even harder to see. That was when I realized that it had been going into eclipse practically the whole time I was watching it. It was already significantly dimmed by 7:43, very dim indeed by 7:48 and gone -- in the 80mm -- by 7:49:20, though I suspect it still would have been visible in a larger scope with clean eyepieces.

So that's why the times in different programs varied so much! Galilean moons aren't point sources: you can't predict a single time for a moon disappearance, appearance or eclipse. Do you want to predict the beginning of the event, the end of the event or the time at the moon's center point? And that goes double for eclipses, where the moon is gradually sliding into the shadow of Jupiter's atmosphere. I found that it took over seven minutes the moon to go from full brightness to fully eclipsed. So what part of that do you predict?

All in all, a very interesting observing session. I'm looking forward to observing more of these eclipses, doing more timings, and tuning my program to give better predictions. (I notice my program was significantly late on both the appearance and the eclipse. I'll work on that. Better to err on the early side, and not miss anything!)

While I was adding eclipses to my Jupiter program, I also added longer-range predictions, so it would be easier to find out when these events will happen. Once that was implemented, I looked for upcoming Whac-a-Moon events. I found one on Mar 26, when Ganymede appears at 7:29pm PDT (add 7 hours for GMT). Europa and its shadow are transiting Jupiter's disk, too, so there's plenty to look at. Ganymede then enters eclipse at 9:40pm PDT. A long time between the events, I know, but it's easy enough to leave a scope set up in the backyard and go out to check it now and then.

These times are from my Javascript Jupiter program and may be a few minutes late. Always be ready at least five minutes early in case the predictions are off, no matter which program you use. Don't say I didn't warn you.

I found no events in April visible at night in California (for other time zones, you can generate predictions on my Javascript Jupiter page). But May 8 has a decent one: Ganymede appears at 9:44pm PDT, then disappears into eclipse at 9:46. Based on what I saw tonight with Europa, that means the moon should start to fade almost immediately after it has emerged from behind Jupiter, maybe even before it has fully emerged. Ganymede's larger size may also mean the fade-out will take longer. Stay tuned. Jupiter will be very low by then, only 7 degrees above the horizon.

Not many events to observe -- this is a bit rarer than I'd thought. Of course, there are lots of moons disappearing into eclipse and appearing from out of it every night, so watching that long gradual appearance or disappearance isn't difficult; the only rare part is when they appear briefly between Jupiter and Jupiter's shadow. That is relatively rare, and I'm glad I had a chance to catch it.

Tags: astronomy, science
[ 19:55 Mar 12, 2013    More science/astro | permalink to this entry | ]

Whac-a-Moon: Watch Europa appear and disappear this Sunday

This is an edited and updated version of my "Shallow Sky" column this month in the SJAA Ephemeris newsletter.

A few months ago, I got email from a Jupiter observer calling my attention to an interesting phenomenon of Jupiter's moons that I hadn't seen before. The person who mailed me described himself as a novice, and wasn't quite sure what he had seen, but he knew it was odd. After some further discussion we pinned it down.

He was observing Jupiter at 11/11/12 at 00.25 UT (which would have been mid-afternoon here in San Jose). Three of the moons were visible, with only Ganymede missing. Then Ganymede appeared: near Jupiter's limb, but not right on it. As he watched over the next few minutes, Ganymede seemed to be moving backward -- in toward Jupiter rather than away from it. Eventually it disappeared behind the planet.

It turned out that what he was seeing was the end of an eclipse. Jupiter was still a few months away from opposition, so the shadow thrown by the planet streamed off to one side as seen from our inner-planet vantage point on Earth. At 0:26 UT on that evening, long before he started observing, Ganymede, still far away from Jupiter's limb, had entered Jupiter's shadow and disappeared into eclipse. It took over two hours for Ganymede to cross Jupiter's shadow; but at 2:36, when it left the shadow, it hadn't yet disappeared behind the planet. So it became visible again. It wasn't until 2:50 that Ganymede finally disappeared behind Jupiter.

So it was an interesting effect -- bright Ganymede appearing out of nowhere, moving in toward Jupiter then disappearing again fourteen minutes later. It was something I'd never seen, or thought to look for. It's sort of like playing Whac-a-mole -- the moon appears only briefly, so you've got to hit it with your telescope at just the right time if you want to catch it before it disappears again.

A lot of programs don't show this eclipse effect -- including, I'm sad to say, my own Javascript Jupiter's moons web page. (I have since remedied that.) The open source program Stellarium shows the effect; on the web, Sky and Telescope's Jupiter's Moons page shows it, and even prints out a table of times of various moon events, including eclipses.

These eclipse events aren't all that uncommon -- but only when the sun angle is just right. Searching in late February and early March this year, I found several events for Ganymede and Europa (though, sadly, many of them were during our daytime). By mid-March, the angles have changed so that Europa doesn't leave Jupiter's shadow until after it's disappeared behind the planet's limb; but Ganymede is farther out, so we can see Ganymede appearances all the way through March and for months after.

The most interesting view, it seems to me, is right on the boundary when the moon only appears for a short time before disappearing again. Like the Europa eclipse that's happening this Sunday night, March 10.

Reporting on that one got a little tricky -- because that's the day we switch to Daylight Savings time. I have to confess that I got a little twisted up trying to compare results between programs that use UTC and programs that use local time -- especially when the time zone converter I was using to check my math told me "That time doesn't exist!" Darnit, if we'd all just use UTC all the time, astronomy calculations would be a lot easier! (Not to mention dropping the silly Daylight Savings Time fiasco, but that's another rant.)

Before I go into the details, I want to point out that Jupiter's moons are visible even in binoculars. So even if you don't have a telescope, grab binoculars and set them up in as steady a way as you can -- if you don't have a tripod adaptor, try bracing them on the top of a gate or box.

On Sunday night, March 10, at some time around 7:40 pm PDT, Europa peeks out from behind Jupiter's northeast limb. (All times are given in PDT; add 7 hours for GMT.) The sky will still be bright here in California -- the sun sets at 7:12 that night -- but Jupiter will be 66 degrees up and well away from the sun, so it shouldn't give you too much trouble. Once Europa pops out, keep a close eye on it -- because if Sky & Tel's calculations are right, it will disappear again just four minutes later, at 7:44, into eclipse in Jupiter's shadow. It will remain invisible for almost three hours, finally reappearing out of nowhere, well off Jupiter's limb, at around 10:24 pm.

Here's a link to my Javascript Jupiter just before Europa reappears.

I want to stress that those times are approximate. In fact, I tried simulating the event in several different programs, and got wildly varying times:

	Io disappears	Europa appears	Europa disappears	Europa reappears	Io appears
XEphem	7:15	7:43	7:59	10:06	10:43
S&T Jupiter's Moons	7:16	7:40	7:44	10:24	10:48
Javascript Jupiter	7:17	7:45	7:52	10:15	10:41
Stellarium	6:21	6:49	7:05	9:32	10:01

You'll note Stellarium seems to have a time zone problem ... maybe because I ran the prediction while we were still in standard time, not daylight savings time.

I'm looking forward to timing the events to see which program is most accurate. I'm betting on XEphem. Once I know the real times, maybe I can adjust my Javascript Jupiter's code to be more accurate. If anyone else times the event, please send me your times, in case something goes wrong here!

Anyway, the spread of times makes it clear that when observing this sort of phenomenon, you should always set up the telescope ten or fifteen minutes early, just in case. And ten extra minutes spent observing Jupiter -- even without moons -- is certainly never time wasted! Just keep an eye out for Europa to appear -- and be ready to whack that moon before it disappears again.

Tags: astronomy, programming, javascript
[ 11:30 Mar 09, 2013    More science/astro | permalink to this entry | ]

How to enable/disable laptop wireless hardware

My Lenovo laptop has a nifty button, Fn-F5, to toggle wi-fi and bluetooth on and off. Works fine, and the indicator lights (of which the Lenovo has many -- it's quite nice that way) obligingly go off or on.

But when I suspend and resume, the settings aren't remembered. The machine always comes up with wireless active, even if it wasn't before suspending.

Since wireless can be a drain on battery life, as well as a potential security issue, I don't want it on when I'm not actually using it. So I wanted a way to turn it off programmatically.

The answer, it turns out, is rfkill.

$ rfkill list
0: tpacpi_bluetooth_sw: Bluetooth
        Soft blocked: yes
        Hard blocked: no
0: phy0: Wireless LAN
        Soft blocked: yes
        Hard blocked: no

tells you what hardware is currently enabled or disabled.

To toggle something off,

$ rfkill block bluetooth
$ rfkill block wifi

Type rfkill -h for more details on arguments you can use.

Fn-F5 still works to enable or disable them together. I think this is being controlled by /etc/acpi/ibm-wireless.sh, though I can't find where it's tied to Fn-F5.

You can make it automatic by creating /etc/pm/sleep.d/. (That's on Ubuntu; of course, the exact file location may vary with distro and version.) To disable wireless on resume, do this:

#! /bin/sh

case "$1" in
  resume)
    rfkill block bluetooth
    rfkill block wifi
    ;;
esac
exit $?

Of course, you can also tie that into other things, like your current network scheme, or what wireless networks are visible (which you can get with iwlist wlan0 scan).

Tags: linux, ubuntu, laptop, tip
[ 19:46 Mar 04, 2013    More linux/laptop | permalink to this entry | ]