Shallow Thoughts : : Aug

Read Excel XLS spreadsheets with Python

Someone mailed out information to a club I'm in as an .XLS file. Another Excel spreadsheet. Sigh.

I do know one way to read them. Fire up OpenOffice, listen to my CPU fan spin as I wait forever for the app to start up, open the xls file, then click in one cell after another as I deal with the fact that spreadsheet programs only show you a tiny part of the text in each cell. I'm not against spreadsheets per se -- they're great for calculating tables of interconnected numbers -- but they're a terrible way to read tabular data.

Over the years, lots of open-source programs like word2x and catdoc have sprung up to read the text in MS Word .doc files. Surely by now there must be something like that for XLS files?

Well, I didn't find any ready-made programs, but I found something better: Python's xlrd module, as well as a nice clear example at ScienceOSS of how to Read Excel files from Python.

Following that example, in six lines I had a simple program to print the spreadsheet's contents:

import xlrd

for filename in sys.argv[1:] :
    wb = xlrd.open_workbook(filename)
    for sheetname in wb.sheet_names() :
        sh = wb.sheet_by_name(sheetname)
        for rownum in range(sh.nrows) :
            print sh.row_values(rownum)

Of course, having gotten that far, I wanted better formatting so I could compare the values in the spreadsheet. Didn't take long to write, and the whole thing still came out under 40 lines: xlsrd. And I was able to read that XLS file that was mailed to the club, easily and without hassle.

I'm forever amazed at all the wonderful, easy-to-use modules there are for Python.

Tags: python, programming
[ 10:58 Aug 31, 2011    More programming | permalink to this entry | ]

Teaching programming with "program a person"

A few weeks ago, at the annual GetSET engineering summer camp for high school girls, I taught my usual one-day workshop on beginning programming in Javascript.

The big question every year is always how to make the class more interactive. The girls who come to GetSET are great -- smart and motivated -- but after six hours of lectures and working through exercises, anyone, of any age, is going to glaze over. Especially when it's their first introduction to programming and they only have a day to learn it. People learn better when they're asking questions, thinking and solving problems, not just listening or following instructions.

For years I've heard vague references to "programming a person" as an exercise for teaching the basic idea of programming. The idea is to get the students to come up with step-by-step instructions for someone to do something -- say, walk across the room and pick up a water bottle -- so they realize how specific you have to be. It also solves another problem: giving everyone a break from sitting still and focusing on a computer screen.

But how do you really do it? What kind of problems work best in practice? How much time should you allow? If you have a volunteer carrying out the instructions, how do you keep them from skipping steps? Surprisingly, I couldn't find anything written up to help an inexperienced would-be teacher of programming.

What I needed was a chance to try out some ideas, or watch someone with more of a clue on this sort of teaching. This year, I found opportunities for both.

First try: Toastmasters

One of the reasons I love Toastmasters, especially with a small and friendly club like Coherent Communicators, is that it offers a safe place to try new presentation techniques and get good feedback about what does and doesn't work. So I made my first try at a Toastmasters meeting a few weeks before the GetSET workshop.

I allowed 15-20 minutes for the exercise. I explained to the audience that I wanted them to get me to turn left, walk over to the easel at the side of the room, touch it, turn around, walk back to the lectern, pick up the gavel and pound it on the lectern. I would solicit a command from them, write it on the whiteboard, then carry out the command and ask for the next command.

The day's audience was a fairly even mix of techies and non. I had wondered whether the audience would be widely mixed in how specific their instructions were, but they were fairly uniform -- mostly along the lines of "Turn 90 degrees left." "Take 5 steps." "Take 2 more steps". Of course, there were a few joking suggestions from the techies, like "send an electrical impulse from your brain to your left quadriceps", that you wouldn't expect with a high school group, but mostly everyone was on the same page.

When I got near the easel, we hit "Raise your right arm". (Oops, not close enough yet.) "Um ... lean forward about a foot?" A good illustration of being specific ... just the sort of thing I was hoping for.

They got me back to the lectern, got me to pick up the gavel (I was letting them skip a few steps by this point) ... and improvised a little, getting me to knock my head rather than the lectern. That was fun, and got some laughs ... it worked well.

I had hoped to do a second run where I guided them into understanding a while loop ("while (not yet to the easel), take another step"). But seeing a yellow light from the timer, I opted for a quick explanation of how a loop would work rather than guiding the audience into it. I found out later that the timer had hit the wrong button and only given me 8 minutes rather than my requested 15-20 ... so 20 minutes actually would have been plenty of time to cover loops as well as basic instructions. Disappointing ... but I was surprised we'd gotten so much done in so little time.

Lessons learned:

• Draft a volunteer to write the instructions on the board. It was distracting and time-wasting to run back from the side of the room to write each new instruction.
• You can teach the basic concepts in less than ten minutes.

Try 2: "Program a blind robot"

For the real workshop, I had help in the form of Esther Heller, an experienced girl scout leader as well as many year GetSET veteran. Esther had done exercises like this before and was willing to take the lead; I was looking forward to learning from her. We had discussed two different variants, and decided to try both of them at different times during the day.

For the first variant, we waited until mid-morning when the class was bogging down a bit and looked like they needed a break. Esther called for two volunteers: one programmer and one robot. The girl playing the robot was blindfolded with a bandanna and escorted to the door of the room, while Esther whispered the task to the other girl. The task was something like walking over to a water bottle, picking it up, walking over to another girl and handing it to her -- though the rest of us didn't know that until it was completed.

The instructions suggested by the girls were quite similar to the ones I'd heard in Toastmasters. There was lots of "Take 5 steps" ... "take two more steps", guessing at how many steps it would take to get from one place to another. No one came up with anything like a loop or conditional. I'd wondered if anyone would try remote control -- "walk" then wait until the right moment to yell "STOP!" -- but no one did.

The blindfolding worked really well. I'd worried that with a volunteer chosen to be the robot, she might skip steps she hadn't been given. But if the "robot" is blindfolded and doesn't know the task, she can't skip steps; she can only do what she's programmed to. The only problem was that a blindfolded person told to walk straight ahead does not necessarily hold to a straight line, much to the consternation of the girl playing the programmer.

There was a lot of "turn right" ... "no, not that much, turn back left again" ... "now turn JUST A LITTLE to the right" that helped stress the need for specificity -- exactly what we were after. I had wondered beforehand whether anyone would ever suggest anything like "turn right by 30 degrees", but no one, either in Toastmasters or GetSET, ever did.

The exercise was successful and everybody seemed to have fun, so it broke up the morning well. We didn't get to loops or conditionals, though. I didn't record how long we spent, but it was probably in the neighborhood of 20 minutes.

Lessons:

• Blindfolding and choosing a volunteer definitely helps this exercise: it solves the problem of a volunteer who might skip steps.
• I wished the whole room knew what the task was ... but I'm not sure how to accomplish that. Either you have to escort the "robot" far enough away that she can't hear you explain it, or write it on the board after she's blindfolded. Extra time either way.

Try 3, in groups: "The muffin is ready"

At the end of the day, we tried Esther's favorite variant. You're watching TV, and you want to go to the kitchen, get an English muffin, toast it, put butter/jam/peanut butter/whatever on it, take it back to your seat and eat it. What are the steps?

Esther divided the girls into groups of 4-5 and passed out post-its on which to write the steps. There was some inertia getting started ... it was late in the day and everybody was tired. (That's not unique to this exercise -- it's always a challenge to come up with something that will hold the girls' interest for the last hour. It's a long day for everyone.)

Eventually they got rolling and got into it -- I saw some very long stacks of post-its from various groups. With ten minutes left to go in the session, Esther picked two volunteers from one group: one to read the instructions, one to execute them. She pointed out places where they skipped steps -- "Hey, wait, how can she get the muffins out of the cupboard without opening the cupboard first?" After a minute or two, Esther called on a new pair from a different group to continue where the first pair had left off.

As she worked through all the groups, you could see each group becoming more cognizant of steps they had skipped, and improvising them on the spot. Despite the end-of-day crankiness, you could see they were learning from the exercise.

Lessons:

• Splitting into groups allows for more discussion among the girls, and comparing various groups' answers is fun.
• Splitting into groups takes a lot of time, and you have to monitor to make sure all the groups are actually working on the problem and not just chatting.

So which is better? The muffin exercise was definitely more time consuming than the previous "robot" exercise, due to overhead of splitting into groups and bringing up volunteers from each group. On the other hand, I could see there was benefit in having them work in small groups, and in the touch of competition in comparing their group's answers with the ones from other groups.

It was hard to compare the two exercises directly to see which one worked better, because of end-of-day crankiness. But they both worked well -- I'm going to keep using some variant of this in future workshops, ideally with loops and conditionals added. Thanks, Esther, for your expertise ... and to the students and the rest of the volunteers for making it a successful class!

Tags: education, programming, toastmasters
[ 16:34 Aug 28, 2011    More education | permalink to this entry | ]

Vaio tips for Debian Squeeze

I switched to the current Debian release, "Squeeze", quite a few months ago on my Sony Vaio laptop. I've found that Squeeze, with its older kernel and good attention to power management (compared to the power management regressions in more recent kernels), gets much better battery life than either Arch Linux or Ubuntu on this machine. I'm using Squeeze as the primary OS at least until the other distros get their kernel power management sorted out.

I did have to solve a couple of minor problems when switching over, though.

Suspend/Resume quirks

The first problem was that my Vaio TX650 would freeze on resuming from suspend -- something that every other Linux distro has handled out of the box on this machine.

The solution turned out to be simple though non-obvious, apparently a problem with controlling power to the display:

sudo pm-suspend --quirk-dpms-on

That wasn't easy to find, but ever since then the machine has been suspending without a single glitch. And it's a true suspend, unlike Ubuntu Natty, which on this machine will use up a full battery if I leave it suspended all day -- Natty uses nearly as much power when suspended as it does running.

Adjusting screen brightness: debugging ACPI

Of course, once I got that sorted out, there were the usual collection of little changes I needed to make. Number one was that it didn't automatically handle brightness adjustment with the Fn-F5 and Fn-F6 keys.

It turned out my previous technique for handling the brightness keys didn't work, because the names of the ACPI events in /etc/acpi/events had changed. Previously, /etc/acpi/events/sony-brightness-down had contained references to the Sony I/O Control, or SPIC:

event=sony/hotkey SPIC 00000001 00000010
action=/etc/acpi/sonybright.sh down

That device didn't exist on Squeeze. To find out what I needed now, I ran acpi-listen and typed the function-key combos in question. That gave me the codes I needed. I changed the sony-brightness-down file to read:

event=video/brightnessdown BRTDN 00000087 00000000
action=/etc/acpi/sonybright.sh down

It's probably a good thing, changing to be less Sony-specific ... but as a user it's one of those niggling annoyances that I have to go chase down every time I upgrade to a new Linux version.

Tags: linux, suspend, acpi, debian, sony, vaio
[ 12:07 Aug 27, 2011    More linux/laptop | permalink to this entry | ]

Deleting email from a mail server with Python

How do you delete email from a mail server without downloading or reading it all?

Why? Maybe you got a huge load of spam and you need to delete it. Maybe you have your laptop set up to keep a copy of your mail on the server so you can get it on your desktop later ... but after a while you realize it's not worth downloading all that mail again. In my case, I use an ISP that keeps copies of all mail forwarded from one alias to another, so I periodically need to clean out the copies.

There are quite a few reasons you might want to delete mail without reading it ... so I was surprised to find that there didn't seem to be any easy way to do so.

But POP3 is a fairly simple protocol. How hard could it be to write a Python script to do what I needed?

Not hard at all, in fact. The poplib package does most of the work for you, encapsulating both the networking and the POP3 protocol. It even does SSL, so you don't have to send your password in the clear.

Once you've authenticated, you can list() messages, which gives you a status and a list of message numbers and sizes, separated by a space. Just loop through them and delete each one.

Here's a skeleton program to delete messages:

server = "mail.example.com"
port = 995
user = "myname"
passwd = "seekrit"

pop = poplib.POP3_SSL(server, port)
pop.user(user)
pop.pass_(passwd)

poplist = pop.list()
if poplist[0].startswith('+OK') :
    msglist = poplist[1]
    for msgspec in msglist :
        # msgspec is something like "3 3941", 
        # msg number and size in octets
        msgnum = int(msgspec.split(' ')[0])
        print "Deleting msg %d\r" % msgnum,
        pop.dele(msgnum)
    else :
        print "No messages for", user
else :
    print "Couldn't list messages: status", poplist[0]
pop.quit()

Of course, you might want to add more error checking, loop through a list of users, etc. Here's the full script: deletemail.

Tags: email, python, programming
[ 17:41 Aug 25, 2011    More programming | permalink to this entry | ]

Dragonfly nymph

A recent short hike at Sanborn was unexpectedly productive for creepy-crawlies.

At the lower pond, we looked for California newts. There were lots of newts last week a few miles away at Montebello, so we thought we'd see some at Sanborn too. But there weren't many adult newts in the pond -- we could only find three. That pond has never recovered from its draining three years ago, which seems to have killed all the fish and crayfish and driven away most of the newts.

But we did see one very interesting sight: a large underwater bug, at least 2 inches long. It first caught our attention jetting through the water to the shallows near where we stood, where it sank to the bottom and rested for a while (posing for pictures!) It moved only slightly during the couple of minutes we watched it ... then it suddenly jetted off toward another part of the pond. I say "jetted" because it didn't move its legs or proto-wings at all; it moved like a torpedo, presumably propelled by a jet of water.

Upon returning home, at tip from a friend (thanks, Wolf!) I looked up dragonfly nymphs. Indeed, that's what this was. Much more massive than an adult dragonfly, these larvae apparently live underwater for several years, eating bugs, fish and small amphibians, until they're finally ready to metamorphose into the beautiful winged adults we're familiar with.

An interesting creature, and one I'd never seen before.

The small upper pond, unlike the lower one, was full of life. Small fish up to about an inch and a half schooled in the shallows. Some larger koi lurked near the reeds. But I spotted something that clearly wasn't a fish: yes, there's still at least one larval newt left in the pond. It obligingly lounged in a sunny spot near the pond's edge so I could snap pictures capturing its feathery gills as well as four tiny feet.

We also stopped by the scum pond at Walden West. No bullfrogs, no turtles. The only life we saw there was a couple of female mallards, eagerly vacuuming up the scum. That pond, with its surface completely covered with algae, must be paradise for an algae-eating duck ... I wonder why I don't see more of them there.

And as long as the subject is crawling animals, I can't resist throwing in a snapshot of a garter snake I spotted today at Huddart. Nothing especially rare or exotic, but a pretty little thing nontheless.

Tags: nature
[ 19:39 Aug 21, 2011    More nature | permalink to this entry | ]

Beginning Python: Sorting lists of objects

The Beginning Python class has pretty much died down -- although there are still a couple of interested students posting really great homework solutions, I think most people have fallen behind, and it's time to wrap up the course.

So today, I didn't post a formal lesson. But I did have something to share about how I used Python's object-oriented capabilities to solve a problem I had copying new podcast files onto my MP3 player. I used Python's built-in list sort() function, along with the easy way it lets me define operators like < and > for any object I define.

You can read all about it in my post to the Courses list describing how I sorted my list of podcast objects. Or just go straight to the final program, pods.

Tags: python, programming, education
[ 19:48 Aug 19, 2011    More education | permalink to this entry | ]

Green Delicious apples

This past spring I planted an apple tree.

I expected it would be simple, even though I had a couple of goals I wanted to meet. I prefer tart green apples -- no mealy too-sweet red delicious types ... or worse, golden delicious. And I was hoping to get something that matured any time other than mid-October -- because that's when the guava trees go crazy and we're inundated with fruit. So, go to the nursery, find a green apple tree that matures at some other time, buy it and plant it. Right?

Turns out apples are complicated. Some apple varieties are triploid, which has to do with how many chromosomes need to group together to produce fruit. Diploid apple trees can produce fruit all by themselves ("self pollinating"), while triploid varieties need another apple tree nearby -- one that flowers at about the same time -- to pollinate them.

In addition, apparently you can't just take a seed out of an apple you ate and plant it. Well, you can, but it won't grow as well. Modern apple trees take branches from varieties that make good fruit, and graft them to rootstock from different, presumably hardier, varieties.

But as long as they're grafting anyway, that means it's just as easy to make a tree that has branches of several different types. Cool! And with any luck, they'll be types that can either pollinate each other, if they don't self-pollinate.

After failing to locate any pippins or other non-granny green apples, I ended up with a little tree with four branches: fuji, gala, granny smith (we'll just have to compete with the guavas) and ... golden delicious. Yes, it turns out that you can't buy a multi-variety apple tree that doesn't include golden delicious. My least favorite apple. I have no idea why they all include it. Maybe it's an exceptionally good pollinator for the varieties that actually taste good.

I planted the little tree, and amazingly, it flourished. The nice man at God's Little Acre said it would bear this year. I raised an eyebrow -- apples from a little tree that only came up to my waist? (Readers who haven't met me, just take my word that isn't very high.)

But a month or so after planting, the tree was a foot taller and covered with flowers. And a few weeks after that, there were three tiny apples growing: a fuji, a granny and a golden. How exciting!

Exciting for a few weeks -- until two of the three little grape-sized apples-to-be vanished. I still don't know if some bird mistook them for a berry, or a mischievous squirrel wanted something to bury. All I was left with was -- doesn't it just figure! - the golden delicious, steadily growing on its branch.

But wait. Apples all start out green, right? This one certainly was. What if I picked it before it turned yellow? Would that give me that early-maturing green apple I'd been hoping for? Maybe golden delicious wasn't so bad after all.

I eagerly watched over the next month or two as my single apple grew and matured. And last week, it finally started to change from a deep pippin-like hue to a more yellowish green.

So I picked it. And ate it for breakfast. It was excellent: tart and firm.

I hereby announce the invention of the "green delicious" apple variety. I definitely recommend it. I'm looking forward to next year's crop ... which I hope will be a bit larger than this year's.

Tags: garden
[ 19:54 Aug 18, 2011    More nature | permalink to this entry | ]

Fixing broken highlighting in Google search bar

Google has been doing a horrible UI experiment with me recently involving its search field.

I search for something -- fine, I get a normal search page page. At the top of the page is a text field with my search terms, like this:

Now suppose I want to modify my search. Suppose I double-click the word "ui", or drag my mouse across it to select it, perhaps intending to replace it with something else. Here's what happens:

Whoops! It highlighted something other than what I clicked, changed the font size of the highlighted text and moved it. Now I have no idea what I'm modifying.

This started happening several weeks ago (at about the same time they made Instant Seach mandatory -- yuck). It only happens on one of my machines, so I can only assume they're running one of their little UI experiments with me, but clearing google cookies (or even banning cookies from Google) didn't help. Blacklisting Google from javascript cures it, but then I can't use Google Maps or other services.

For a week or so, I tried using other search engines. Someone pointed me to Duck Duck Go, which isn't bad for general searches. But when it gets to technical searches, or elaborate searches with OR and - operators, google's search really is better. Except for, you know, minor details like not being able to edit your search terms.

But finally it occurred to me to try firebug. Maybe I could find out why the font size was getting changed. Indeed, a little poking around with firebug showed a suspicious-looking rule on the search field:

.gsfi, .lst {
    font: 17px arial,sans-serif;
}

and disabling that made highlighting work again.

So to fix it permanently, I added the following to chrome/userContent.css in my Firefox profile directory:

.gsfi, .lst {
  font-family: inherit !important;
  font-size: inherit !important;
}

And now I can select text again! At least until the next time Google changes the rule and I have to go back to Firebug to chase it down all over again.

Note to Google UI testers:

No, it does not make search easier to use to change the font size in the middle of someone's edits. It just drives the victim away to try other search engines.

Tags: tech, google, css, web
[ 22:05 Aug 16, 2011    More tech/web | permalink to this entry | ]

Beginning Python, Lesson 9: More extras

Lesson 9 in my online Python course is up: Lesson 9: Extras (requested topics), including string operations, web development and GUI toolkits.

The web development and GUI toolkits are topics which were requested by students, while the string ops are things that just seemed too useful not to include.

Tags: python, programming, education
[ 17:45 Aug 12, 2011    More education | permalink to this entry | ]

Changing your email address in Yahoo Groups

A while ago I switched ISPs, and maintaining a lot of email addresses got more complicated. So I decided to consolidate.

But changing your email address turns out to be tricky on some sites. For example, on Amazon it apparently requires a phone call to customer support (I haven't gotten around to it yet, but that's what their email support people told me to do).

Then there's Yahoo groups. I'm in quite a few groups, so when I made the switch, I went to groups.yahoo.com, added a valid address and made it my primary address. Great -- thought I was done.

Weeks later, it occurred to me that I hadn't been getting any mail from a bunch of groups I used to get mail from. I went to Yahoo groups and clicked around for five minutes trying to find something that would show me my email addresses. Eventually I gave up on that, went to one of the groups I hadn't been getting, and saw a notice at the top:

The email address you are using for this group is currently bouncing. More info here.

So naturally, I clicked on the More info here link, and got taken to a page that said:

Groups Error: No Permission

No Permission
You do not have permission to access this page.

Gosh, that's some helpful info, Yahoo!

So how do you really change it?

There are lots of ways to get to the Yahoo Groups "Manage your email addresses" page -- but it shows only the new address, listed as primary, as primary, and doesn't show the old address where it's actually trying to send all the mail. No way to delete it from there.

Now, you can Edit membership in any particular group: that shows both the old nonworking address (with the box checked) and the new one (check the box to change it). Great -- so I'm supposed to do that for all 25 or so groups I'm in? Seriously?

After much searching, I finally found an old discussion thread with a link to the Edit my groups page. Scroll down to the bottom and look for "Set all of the above to".

It's still not a one-step operation -- my groups are spread across three pages and there's no "View all on one page", and each time you submit a page, it takes you back to "View groups" mode so you have to click on the next page, then click on "Edit groups" again. Still, it's a heck of a lot faster than going through all the groups one by one.

In theory it's all changed now. But then, I thought that last time ... time will tell whether the mail actually starts flowing again.

Meanwhile, Yahoo developers: you might want to take a look at that "More info" page that just gives a permission error.

Tags: email, web
[ 18:58 Aug 09, 2011    More tech | permalink to this entry | ]

Beginning Python, Lesson 8: Extras

Lesson 8 in my online Python course is up: Lesson 8: Extras, including exception handling, optional arguments, and running system commands. A motley collection of fun and useful topics that didn't quite fit anywhere in the earlier formal lessons, but you'll find a lot of use for them in writing real-world Python scripts. In the homework, I have some examples of some of my scripts using these techniques; I'm sure the students will have lots of interesting problems of their own.

Tags: python, programming, education
[ 14:56 Aug 05, 2011    More education | permalink to this entry | ]

Chlorine and reptiles

We went exploring around the upper Skyline-to-the-Sea trail yesterday. The mysterious chlorine smell was very evident, for the first time this year. Usually I've first noticed it in early July or even June, but although we had some very hot weather in early June this year, it wasn't enough to bring out the smell. I've made no progress in identifying it, but I continue to suspect tanoaks as the chlorine culprit.

It was a good day for reptiles, too. We surprised the biggest ring-necked snake I've ever seen -- well over two feet long and thicker than my thumb (which admittedly isn't saying much). It hastened off the trail before I could get the camera out. Then back at home, I found a small young alligator lizard splayed out in the shade on the sidewalk of our back yard. We've occasionally had alligator lizards here before, but never such a small one. Again, no picture; instead we just watched as it made its way across the yard to hide under the rosemary. I hope it stays around.

Tags: chlorine, nature
[ 11:31 Aug 01, 2011    More nature | permalink to this entry | ]