Shallow Thoughts : : programming

Using Census Population Data

A couple of us in the local League of Women Voters chapter have been talking about how our county's school board is elected. There are five school districts, to go with the five elementary schools ... but when it comes time to vote for the school board, the voting districts aren't the same as the school districts. For example, a parent whose kid goes to Barranca might be voting for the school board rep from the Aspen district. This confuses pretty much everybody.

Apparently the reason it's set up this way is that the voting districts need to have roughly equal population, which the actual school districts don't. That made us curious about how the populations of the actual school districts compared. But it turns out if you ask that question, no one has those numbers, or at least, we couldn't find anyone who would release them.

"No problem!" I chirped. "I can get population data from the Census website, and combine that with the GIS for the school districts!"

Little did I know, when I promised that, what a soul-sucking pit of despair the Census website is, and how difficult it is to get data out of it.

Read more ...

Creating an Image with Wrapped Text using the Python Imaging Library

I stumbled onto the page for this year's Asimov's Magazine Readers' Award Finalists. They offer all the stories right there -- but only as PDF. I prefer reading fiction on my ebook reader (a Kobo Clara with 6" screen), away from the computer. I spend too much time sitting at the computer as it is. But trying to read a PDF on a 6" screen is just painful.

The open-source ebook program Calibre has a command-line program called ebook-convert that can convert some PDF to epub. It did an okay job in this case — except that the PDFs had the wrong author name (they all have the same author, so I'm guessing it's the name of the person who prepared the PDFs for Asimov's), and the wrong title information (or maybe just no title), and ebook-convert compounded that error by generating cover images for each work that had the wrong title and author.

I went through the files and fixed each one's title and author metadata using my epubtag.py Python script. But what about the cover images? I wasn't eager to spend the time GIMPing up a cover image by hand for each of the stories.

Read more ...

How Common is Easter in March?

"Easter is March 31 this year," my husband said. "I think that's rare, having Easter in March."

"I guess so," I said.

"There's a Cray somewhere that they use to calculate the date each year," he joked.

And of course that made me want to find out if Easters in March really are rare.

Read more ...

Drag-and-Drop in Python Qt6

I had a need for a window to which I could drag and drop URLs.

I don't use drag-and-drop much, since I prefer using the commandline rather than a file manager and icon-studded desktop. Usually when I need some little utility and can't immediately find what I need, I whip up a little Python script.

This time, it wasn't so easy. Python has a GUI problem (as does open source in general): there are quite a few options, like TkInter, Qt, GTK, WxWidgets and assorted others, and they all have different strengths and (especially) weaknesses.

Drag-and-drop turns out to be something none of them do very well.

Read more ...

Los Alamos Voting Data on a Folium Choropleth Map

Somebody in a group I'm in has commented more than once that White Rock is a hotbed of Republicanism whereas Los Alamos leans Democratic. (For outsiders, our tiny county has two geographically-distinct towns in it, with separate zip codes, though officially they're both part of Los Alamos township which covers all of Los Alamos county. White Rock is about half the size of Los Alamos.)

After I'd heard her say it a couple times, I got curious. Was it true? I asked her for a reference, but she didn't have one. I decided to find out.

Read more ...

Clicking through a Translucent Image Window

Five years ago, I wrote about Clicking through a translucent window: using X11 input shapes and how I used a translucent image window that allows click-through, positioned on top of PyTopo, to trace an image of an old map and create tracks or waypoints.

But the transimageviewer.py app that I wrote then was based on GTK2, which is now obsolete and has been removed from most Linux distro repositories. So when I found myself wanting GIS to help investigate a growing trail controversy in Pueblo Canyon, I discovered I didn't have a usable click-through image viewer.

Read more ...

Importing Cookies from a Firefox Profile in Python

I wrote at length about my explorations into selenium to fetch stories from the New York Times (as a subscriber). But I mentioned in Part III that there was a much easier way to fetch those stories, as long as the stories didn't need JavaScript.

That way is to use normal file fetching (using urllib or requests), but with a CookieJar object containing the cookies from a Firefox session where I'd logged in.

Read more ...

Wikipedia: All Roads Lead to ... Philosophy?

At a recent LUG meeting, we were talking about various uses for web scraping, and someone brought up a Wikipedia game: start on any page, click on the first real link, then repeat on the page that comes up. The claim is that this chain always gets to Wikipedia's page on Philosophy.

We tried a few rounds, and sure enough, every page we tried did eventually get to Philosophy, usually via languages, which goes to communication, goes to discipline, action, intention, mental, thought, idea, philosophy.

It's a perfect game for a discussion of scraping. It should be an easy exercise to write a scraper to do this, right?

Read more ...