Shallow Thoughts

Akkana's Musings on Open Source Computing and Technology, Science, and Nature.

Mon, 11 Nov 2019

Mercury Transit: Comparing Between H-alpha and White Light

[Mercury transit 11/11/2019] The Mercury transit is over. But we learned some interesting things.

I'd seen Mercury transits before, but this is the first time we had an H-alpha scope (a little 50mm Coronado PST) in addition to a white light filter (I had my 102mm refractor set up with the Orion white-light filter).

As egress approached, Dave was viewing in the H-alpha while I was on the white light scope. When I saw the black-drop effect at third contact, Mercury was still nowhere near the edge in the H-alpha: the H-alpha shows more of the solar atmosphere so the sun's image is noticably bigger. This was the point when we realized that we should have expected this and been timing and recording. Alas, it was too late.

Mercury was roughly 60% out in the white light filter -- just past the point where the "bite" it made in the limb of the sun -- by the time Dave called out third contact. We guessed it was roughly a minute, but that could be way off.

For fourth contact, Dave counted roughly 45 seconds between when I couldn't see Mercury any more and when he lost track of it. This is pretty rough, because it was windy, seeing was terrible and there was at least a 15-second slop when I wasn't sure if I could any indentation in the limb; I'm sure it was at least as hard in the Coronado, which was running at much lower magnification.

So we had a chance to do interesting science and we flubbed it. And the next chance isn't til 2032; who knows if we'll still be actively observing then.

I wanted to at least correlate those two numbers: 45 seconds and 60% of a Mercury radius.

Mercury is about 10" (arcseconds) right now. That was easy to find. But how fast does it move? I couldn't find anything about that, searching for terms like mercury transit angular speed OR velocity. I tried to calculate it with PyEphem but got a number that was orders of magnitude off. Maybe I'll figure it out for a later article, but I wanted to get this posted quickly.

[Mercury transit 11/11/2019 in H-alpha] I didn't spend much time trying photography. I got a couple afocal snaps with my pocket digital camera through the white-light scope that worked out pretty well. I wasn't sure that would work for the Coronado: the image is fairly dim. The snaps I did get show Mercury, though none of the interesting detail like faculae and the one tiny prominence that was visible. But the interesting thing is the color. To the eye, the H-alpha scope image is a slightly orangy red, but in the digital camera it came out a startling purplish pink. This may be due to the digital camera's filters passing some IR, confusing the algorithms that decide how to shift the color. Of course, I could have adjusted the color in GIMP back to the real color, but I thought it was more interesting to leave it the hue it came out of the camera. (I did boost contrast and run an unsharp mask filter, to make it easier to see Mercury.)

Anyway, fun and unexpectedly edifying! I wish we had another transit happening sooner than 2032.

Tags: ,
[ 12:15 Nov 11, 2019    More science/astro | permalink to this entry | comments ]

Fri, 08 Nov 2019

Mercury Transit Next Monday

[Mercury Transit 2006, photo by Brocken Inaglory]
Mercury Transit 2006, photo by Brocken Inaglory
Next Monday, November 11, is a transit of Mercury across the sun.

Mercury transits aren't super rare -- not once- or twice-in-a-lifetime events like Venus transits -- but they're not that common, either. The last Mercury transit was in 2016; the next one won't happen til 2032.

This year's transit isn't ideal for US observers. The transit will already be well underway by the time the sun rises, at least in the western US. Here in New Mexico (Mountain time), the sun rises with Mercury transiting, and the transit lasts until 11:04 MST. Everybody else, check timeanddate's Mercury Transit page for your local times.

Mercury is small, unfortunately, so it's not an easy thing to see without magnification. Of course, you know that you should never look at the sun without an adequate filter. But even if you have safe "eclipse glasses", it may be tough to spot Mercury's small disk against the surface of the sun.

[binocular projection of a solar eclipse] One option is to take some binoculars and use them to project an image. Point the big end of the binoculars at the sun, and the small end at a white surface, preferably leaning so it's perpendicular to the sun. I don't know if binocular projection will give a big enough image to show Mercury, so a very smooth and white background, tilted so it's perpendicular to the sun, will help. (Don't be tempted to stick eclipse glasses in front of a binocular or telescope and look through the eyepiece! Stick to projection unless you have filters specifically intended for telescopes or binoculars.)

Of course, a telescope with a safe solar filter is the best way to see a transit. If you're in the Los Alamos area, I hear the Pajarito Astronomers are planning to set up telescopes at Overlook Park. They don't seem to have announced it in any of the papers yet, but I see it listed on the Pajarito Astronomers website. There's also an event planned at the high school where the students will be trying to time Mercury's passage, but I don't know if that's open to the public. Elsewhere in the world, check with your local astronomy club for Mercury transit parties: I'm sure most clubs have something planned.

I was discussing the transit with a couple of local astronomers earlier this week, and one of them related it to the search for exoplanets. One of the main methods of detecting exoplanets is to measure the dimming of a star's light as a planet crosses its face. For instance, in 55 Cancri e, you can see a dimming as the planet crosses the star's face, and a much more subtle dimming when the planet disappears behind the star. As Mercury crosses the Sun's face, it blocks some of the sun's light in the same way. By how much?

The radius of Mercury is 0.0035068 solar radii, and the dimming is proportional to area so it should be 0.00350682, or 0.0000123, a 0.00123% dimming. Not very much!

But it looks like in the 55 Cancri e case, they're detecting dips of around .001% -- it seems amazing that you could detect a planet as small as Mercury this way (and certainly the planet is much bigger in the case of 55 Cancri e) ... but maybe it's possible.

Anyway, it's fun to think about exoplanets as you watch tiny Mercury make its way across the face of the Sun. Wherever you are, I hope you get a chance to look!

Update: A report from the transit: Mercury Transit: Comparing Between H-alpha and White Light.

Tags: ,
[ 11:36 Nov 08, 2019    More science/astro | permalink to this entry | comments ]

Sun, 03 Nov 2019

Emulating Raspbian on your Linux x86/amd64 System

I was planning to teach a class on Raspberry Pis, and I wanted to start with the standard Raspbian image, update it, and add some programs like GIMP that we'd use in the class. And I wanted to do that just once, before burning the image to a bunch of different SD cards. Is there a way to do that on my regular Linux box, with its nice fast processor and disk?

Why yes, there is, and it's pretty easy. But there are a lot of unclear or misleading tutorials out there, so I hope this is a bit simpler and easier to follow.

I got most of this from a tutorial that no longer seems to be available (but I'll include the link in case it comes back): Solar Staker/RPI Image/Creation.

I've tested this on Ubuntu 19.10, Debian Stretch and Buster; the instructions should be pretty general except for the name of the loopback mount. Commands you type are in bold; the rest is the output you should see. $ is your normal shell prompt and # is a root prompt.

Required Packages

You'll need kpartx, qemu and some supporting packages. On Debian or Ubuntu:

$ sudo apt install kpartx qemu binfmt-support qemu-user-static

You've probably already downloaded a Raspbian SD card image.

Set Up the Loopback Devices

kpartx can read the Raspbian ISO image and split it into the two filesystems it would have if you wrote it to an SD card. It turns the filesystems into loopback devices you can mount like regular filesystems.

$ sudo kpartx -av 2019-09-26-raspbian-buster-lite.img
add map loop10p1 (253:1): 0 524288 linear 7:10 8192
add map loop10p2 (253:2): 0 3858432 linear 7:10 532480

Make a note of those loopback device names. They may not always be loop10p1 and loop10p2, but they'll probably always end in 1 and 2. 1 is the Raspbian /boot filesystem, 2 is the Raspbian root.

(Optional) Check and Possibly Resize the Raspbian Filesystem

Make sure that the Raspbian filesystem is intact, and that it's a reasonable size.

In practice, I didn't find this made any difference (everything was fine to begin with), but it doesn't hurt to make sure.

$ sudo e2fsck -f /dev/mapper/loop10p2
e2fsck 1.45.3 (14-Jul-2019)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
rootfs: 44367/120720 files (0.3$ non-contiguous), 292014/482304 blocks

$ sudo resize2fs /dev/mapper/loop10p2
resize2fs 1.45.3 (14-Jul-2019)
The filesystem is already 482304 (4k) blocks long.  Nothing to do!

Mount the Loopback Filesystems

You're ready to mount the two filesystems. A Raspbian SD card image contains two filesystems.

Partition 1 is a small vfat /boot filesystem containing the kernel and some other files it needs for booting, plus the two important configuration files cmdline.txt and config.txt.

Partition 2 is the Raspbian root filesystem in ext4 format. The Raspbian root includes an empty /boot directory; mount the root first, then mount the Raspbian boot partition on Raspbian's /boot:

$ sudo mkdir /mnt/pi_image
$ sudo mount /dev/mapper/loop10p2 /mnt/pi_image
$ sudo mount /dev/mapper/loop10p1 /mnt/pi_image/boot

Prepare for Chroot

You're going to chroot to the Raspbian filesystem. Chroot limits the filesystem you can access, so when you type /, instead of your host filesystem's / you'll see the root of the Raspbian filesystem, /mnt/pi_image. That means you won't have access to your host system's /usr/bin, any more.

But qemu needs /usr/bin/qemu-arm-static to be able to emulate ARM binaries in user mode. So copy that to the Raspbian filesystem so you'll still be able to access it after the chroot:

$ sudo cp /usr/bin/qemu-arm-static /mnt/pi_image/usr/bin

Chroot to the Raspbian System

$ sudo chroot /mnt/pi_image /bin/bash
root:/#

Now you're running in Raspbian's root filesystem. All the binaries in your path (e.g. /bin/ls, /bin/bash) are ARM binaries, but if you try to run them, qemu will see qemu-arm-static and run the program as though you're on an actual Raspberry Pi.

Run Stuff!

Now you can run Raspbian commands.

# file /bin/ls
/bin/ls: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=67a394390830ea3ab4e83b5811c66fea9784ee69, stripped
#
# /bin/ls
bin   dev  home  lost+found  mnt  proc	run   srv  tmp	var
boot  etc  lib	 media	     opt  root	sbin  sys  usr

# file /bin/cat
/bin/cat: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=2239a192f2f277bd1a4892e39a41eba97266b91f, stripped
#
# cat /etc/issue
Raspbian GNU/Linux 10 \n \l

You can even install packages or update the whole system:

# apt update
Get:1 http://raspbian.raspberrypi.org/raspbian buster InRelease [15.0 kB]
Get:2 http://archive.raspberrypi.org/debian buster InRelease [25.2 kB]
Get:3 http://raspbian.raspberrypi.org/raspbian buster/main armhf Packages [13.0 MB]
Get:4 http://archive.raspberrypi.org/debian buster/main armhf Packages [259 kB]
Fetched 13.3 MB in 20s (652 kB/s)
Reading package lists... Done

# apt dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following NEW packages will be installed:
  busybox initramfs-tools initramfs-tools-core klibc-utils libklibc linux-base
  pigz
The following packages will be upgraded:
  dhcpcd5 e2fsprogs file firmware-atheros firmware-brcm80211 firmware-libertas
  firmware-misc-nonfree firmware-realtek libcom-err2 libext2fs2 libmagic-mgc
  libmagic1 libraspberrypi-bin libraspberrypi-dev libraspberrypi-doc
  libraspberrypi0 libss2 libssl1.1 libxml2 libxmuu1 openssh-client
  openssh-server openssh-sftp-server openssl raspberrypi-bootloader
  raspberrypi-kernel raspi-config rpi-eeprom rpi-eeprom-images ssh sudo
  wpasupplicant
32 upgraded, 7 newly installed, 0 to remove and 0 not upgraded.
Need to get 133 MB of archives.
After this operation, 3192 kB of additional disk space will be used.
Do you want to continue? [Y/n]

Pretty neat! Although you're not actually running Raspbian, you can run Raspbian executables with the Raspbian root filesystem mounted as though you were actually running on your Raspberry Pi.

Cleaning Up

When you're done with the chroot, just exit that shell (Ctrl-D or exit). If you want to undo everything else afterward:

$ sudo rm /mnt/pi_image/usr/bin/qemu-arm-static

$ sudo umount /mnt/pi_image/boot
$ sudo umount /mnt/pi_image

$ sudo kpartx -dv /dev/loop0
$ sudo losetup -d /dev/loop0

$ sudo rmdir /mnt/pi_image

Limitations

Keep in mind you're not really running Raspbian. You never booted the Raspbian kernel, and you can't test things that depend on Raspbian's init system, like whether networking works, let alone running the Raspbian X desktop or accessing GPIO pins. This is an ARM emulator, not a Raspberry Pi emulator.

More details

If you want to read more about qemu user mode and how it lets you run binaries from other architectures, I recommend these links:

Tags: , ,
[ 16:14 Nov 03, 2019    More linux | permalink to this entry | comments ]

Thu, 31 Oct 2019

Command of the Day: See a Random Command Name and Description

Someone on ##linux was talking about "bro pages", which turns out to be a site that collects random short examples of how to use Linux commands. It reminded me of Command Line Magic, a Twitter account I follow that gives sometimes entertaining or useful command-line snippets.

I hadn't been to that page on the Twitter website in a while (I usually use bitlbee for Twitter), and clicking through some of the tweets on the "Who to follow" accounts took me to someone who'd made a GNU CoreUtils cheat sheet. I didn't really want the printed cheat sheet, but I was interested in the commands used to generate it. The commands involved downloading an HTML page and didn't work any more -- the page was still there but its format has changed -- but that got me to thinking about how it might be fun to generate something that would show me a random command and its description, starting not from coreutils but from the set of all commands I have installed.

I can get a list of commands from the installed man pages in /usr/share/man -- section 1, for basic commands, and section 8, for system-admin commands. (The other sections are for things like library routines, system calls, files etc.)

So I can pick a random man page like this:

ls -1 /usr/share/man/man1/ /usr/share/man/man8 | shuf -n 1
which gives me a filename like xlsfonts.1.gz.

The man pages are troff format, gzipped. You can run zcat on them, but extracting the name and description still isn't entirely trivial. In most cases, it comes right after the .SH NAME line, so you could do something like

zcat $(ls -1 /usr/share/man/man1/* /usr/share/man/man8/* | shuf -n 1) | grep -A1 NAME | tail -1
(the * for the two directories causes ls to list the full pathname, like /usr/share/man/man1/xlsfonts.1.gz, instead of just the filename, xlsfonts.1.gz).

But that doesn't work in every case: sometimes the description is more than one line, or there's a line between the NAME line and the actual description.

A better way is to use apropos (man -k), which already knows how to search through man pages and parse them to extract the command name and description. For that, you need to start with the filename (I'm going to drop those *s from the command since I don't need the full pathname any more) and get rid of everything after the first '.'.

You can do that with sed 's_\.[0-9].*__': it looks for everything starting with a dot (\.) followed by a digit ([0-9] -- sed doesn't understand \d) followed by anything (.*) and replaces all of it with nothing, the empty string.

Here's the full command:

apropos $(ls -1 /usr/share/man/man1/ /usr/share/man/man8 | shuf -n 1 | sed 's_\.[0-9].*__')

Sometimes it will give more than one command: for instance, just now, testing it, it found /usr/share/man/man8/snap.8.gz, pared that down to just snap, and apropos snap found ten different commands. But that's unusual; most of the time you'll just get one or two, and of course you could add another | shuf -n 1 if want to make sure you get only one line.

It's kind of a fun way to discover new commands you may not have heard of. I'm going to put it in my .zlogin.

Tags: ,
[ 13:22 Oct 31, 2019    More linux/cmdline | permalink to this entry | comments ]

Thu, 24 Oct 2019

Fall in Frijoles Canyon

[Tree arch in autumn] We had a lovely hike yesterday down Bandelier's Frijoles Canyon, from Ponderosa to Upper Crossing then down the canyon to the Visitor Center (where we'd left a shuttle car), for a one-way total about eight miles.

Our hiking group had missed the peak of the aspens last week at Santa Barbara, so we were happy to discover that fall is still in full swing in Frijoles.

There are no aspens down this low, but there are plenty of cottonwoods, a related tree that turns the same vibrant yellow. Plus maples, Gambel oaks giving a nice multicolored show this year, and plenty of other trees I can't identify. (For people who really need a dose of aspen, the slopes of Pajarito Mountain still have some nice bright yellow patches.)

[Frijoles natural bridge] We had lunch by the natural bridge: a huge rock slab that collapsed into the opposite canyon wall, probably some time in the last ten years. No one is sure exactly when it happened because the canyon was impassable after the floods of 2013 and not very passable even in the decade before that. But old-timers who used to hike the canyon decades ago swear there was nothing like the natural bridge there, and it's the sort of thing you'd remember. When the trail was cleared out and made passable again a couple of years ago, the bridge was a surprise to everyone. Geology in action in our lifetimes!

[October snow] I started the hike in two sweaters and gloves, but was down to a t-shirt by the time we got to the canyon.

The next day, today, the temperature was around 31 at the house, and it snowed! New Mexico weather is so much more fun than California.

As I write this, there's still snow on the ground, the juncos are going crazy fueling up for winter, the sun is out here but I'm looking down at drifting fog over White Rock Canyon. And the oatmeal cookies just came out of the oven.

More photos from Frijoles Canyon: Frijoles Canyon Trail in October 2019.

Tags:
[ 14:18 Oct 24, 2019    More nature | permalink to this entry | comments ]

Thu, 17 Oct 2019

Adjusting PulseAudio Volume from the Command-Line (or a Window Manager)

I have a new laptop, a birthday present to myself last month. For once, rather than buying a cut-rate netbook, I decided to treat myself to a fancy Lenovo Carbon X1 with an up-to-date processor and lots of RAM.

I decided that since I have way more resources than I'm used to, I'd try installing a full Ubuntu. Not that I'm running a GNOME desktop: I'm very happy with my lightweight, fast, highly configurable Openbox setup, which does just what I tell it and no more, and doesn't surprise me with random redesigns or features that work on one version but not another. But I did let it install some system utilities I've always had problems with in the past, like NetworkManager and PulseAudio. I decided I'd give them a chance, see if they've gotten better since I last checked.

They have, though they're still a bit of a hassle to deal with. NetworkManager can be controlled through nmcli, which is poorly documented but works okay if you google long enough to find the proper incancations. PulseAudio gave me a bit more trouble, though.

The standard GUI for controlling PulseAudio is called pavucontrol. I first tried it with my laptop docked through a Totu tt-hb003a 11-in-1 USB-C hub, which lets me connect to the charger, external monitor, ethernet, SD and micro-SD slots, and extra USB ports without juggling a lot of extra cables. Totu was one of the few USB hubs that claimed Linux support, and everything I've tried on it works fine, including DVI, ethernet, SD, micro SD and USB; I haven't tried the VGA port but I assume that works too.

Oh, but wait: turns out there's another aspect of the hub I hadn't tried. pavucontrol showed two audio devices: "USB PnP Audio Device Analog Stereo" and "Built-in Audio Analog Stereo". It turned out, after much fiddling, that the "USB PnP" device was part of the hub. Pulse assumes (probably reasonably, though it's wrong in this case) that if I have a USB audio device plugged in, I probably want to use it in preference to the laptop's built-in audio. That would make sense if I actually owned any speakers I could plug in to the Totu, but I left all my computer speakers behind when I moved in 2014.

Mute/Unmute via the Keyboard

The Lenovo, like most laptops, has a dedicated key for muting, Fn-F1. It even has a little light on it to show whether it's muted. In Openbox, pressing Fn-F1 actually muted the sound, and even turned on the light. This is probably because on my previous laptop I'd set key="XF86AudioMute" to run amixer set Master toggle in .config/openbox/rc.xml. The problem is that pressing it again didn't unmute it. Instead, it was unmuting the Totu's external USB audio. Clicking "Set as fallback" on the built-in audio in pavucontrol didn't help.

It turns out that it is virtually impossible to persuade PulseAudio to use "Built-in Audio" when a "USB PnP Audio Device" is available. I finally found the secret: in pavucontrol's Configuration tab, set Profile for the USB device to Off. Now only the built-in device shows up in the other tabs.

But that amixer command still wasn't unmuting properly, so the next step was to find a command I could have Openbox run that would properly unmute. Someone on #linux suggested pactl set-sink-mute @DEFAULT_SINK@ toggle which worked great from the command line, but didn't work from an openbox key. I still don't understand why that is; I wasted a lot of time comparing my shell environment to openbox's environment, to no enlightenment.

Finally I went back to web searching, and found an askbuntu thread suggesting some Openbox stanzas. In particular, it seems it works better to use alsamixer rather than pactl. This worked for toggling mute:

    <keybind key="XF86AudioMute">
        <action name="Execute">
            <command>amixer -q -D pulse sset Master toggle</command>
        </action>
    </keybind>

Volume Controls via Function Keys

Partial success! Unfortunately, the volume control commands in that same askbuntu post, amixer -q -D pulse sset Master 3%+ unmute, didn't work; they did nothing. But I had already noticed that in pavucontrol, the volume controls didn't work either. In fact, if I started some music playing and then called up alsamixer, channels like Master and Speaker didn't do anything; the only channel that affected volume was PCM. After some fiddling, I discovered that I had to change Master to PCM and remove the -D pulse:

    <keybind key="XF86AudioRaiseVolume">
      <action name="Execute">
            <command>amixer sset PCM 4%+ unmute</command>
        </action>
    </keybind>
    <keybind key="XF86AudioLowerVolume">
        <action name="Execute">
            <command>amixer sset PCM 4%- unmute</command>
        </action>
    </keybind>

I'm sure I'll eventually need to fiddle some more; for one thing, if I ever want to use audio during a talk (as I did briefly at my Stonehenge talk earlier this year) I'll need to figure out how to enable a HDMI sound sink temporarily. But for now, I'm happy to have the basic laptop audio keys working.

Tags: , ,
[ 14:09 Oct 17, 2019    More linux | permalink to this entry | comments ]

Thu, 10 Oct 2019

RSS was Down, Now Back Up, with Atom

A reader pointed out to me that the RSS page on my blog hadn't been updated since May.

That must be when I updated PyBlosxom to the latest version and switched to the python3 branch. Python 2, as you probably know, is no longer going to be supported starting sometime next year (the exact date in 2020 doesn't seem to be set). Anyway, PyBlosxom was one of the few tools I use that still depend only on Python 2, and since I saw they had a python3 branch, I tried it.

PyBlosxom is no longer a project in active development: as I understand it, some of the developers drifted off to other blogging systems, others decided that it worked well enough and didn't really need further tweaking. But someone, at some point, did make a try at porting it to Python 3; and when I tried the python3 branch, I was able to make it work after I made a couple of very minor tweaks (which I contributed back to the project on GitHub, and they were accepted).

Everything went fine for a few months, until I received the alert that the index.rss and index.rss20 files weren't being generated. Curiously, the RSS files for each individual post are still there; just not the index.rss and index.rss20.

I found there was already a bug filed on that. I tried the workaround mentioned in the bug, at the same time adding Atom to the RSS 0.9.1 and RSS 2.0 flavors I've had in the past. I haven't generated Atom for all the old entries, but any new entries starting now should be available as Atom.

Fingers crossed! if you're reading this, then it worked and my RSS pages are back. Sorry about the RSS hiatus.

Tags: , ,
[ 09:10 Oct 10, 2019    More blogging | permalink to this entry | comments ]

Tue, 01 Oct 2019

Making Web Maps using Python, Folium and Shapefiles

A friend recently introduced me to Folium, a quick and easy way of making web maps with Python.

The Folium Quickstart gets you started in a hurry. In just two lines of Python (plus the import line), you can write an HTML file that you can load in any browser to display a slippy map, or you can display it inline in a Jupyter notebook.

Folium uses the very mature Leaflet JavaScript library under the hood. But it lets you do all the development in a few lines of Python rather than a lot of lines of Javascript.

Having run through most of the quickstart, I was excited to try Folium for showing GeoJSON polygons. I'm helping with a redistricting advocacy project; I've gotten shapefiles for the voting districts in New Mexico, and have been wanting to build a map that shows them which I can then extend for other purposes.

Step 1: Get Some GeoJSON

The easiest place to get voting district data is from TIGER, the geographic arm of the US Census.

For the districts resulting from the 2010 Decadal Census, start here: Cartographic Boundary Files - Shapefile (you can also get them as KML, but not as GeoJSON). There's a category called "Congressional Districts: 116th Congress", and farther down the page, under "State-based Files", you can get shapefiles for the upper and lower houses of your state.

You can also likely download them from at www2.census.gov/geo/tiger/TIGER2010/, as long as you can figure out how to decode the obscure directory names. ELSD and POINTLM, so the first step is to figure out what those mean; I never found anything that could decode them.

(Before I found the TIGER district data, I took a more roundabout path that involved learning how to merge shapes; more on that in a separate post.)

Okay, now you have a shapefile (unzip the TIGER file to get a bunch of files with names like cb_2018_35_sldl_500k.* -- shape "files" are an absurd ESRI concept that actually use seven separate files for each dataset, so they're always packaged as a zip archive and programs that read shapefiles expect that when you pass them a .shp, there will be a bunch of other files with the same basename but different extensions in the same directory).

But Folium can't handle shapefiles, only GeoJSON. You can do that translation with a GDAL command:

ogr2ogr -t_srs EPSG:4326 -f GeoJSON file.json file.shp

Or you can do it programmatically with the GDAL Python bindings:

def shapefile2geojson(infile, outfile, fieldname):
    '''Translate a shapefile to GEOJSON.'''
    options = gdal.VectorTranslateOptions(format="GeoJSON",
                                          dstSRS="EPSG:4326")
    gdal.VectorTranslate(outfile, infile, options=options)

The EPSG:4326 specifier, if you read man ogr2ogr, is supposedly for reprojecting the data into WGS84 coordinates, which is what most web maps want (EPSG:4326 is an alias for WGS84). But it has an equally important function: even if your input shapefile is already in WGS84, adding that option somehow ensures that GDAL will use degrees as the output unit. The TIGER data already uses degrees so you don't strictly need that, but some data, like the precinct data I got from UNM RGIS, uses other units, like meters, which will confuse Folium and Leaflet. And the TIGER data isn't in WGS84 anyway; it's in GRS1980 (you can tell by reading the .prj file in the same directory as the .shp). Don't ask me about details of all these different geodetic reference systems; I'm still trying to figure it all out. Anyway, I recommend adding the EPSG:4326 as the safest option.

Step 2: Show the GeoJSON in a Folium Map

In theory, looking at the Folium Quickstart, all you need is folium.GeoJson(filename, name='geojson').add_to(m). In practice, you'll probably want to more, like

Each of these requires some extra work.

You can color the regions with a style function:

folium.GeoJson(jsonfile, style_function=style_fcn).add_to(m)

Here's a simple style function that chooses random colors:

import random

def random_html_color():
    r = random.randint(0,256)
    g = random.randint(0,256)
    b = random.randint(0,256)
    return '#%02x%02x%02x' % (r, g, b)

def style_fcn(x):
    return { 'fillColor': random_html_color() }

I wanted to let the user choose regions by clicking, but it turns out Folium doesn't have much support for that (it may be coming in a future release). You can do it by reading the GeoJSON yourself, splitting it into separate polygons and making them all separate Folium Polygons or GeoJSON objects, each with its own click behavior; but if you don't mind highlights and popups on mouseover instead of requiring a click, that's pretty easy. For highlighting in red whenever the user mouses over a polygon, set this highlight_function:

def highlight_fcn(x):
    return { 'fillColor': '#ff0000' }

For tooltips:

tooltip = folium.GeoJsonTooltip(fields=['NAME'])
In this case, 'NAME' is the field in the shapefile that I want to display when the user mouses over the region. If you're not sure of the field name, the nice thing about GeoJSON is that it's human readable. Generally you'll want to look inside "features", for "properties" to find the fields defined for each polygon. For instance, if I use jq to prettyprint the JSON generated for the NM state house districts:
$ jq . House.json | less
{
  "type": "FeatureCollection",
  "name": "cb_2018_35_sldl_500k",
  "crs": {
    "type": "name",
    "properties": {
      "name": "urn:ogc:def:crs:OGC:1.3:CRS84"
    }
  },
  "features": [
    {
      "type": "Feature",
      "properties": {
        "STATEFP": "35",
        "SLDLST": "009",
        "AFFGEOID": "620L600US35009",
        "GEOID": "35009",
        "NAME": "9",
        "LSAD": "LL",
        "LSY": "2018",
        "ALAND": 3405159792,
        "AWATER": 5020507
      },
      "geometry": {
        "type": "Polygon",
        "coordinates": [
...

If you still aren't sure which property name means what (for example, "NAME" could be anything), just keep browsing through the JSON file to see which fields change from feature to feature and give the values you're looking for, and it should become obvious pretty quickly.

Here's a working code example: polidistmap.py, and here's an example of a working map:

Tags: , ,
[ 12:29 Oct 01, 2019    More mapping | permalink to this entry | comments ]