Shallow Thoughts : : Nov

Akkana's Musings on Open Source Computing and Technology, Science, and Nature.

Sat, 30 Nov 2019

Installing Lenovo Firmware Packaged as a .exe on a Linux Machine

My new Lenovo Carbon X1 Gen 7 has one irritating problem: the trackpad sometimes disappears, flooding dmesg with messages like "i2c_designware i2c_designware.1: controller timed out". Once this happens, the only fix is to reboot.

Lenovo has a fix -- new trackpad firmware -- but unlike their BIOS updates, which are installable from Linux, device firmware updates are distributed as Windows EXE files that require running Windows on the bare metal, leaving Linux users out in the cold. Ironic, since Lenovo is so popular among Linux users and is a member of the Linux Firmware Service, and the CX1 is supposedly Ubuntu certified.

Those Linux users on the forums who managed to install the firmware update raved about it, saying that indeed it solved their problem. But finding a way to to install it led me on a not-so-merry four-day quest.

Here's how I installed the firmware, in the end:

Make a Windows to Go using Rufus on a Real Windows Box

  1. Back up anything you don't want to lose, because you never know.

  2. Borrow a real Windows box. I tried many times using Windows inside VirtualBox and QEmu on top of Linux, but it never worked.
  3. On Windows, install Rufus.

  4. Download the Windows 10 Installer ISO (5 gigabytes, give or take)

  5. Find a USB stick or SD card, 16G or larger. Actually, find a bunch of them: this process is incredibly finicky about the stick you use and the only way you find out is that it doesn't work and you have to try again (see below).

  6. Use Rufus to create a Windows to Go image. The alternative is to make a Windows installer; that won't work, because you can't run anything useful from the installer, and you don't want to actually install Windows, or you wouldn't be in this fix in the first place.

    Be patient: creating a W2G image takes several hours. Click on Rufus' log file button (it's the rightmost of four obscure icons down near the lower left of the Rufus window; it has a mouseover tooltip) at any time to see what's happening; if things don't go right you might at least get some idea why.

  7. When the W2G stick is finished (whew!), move it to your Linux machine and mount its second partition (/dev/sdb2 or whatever). This will tell you it wasn't properly unmounted and it's fixing it, giving you a heart attack about whether Linux is going to change the filesystem in some way that makes it fail after you waited all that time creating it.

  8. Copy the firmware .exe to it. Wherever you want; I just put it at the root of the filesystem. Sync and unmount it.

  9. Boot your computer from the USB stick. This will take forever and may fail if the phase of the moon is wrong.

    If you're lucky and the planets are in alignment, eventually a Windows installer will come up and ask you a bunch of annoying questions about language, keyboard, whether you consent to having Microsoft spy on you in a skillion different ways, name, password, three security questions, etc. Meanwhile, you're having another heart attack because does this mean it's going to install Windows to your real disk on top of Linux? Hopefully not -- at least it didn't in my case -- but here's where you really want to have that recent backup.

  10. If you make it all the way through the questions and get a Windows screen, rejoice! Navigate to wherever you put your exe and run it. Cross your fingers -- maybe you're done!

  11. If it hangs or bluescreens during boot, or Rufus fails to create the W2G stick in the first place, try running Rufus again with a different USB stick. I think I tried five before finally finding one that worked, and the successful one (a Transcend SD card in an old Patriot USB adapter) wasn't the newest, or the fastest, or the largest. It's a mystery.

Some Approaches that Didn't Work

Before I finally got this working, I wasted four days trying many other approaches. Many of them sound very clever and reasonable and ought to work, but they didn't work for me. These include:

So, lots of different ways. Some of them have worked at some time for someone. Also, I never did try Wine. I don't think Wine would be able to run the actual exe and update the trackpad firmware (I was afraid to try it), but it's possible that Rufus in Wine might have been able to make a Windows To Go stick.

If anyone manages that -- or any other way of getting this to work -- I'd love to hear about it.

Tags: ,
[ 20:16 Nov 30, 2019    More linux | permalink to this entry | ]

Sun, 24 Nov 2019

Adjusting PulseAudio Volume from the Command-Line (or a Window Manager)

I have a new laptop, a birthday present to myself last month. For once, rather than buying a cut-rate netbook, I decided to treat myself to a fancy Lenovo Carbon X1 with an up-to-date processor and lots of RAM.

Since I have way more resources than I'm used to, I decided I'd try installing a full Ubuntu and not trying to pare it down to a super lightweight system. I'm still running the lightweight, fast, highly configurable Openbox window manager instead of a full Gnome desktop: Openbox does just what I tell it and no more, and doesn't surprise me with random redesigns. But I did let Ubuntu install some system utilities I've always avoided in the past, like NetworkManager and PulseAudio. I decided I'd give them a chance, see if they've gotten better since I last checked.

They have, though they're still a bit of a hassle to deal with. NetworkManager can be controlled through nmcli, which is poorly documented but works okay if you google long enough to find the proper incancations. PulseAudio gave me a bit more trouble.

The standard GUI for controlling PulseAudio is pavucontrol. It showed two audio devices: "USB PnP Audio Device Analog Stereo" and "Built-in Audio Analog Stereo". Turns out the USB PnP option is a sound card built into the USB hub, a Totu tt-hb003a 11-in-1 USB-C hub that lets me connect to a charger, external monitor, SD and micro-SD slots, and extra USB ports without juggling a lot of extra cables.

Pulse assumes -- probably reasonably, though it's wrong in this case -- that if I have a USB audio device connected, I probably want to use it in preference to the laptop's built-in audio. That would make sense if I had external speakers plugged in, but I left all my computer speakers behind when I moved. I should probably order some speakers. But meanwhile, I needed to persuade PulseAudio to ignore the hub and use the laptop's built-in sound system.

Mute/Unmute via the Keyboard

The Lenovo, like most laptops, has a dedicated key for muting, Fn-F1. It even has a little light on it to show whether it's muted. In Openbox, pressing Fn-F1 actually muted the sound, and even turned on the light. This is probably because I'd previously set key="XF86AudioMute" to run amixer set Master toggle in .config/openbox/rc.xml, which worked on my Pulse-free pared-down Debian netbook. The problem is that pressing iFn-F1 again didn't bring the sound back. Instead, it was unmuting the USB hub's audio. Clicking "Set as fallback" on the built-in audio in pavucontrol made no difference.

It turns out that it is virtually impossible to persuade PulseAudio to use "Built-in Audio" when a "USB PnP Audio Device" is available. I finally found the secret: in pavucontrol's Configuration tab, set Profile for the PnP USB device to Off. Now only the built-in device shows up in the other tabs.

But that amixer command still wasn't unmuting properly, so the next step was to find a command that would actually unmute. Someone on #linux suggested pactl set-sink-mute @DEFAULT_SINK@ toggle and that worked great from the command line. But when I tried to bind it in Openbox to the XF86AudioMute key, it did nothing. I still don't understand why not; I wasted a lot of time comparing my shell environment to openbox's environment and never found the difference.

Back to web searching, I found an askbuntu thread suggesting some Openbox stanzas. In particular, it apparently works better to use alsamixer rather than pactl. This finally worked for toggling mute:

    <keybind key="XF86AudioMute">
        <action name="Execute">
            <command>amixer -q -D pulse sset Master toggle</command>
        </action>
    </keybind>

Volume Controls via Function Keys

Partial success! Unfortunately, the volume control commands in that same askbuntu post, amixer -q -D pulse sset Master 3%+ unmute, did nothing. I had already noticed that in pavucontrol, the volume controls didn't work either. In fact, if I started some music playing and then called up alsamixer, channels like Master and Speaker didn't do anything; the only channel that affected volume was ALSA PCM. After some fiddling, I discovered that I had to change Master to PCM and remove the -D pulse:

    <keybind key="XF86AudioRaiseVolume">
      <action name="Execute">
            <command>amixer sset PCM 4%+ unmute</command>
        </action>
    </keybind>
    <keybind key="XF86AudioLowerVolume">
        <action name="Execute">
            <command>amixer sset PCM 4%- unmute</command>
        </action>
    </keybind>

I'm sure I'll eventually need to fiddle some more. For one thing, if I ever want to use audio during a talk (as I did briefly at my Stonehenge talk earlier this year) I'll need to figure out how to enable a temporary HDMI sound sink quickly without needing to fiddle with pavucontrol. But for now, I'm happy to have the basic laptop volume and mute keys working.

Tags: , , ,
[ 15:43 Nov 24, 2019    More linux | permalink to this entry | ]

Fri, 15 Nov 2019

Blog Post Length: Fun with Shell Pipelines, GNUplot and Matplotlib

Sometimes I tend to ramble on, and wonder if articles I'm writing are really too long for a blog post. I try to keep them under about 200 lines, but sometimes a really meaty topic demands more. It occurred to me to wonder how long a typical Shallow Thoughts post is.

A quick measure is lines, which I can measure this way starting in the directory where I have the source files for all my past posts:

find . -name '*.blx' -exec wc -l '{}' \; | sort -h >/tmp/bloglen.dat

The find produces lines like:

79 ./linux/cmdline/random-command.blx
so if I sort -h (human-readable numbers), it will sort on the first column and give me a sorted list of all posts in order of size. The shortest posts, three of them, were only five lines; the longest was 346 lines.

But what's the distribution of lengths?

[Length of all blog posts, sorted] I can plot the sorted data easily with gnuplot:

gnuplot -p -e 'plot "/tmp/bloglen.dat"'
or, if I didn't want the temp file, I could have done that all with one command:
find . -name '*.blx' -exec wc -l '{}' \; | sort -h | gnuplot -p -e 'plot "/dev/stdin"'

That's kind of interesting. But I was really more interested in seeing a frequency distribution: do I have a lot more shorter posts, or longer ones? For that I do need the temp file.

I wasted some time trying to find a way in gnuplot to plot frequency distribution. The best I found was

set style fill solid
plot '/tmp/bloglen' u ($1):(1) t 'data' smooth frequency w boxes
pause mouse close
(put that in a file and then run gnuplot on that file).

But it's not actually right: the bargraph shows 1 for lots of blog post lengths that aren't represented in the data.

I finally gave up on gnuplot, having wasted enough time that I could easily have written a Python script, and did so, which only took a few minutes.

import matplotlib.pyplot as plt

posts = []
with open('/tmp/bloglen') as fp:
    for line in fp:
        posts.append(int(line.split()[0]))

plt.hist(posts, bins=max(posts))

plt.show()

[Length of all blog posts, frequency distribution] Turns out I'm doing pretty well at keeping them under 200 lines. The vast majority of posts are fairly short, with a peak around 50 lines, and relatively few exceed 200. Only a couple of outliers get over 300.

I think I'm okay with that. Whether you, the readers, agree -- well, feel free to tell me!

For comparison, this post is 95 lines.

Tags: , , ,
[ 21:28 Nov 15, 2019    More blogging | permalink to this entry | ]

Mon, 11 Nov 2019

Mercury Transit: Comparing Between H-alpha and White Light

[Mercury transit 11/11/2019] The Mercury transit is over. But we learned some interesting things.

I'd seen Mercury transits before, but this is the first time we had an H-alpha scope (a little 50mm Coronado PST) in addition to a white light filter (I had my 102mm refractor set up with the Orion white-light filter).

As egress approached, Dave was viewing in the H-alpha while I was on the white light scope. When I saw the black-drop effect at third contact, Mercury was still nowhere near the edge in the H-alpha: the H-alpha shows more of the solar atmosphere so the sun's image is noticably bigger. This was the point when we realized that we should have expected this and been timing and recording. Alas, it was too late.

Mercury was roughly 60% out in the white light filter -- just past the point where the "bite" it made in the limb of the sun -- by the time Dave called out third contact. We guessed it was roughly a minute, but that could be way off.

For fourth contact, Dave counted roughly 45 seconds between when I couldn't see Mercury any more and when he lost track of it. This is pretty rough, because it was windy, seeing was terrible and there was at least a 15-second slop when I wasn't sure if I could any indentation in the limb; I'm sure it was at least as hard in the Coronado, which was running at much lower magnification.

So we had a chance to do interesting science and we flubbed it. And the next chance isn't til 2032; who knows if we'll still be actively observing then.

I wanted to at least correlate those two numbers: 45 seconds and 60% of a Mercury radius.

Mercury is about 10" (arcseconds) right now. That was easy to find. But how fast does it move? I couldn't find anything about that, searching for terms like mercury transit angular speed OR velocity. I tried to calculate it with PyEphem but got a number that was orders of magnitude off. Maybe I'll figure it out for a later article, but I wanted to get this posted quickly.

[Mercury transit 11/11/2019 in H-alpha] I didn't spend much time trying photography. I got a couple afocal snaps with my pocket digital camera through the white-light scope that worked out pretty well. I wasn't sure that would work for the Coronado: the image is fairly dim. The snaps I did get show Mercury, though none of the interesting detail like faculae and the one tiny prominence that was visible. But the interesting thing is the color. To the eye, the H-alpha scope image is a slightly orangy red, but in the digital camera it came out a startling purplish pink. This may be due to the digital camera's filters passing some IR, confusing the algorithms that decide how to shift the color. Of course, I could have adjusted the color in GIMP back to the real color, but I thought it was more interesting to leave it the hue it came out of the camera. (I did boost contrast and run an unsharp mask filter, to make it easier to see Mercury.)

Anyway, fun and unexpectedly edifying! I wish we had another transit happening sooner than 2032.

Tags: ,
[ 12:15 Nov 11, 2019    More science/astro | permalink to this entry | ]

Fri, 08 Nov 2019

Mercury Transit Next Monday

[Mercury Transit 2006, photo by Brocken Inaglory]
Mercury Transit 2006, photo by Brocken Inaglory
Next Monday, November 11, is a transit of Mercury across the sun.

Mercury transits aren't super rare -- not once- or twice-in-a-lifetime events like Venus transits -- but they're not that common, either. The last Mercury transit was in 2016; the next one won't happen til 2032.

This year's transit isn't ideal for US observers. The transit will already be well underway by the time the sun rises, at least in the western US. Here in New Mexico (Mountain time), the sun rises with Mercury transiting, and the transit lasts until 11:04 MST. Everybody else, check timeanddate's Mercury Transit page for your local times.

Mercury is small, unfortunately, so it's not an easy thing to see without magnification. Of course, you know that you should never look at the sun without an adequate filter. But even if you have safe "eclipse glasses", it may be tough to spot Mercury's small disk against the surface of the sun.

[binocular projection of a solar eclipse] One option is to take some binoculars and use them to project an image. Point the big end of the binoculars at the sun, and the small end at a white surface, preferably leaning so it's perpendicular to the sun. I don't know if binocular projection will give a big enough image to show Mercury, so a very smooth and white background, tilted so it's perpendicular to the sun, will help. (Don't be tempted to stick eclipse glasses in front of a binocular or telescope and look through the eyepiece! Stick to projection unless you have filters specifically intended for telescopes or binoculars.)

Of course, a telescope with a safe solar filter is the best way to see a transit. If you're in the Los Alamos area, I hear the Pajarito Astronomers are planning to set up telescopes at Overlook Park. They don't seem to have announced it in any of the papers yet, but I see it listed on the Pajarito Astronomers website. There's also an event planned at the high school where the students will be trying to time Mercury's passage, but I don't know if that's open to the public. Elsewhere in the world, check with your local astronomy club for Mercury transit parties: I'm sure most clubs have something planned.

I was discussing the transit with a couple of local astronomers earlier this week, and one of them related it to the search for exoplanets. One of the main methods of detecting exoplanets is to measure the dimming of a star's light as a planet crosses its face. For instance, in 55 Cancri e, you can see a dimming as the planet crosses the star's face, and a much more subtle dimming when the planet disappears behind the star. As Mercury crosses the Sun's face, it blocks some of the sun's light in the same way. By how much?

The radius of Mercury is 0.0035068 solar radii, and the dimming is proportional to area so it should be 0.00350682, or 0.0000123, a 0.00123% dimming. Not very much!

But it looks like in the 55 Cancri e case, they're detecting dips of around .001% -- it seems amazing that you could detect a planet as small as Mercury this way (and certainly the planet is much bigger in the case of 55 Cancri e) ... but maybe it's possible.

Anyway, it's fun to think about exoplanets as you watch tiny Mercury make its way across the face of the Sun. Wherever you are, I hope you get a chance to look!

Update: A report from the transit: Mercury Transit: Comparing Between H-alpha and White Light.

Tags: ,
[ 11:36 Nov 08, 2019    More science/astro | permalink to this entry | ]

Sun, 03 Nov 2019

Emulating Raspbian on your Linux x86/amd64 System

I was planning to teach a class on Raspberry Pis, and I wanted to start with the standard Raspbian image, update it, and add some programs like GIMP that we'd use in the class. And I wanted to do that just once, before burning the image to a bunch of different SD cards. Is there a way to do that on my regular Linux box, with its nice fast processor and disk?

Why yes, there is, and it's pretty easy. But there are a lot of unclear or misleading tutorials out there, so I hope this is a bit simpler and easier to follow.

I got most of this from a tutorial that no longer seems to be available (but I'll include the link in case it comes back): Solar Staker/RPI Image/Creation.

I've tested this on Ubuntu 19.10, Debian Stretch and Buster; the instructions should be pretty general except for the name of the loopback mount. Commands you type are in bold; the rest is the output you should see. $ is your normal shell prompt and # is a root prompt.

Required Packages

You'll need kpartx, qemu and some supporting packages. On Debian or Ubuntu:

$ sudo apt install kpartx qemu binfmt-support qemu-user-static

You've probably already downloaded a Raspbian SD card image.

Set Up the Loopback Devices

kpartx can read the Raspbian ISO image and split it into the two filesystems it would have if you wrote it to an SD card. It turns the filesystems into loopback devices you can mount like regular filesystems.

$ sudo kpartx -av 2019-09-26-raspbian-buster-lite.img
add map loop10p1 (253:1): 0 524288 linear 7:10 8192
add map loop10p2 (253:2): 0 3858432 linear 7:10 532480

Make a note of those loopback device names. They may not always be loop10p1 and loop10p2, but they'll probably always end in 1 and 2. 1 is the Raspbian /boot filesystem, 2 is the Raspbian root.

(Optional) Check and Possibly Resize the Raspbian Filesystem

Make sure that the Raspbian filesystem is intact, and that it's a reasonable size.

In practice, I didn't find this made any difference (everything was fine to begin with), but it doesn't hurt to make sure.

$ sudo e2fsck -f /dev/mapper/loop10p2
e2fsck 1.45.3 (14-Jul-2019)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
rootfs: 44367/120720 files (0.3$ non-contiguous), 292014/482304 blocks

$ sudo resize2fs /dev/mapper/loop10p2
resize2fs 1.45.3 (14-Jul-2019)
The filesystem is already 482304 (4k) blocks long.  Nothing to do!

Mount the Loopback Filesystems

You're ready to mount the two filesystems. A Raspbian SD card image contains two filesystems.

Partition 1 is a small vfat /boot filesystem containing the kernel and some other files it needs for booting, plus the two important configuration files cmdline.txt and config.txt.

Partition 2 is the Raspbian root filesystem in ext4 format. The Raspbian root includes an empty /boot directory; mount the root first, then mount the Raspbian boot partition on Raspbian's /boot:

$ sudo mkdir /mnt/pi_image
$ sudo mount /dev/mapper/loop10p2 /mnt/pi_image
$ sudo mount /dev/mapper/loop10p1 /mnt/pi_image/boot

Prepare for Chroot

You're going to chroot to the Raspbian filesystem. Chroot limits the filesystem you can access, so when you type /, instead of your host filesystem's / you'll see the root of the Raspbian filesystem, /mnt/pi_image. That means you won't have access to your host system's /usr/bin, any more.

But qemu needs /usr/bin/qemu-arm-static to be able to emulate ARM binaries in user mode. So copy that to the Raspbian filesystem so you'll still be able to access it after the chroot:

$ sudo cp /usr/bin/qemu-arm-static /mnt/pi_image/usr/bin

Chroot to the Raspbian System

$ sudo chroot /mnt/pi_image /bin/bash
root:/#

Now you're running in Raspbian's root filesystem. All the binaries in your path (e.g. /bin/ls, /bin/bash) are ARM binaries, but if you try to run them, qemu will see qemu-arm-static and run the program as though you're on an actual Raspberry Pi.

Run Stuff!

Now you can run Raspbian commands.

# file /bin/ls
/bin/ls: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=67a394390830ea3ab4e83b5811c66fea9784ee69, stripped
#
# /bin/ls
bin   dev  home  lost+found  mnt  proc	run   srv  tmp	var
boot  etc  lib	 media	     opt  root	sbin  sys  usr

# file /bin/cat
/bin/cat: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=2239a192f2f277bd1a4892e39a41eba97266b91f, stripped
#
# cat /etc/issue
Raspbian GNU/Linux 10 \n \l

You can even install packages or update the whole system:

# apt update
Get:1 http://raspbian.raspberrypi.org/raspbian buster InRelease [15.0 kB]
Get:2 http://archive.raspberrypi.org/debian buster InRelease [25.2 kB]
Get:3 http://raspbian.raspberrypi.org/raspbian buster/main armhf Packages [13.0 MB]
Get:4 http://archive.raspberrypi.org/debian buster/main armhf Packages [259 kB]
Fetched 13.3 MB in 20s (652 kB/s)
Reading package lists... Done

# apt dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following NEW packages will be installed:
  busybox initramfs-tools initramfs-tools-core klibc-utils libklibc linux-base
  pigz
The following packages will be upgraded:
  dhcpcd5 e2fsprogs file firmware-atheros firmware-brcm80211 firmware-libertas
  firmware-misc-nonfree firmware-realtek libcom-err2 libext2fs2 libmagic-mgc
  libmagic1 libraspberrypi-bin libraspberrypi-dev libraspberrypi-doc
  libraspberrypi0 libss2 libssl1.1 libxml2 libxmuu1 openssh-client
  openssh-server openssh-sftp-server openssl raspberrypi-bootloader
  raspberrypi-kernel raspi-config rpi-eeprom rpi-eeprom-images ssh sudo
  wpasupplicant
32 upgraded, 7 newly installed, 0 to remove and 0 not upgraded.
Need to get 133 MB of archives.
After this operation, 3192 kB of additional disk space will be used.
Do you want to continue? [Y/n]

Pretty neat! Although you're not actually running Raspbian, you can run Raspbian executables with the Raspbian root filesystem mounted as though you were actually running on your Raspberry Pi.

Cleaning Up

When you're done with the chroot, just exit that shell (Ctrl-D or exit). If you want to undo everything else afterward:

$ sudo rm /mnt/pi_image/usr/bin/qemu-arm-static

$ sudo umount /mnt/pi_image/boot
$ sudo umount /mnt/pi_image

$ sudo kpartx -dv /dev/loop0
$ sudo losetup -d /dev/loop0

$ sudo rmdir /mnt/pi_image

Limitations

Keep in mind you're not really running Raspbian. You never booted the Raspbian kernel, and you can't test things that depend on Raspbian's init system, like whether networking works, let alone running the Raspbian X desktop or accessing GPIO pins. This is an ARM emulator, not a Raspberry Pi emulator.

More details

If you want to read more about qemu user mode and how it lets you run binaries from other architectures, I recommend these links:

Tags: , , ,
[ 16:14 Nov 03, 2019    More linux | permalink to this entry | ]