Shallow Thoughts : : programming

Fetching Browser Cookies Programmatically

In my eternal quest for a decent RSS feed for top World/National news, I decided to try subscribing to the New York Times online. But when I went to try to add them to my RSS reader, I discovered it wasn't that easy: their login page sometimes gives a captcha, so you can't just set a username and password in the RSS reader.

A common technique for sites like this is to log in with a browser, then copy the browser's cookies into your news reading program. At least, I thought it was a common technique -- but when I tried a web search, examples were surprisingly hard to find.

None of the techniques to examine or save browser cookies were all that simple, so I ended up writing a browser_cookies.py Python script to extract cookies from chromium and firefox browsers.

Read more ...

Tags: web, programming, python, cookies, privacy
[ 11:19 Mar 30, 2021    More programming | permalink to this entry | ]

Android Studio Workarounds, and Command-Line Gradle Builds

I got a new phone. (Not something that happens often.)

Fun, right? Well, partly, but also something I'd been dreading. I had a feeling that my ancient RSS reader, FeedViewer, which I use daily to read all my news feeds, probably wouldn't work under a modern Android (I wrote it for KitKat and it was last updated under Marshmallow). And that was correct.

Read more ...

Tags: programming, android, java
[ 17:49 Dec 19, 2020    More programming | permalink to this entry | ]

Pho 1.0, Belated Release

I was doing some disk housekeeping and noticed that my venerable image viewer, Pho, was at version 1.0pre1, and had been since 2017. It's had only very minimal changes since that time. I guess maybe it's been long enough that it's time to remove that -pre1 moniker, huh?

Of course I couldn't leave it at that. There were a couple of very minor bugs I'd been ignoring, when you delete from the end or beginning of the image list. So I fixed those, bumped the version, updated the web page, tagged the git tree and made a release. Pho is now 1.0. About time!

Tags: programming, pho, image viewer
[ 10:32 Sep 26, 2020    More programming | permalink to this entry | ]

Styling GTK3 in Python with CSS

Lately I've been running with my default python set to Python 3. Debian still uses Python 2 as the default, which is reasonable, but adding a ~/bin/python symlink to /usr/bin/python3 helps me preview scripts that might become a problem once Debian does switch. I thought I had converted most of my Python scripts to Python 3 already, but this link is catching some I didn't convert.

Python has a nice script called 2to3 that can convert the bulk of most scripts with little fanfare. The biggest hassles that 2to3 can't handle are network related (urllib and urllib2) and, the big one, user interfaces. PyGTK, based on GTK2 has no Python 3 equivalent; in Python 3, the only option is to use GObject Introspection (gi) and GTK3. Since there's almost no documentation on python-gi and gtk3, converting a GTK script always involves a lot of fumbling and guesswork.

A few days ago I tried to play an MP3 in my little musicplayer.py script and discovered I'd never updated it. I have enough gi/GTK3 scripts by now that I thought something with such a simple user interface would be easy. Shows how much I know about GTK3!

I got the basic window ported pretty easily, but it looked terrible: huge margins everywhere, and no styling on the text, like the bold, large-sized text I had previously use to highlight the name of the currently playing song. I tried various approaches, but a lot of the old methods of styling have been deprecated in GTK3; you're supposed to use CSS. Except, of course, there's no documentation on it, and it turns out the CSS accepted by GTK3 is a tiny subset of the CSS you can use in HTML pages, but what the subset is doesn't seem to be documented anywhere.

How to Apply a Stylesheet

The first task was to get any CSS at all working. The GNOME Journal: Styling GTK with CSS was helpful in getting started, but had a lot of information that doesn't work (perhaps it did once). At least it gave me this basic snippet:

    css = '* { background-color: #f00; }'
    css_provider = gtk.CssProvider()
    css_provider.load_from_data(css)
    context = gtk.StyleContext()
    screen = Gdk.Screen.get_default()
    context.add_provider_for_screen(screen, css_provider,
                                    gtk.STYLE_PROVIDER_PRIORITY_APPLICATION)

Built-in Class Names

Great! if all you want to do is turn the whole app red. But in reality, you'll want to style different widgets differently. At least some classes have class names:

    css = 'button { background-color: #f00; }'

I found other pages suggesting using 'GtkButton in CSS, but that didn't work for me. How do you find the right class names? No idea, I never found a reference for that. Just guess, I guess.

User-set Class Names

What about classes -- for instance, make all the buttons in a ButtonBox white? You can add classes this way:

    button_context = button.get_style_context()
    button_context.add_class("whitebutton")

If you need to change a class (for instance, turn a red button green), first remove the old class:

    button_context = button.get_style_context()
    entry_style_context.remove_class("red")

Widget Names, like CSS ID

For single widgets, you can give the widget a name and use it like an ID in CSS. Like this:

    label = gtk.Label()
    label.set_use_markup(True)
    label.set_line_wrap(True)
    label.set_name("red_label")
    mainbox.pack_start(label, False, False, 0)
    css = '#red_label { background-color: #f00; }'
[ ... ]

Properties You Can Set

There is, amazingly, a page on which CSS properties GTK3 supports. That page doesn't mention it, but some properties like :hover are also supported. So you can write CSS tweaks like

.button { border-radius: 15; border-width: 2; border-style: outset; }
.button:hover { background: #dff; border-color: #8bb; }

And descendants work, so you can say somthing like

    buttonbox = gtk.ButtonBox(spacing=4)
    buttonbox.set_name("buttonbox")
    mainbox.pack_end(buttonbox, False, False, 0)

    btn = gtk.Button(label="A")
    buttonbox.add(btn)

    btn = gtk.Button(label="B")
    buttonbox.add(btn)

and then use CSS that affects all the buttons inside the buttonbox:

#buttonbox button { color: red; }

No mixed CSS Inside Labels

My biggest disappointment was that I couldn't mix styles inside a label. You can't do something like

label.set_label('Headline'
                'Normal text')

and expect to style the different parts separately. You can use very simple markup like <b>bold</b> normal, but anything further gives errors like "error parsing markup: Attribute 'class' is not allowed on the <span> tag" (you'll get the same error if you try "id"). I had to make separate GtkLabels for each text size and style I wanted, which is a lot more work. If you wanted to mix styles and have them reflow as the content length changed, I don't know how (or if) you could do it.

Fortunately, I don't strictly need that for this little app. So for now, I'm happy to have gotten this much working.

Tags: programming, python, gtk, gtk3
[ 14:49 Jun 05, 2019    More programming | permalink to this entry | ]

Plotting a Sequence of Graphs in Matplotlib 3D

A friend and I were talking about temperature curves: specifically, the way the temperature sinks in the evening but then frequently rises again before it really starts cooling off.

I thought it would be fun to plot the curve of temperature as a function of time over successive days, as a 3-D plot. I knew matplotlib had a way to do 3D plots, but I've never actually generated one.

Well, it turns out there are lots of examples, but they all start by generating mysterious data blobs, and none of them explain the structure of the data they're using, and the documentation has mysterious parameters like "zs" that aren't explained anywhere. So getting something that worked was a fiddly process. Creating a color version, to distinguish the graphs better, was even more fiddly.

So I wrote an example that I hope will make it a little clearer for anyone trying to use this library. It can plot using just lines:

... or it can plot in color, cycling colors manually because by default matplotlib makes adjacent colors similar, exactly the opposite of what you'd want:

Here's the demo: multiplot3d.py on GitHub.

... Except there's a Bug

All is not perfect. Axes3D gets a bit confused sometimes about which layer is supposed to be in front of which other layer. You can see that on the two plots: in both cases, the fourth and fifth layers from the front are reversed, so the fifth layer is drawn in front of the fourth layer. I haven't yet found anyone in the matplotlib organization who seems to know much about Axes3D; eventually I'll file a bug but I want to write a shorter, clearer test case to illustrate the problem. Still, even with the bugs it's a useful technique to know.

Tags: python, programming, data, matplotlib
[ 09:57 May 30, 2019    More programming | permalink to this entry | ]

Python: Find Your System's Biggest CPU Hogs

My machine has recently developed an overheating problem. I haven't found a solution for that yet -- you'd think Linux would have a way to automatically kill or suspend processes based on CPU temperature, but apparently not -- but my investigations led me down one interesting road: how to write a Python script that finds CPU hogs.

The psutil module can get a list of processes with psutil.process_iter(), which returns Process objects that have a cpu_percent() call. Great! Except it always returns 0.0, even for known hogs like Firefox, or if you start up a VLC and make it play video scaled to the monitor size.

That's because cpu_percent() needs to run twice, with an interval in between: it records the elapsed run time and sees how much it changes. You can pass an interval to cpu_percent() (the units aren't documented, but apparently they're seconds). But if you're calling it on more than one process -- as you usually will be -- it's better not to wait for each process. You have to wait at least a quarter of a second to get useful numbers, and longer is better. If you do that for every process on the system, you'll be waiting a long time.

Instead, use cpu_percent() in non-blocking mode. Pass None as the interval (or leave it blank since None is the default), then loop over the process list and call proc.cpu_percent(None) on each process, throwing away the results the first time. Then sleep for a while and repeat the loop: the second time, cpu_percent() will give you useful numbers.

def hoglist(delay=5):
    '''Return a list of processes using a nonzero CPU percentage
       during the interval specified by delay (seconds),
       sorted so the biggest hog is first.
    '''
    proccesses = list(psutil.process_iter())
    for proc in proccesses:
        proc.cpu_percent(None)    # non-blocking; throw away first bogus value

    print("Sleeping ...")
    sys.stdout.flush()
    time.sleep(delay)
    print()

    procs = []
    for proc in proccesses:
        percent = proc.cpu_percent(None)
        if percent:
            procs.append((proc.name(), percent))

    print(procs)
    procs.sort(key=lambda x: x[1], reverse=True)
    return procs

if __name__ == '__main__':
    prohogscs = hoglist()
    for p in hogs:
        print("%20s: %5.2f" % p)

It's a useful trick. Though actually applying this to a daemon that responds to temperature, to solve my overheating problem, is more complicated. For one thing, you need rules about special processes. If your Firefox goes wonky and starts making your X server take lots of CPU time, you want to suspend Firefox, not the X server.

Tags: programming, python, linux
[ 15:54 Jan 18, 2019    More programming | permalink to this entry | ]

Writing Solar System Simulations with NAIF SPICE and SpiceyPy

Someone asked me about my Javascript Jupiter code, and whether it used PyEphem. It doesn't, of course, because it's Javascript, not Python (I wish there was something as easy as PyEphem for Javascript!); instead it uses code from the book Astronomical Formulae for Calculators by Jean Meeus. (His better known Astronomical Algorithms, intended for computers rather than calculators, is actually harder to use for programming because Astronomical Algorithms is written for BASIC and the algorithms are relatively hard to translate into other languages, whereas Astronomical Formulae for Calculators concentrates on explaining the algorithms clearly, so you can punch them into a calculator by hand, and this ends up making it fairly easy to implement them in a modern computer language as well.)

Anyway, the person asking also mentioned JPL's page HORIZONS Ephemerides page, which I've certainly found useful at times. Years ago, I tried emailing the site maintainer asking if they might consider releasing the code as open source; it seemed like a reasonable request, given that it came from a government agency and didn't involve anything secret. But I never got an answer.

But going to that page today, I find that code is now available! What's available is a massive toolkit called SPICE (it's all in capitals but there's no indication what it might stand for. It comes from NAIF, which is NASA's Navigation and Ancillary Information Facility).

SPICE allows for accurate calculations of all sorts of solar system quantities, from the basic solar system bodies like planets to all of NASA's active and historical public missions. It has bindings for quite a few languages, including C. The official list doesn't include Python, but there's a third-party Python wrapper called SpiceyPy that works fine.

The tricky part of programming with SPICE is that most of the code is hidden away in "kernels" that are specific to the objects and quantities you're calculating. For any given program you'll probably need to download at least four "kernels", maybe more. That wouldn't be a problem except that there's not much help for figuring out which kernels you need and then finding them. There are lots of SPICE examples online but few of them tell you which kernels they need, let alone where to find them.

After wrestling with some of the examples, I learned some tricks for finding kernels, at least enough to get the basic examples working. I've collected what I've learned so far into a new GitHub repository: NAIF SPICE Examples. The README there explains what I know so far about getting kernels; as I learn more, I'll update it.

SPICE isn't easy to use, but it's probably much more accurate than simpler code like PyEphem or my Meeus-based Javascript code, and it can calculate so many more objects. It's definitely something worth knowing about for anyone doing solar system simulations.

Tags: programming, python, astronomy
[ 16:43 Sep 23, 2018    More programming | permalink to this entry | ]

Google Maps API No Longer Free?

A while ago I wrote an interactive trail map page for the PEEC nature center website. At the time, I wanted to use an open library, like OpenLayers or Leaflet; but there were no good sources of satellite/aerial map tiles at the time. The only one I found didn't work because they had a big blank area anywhere near LANL -- maybe because of the restricted airspace around the Lab. Anyway, I figured people would want a satellite option, so I used Google Maps instead despite its much more frustrating API.

This week we've been working on converting the website to https. Most things went surprisingly smoothly (though we had a lot more absolute URLs in our pages and databases than we'd realized). But when we got through, I discovered the trail map was broken. I'm still not clear why, but somehow the change from http to https made Google's API stop working. In trying to fix the problem, I discovered that Google's map API may soon cease to be free:

New pricing and product changes will go into effect starting June 11, 2018. For more information, check out the Guide for Existing Users.

That has a button for "Transition Tool" which, when you click it, won't tell you anything about the new pricing structure until you've already set up a billing account. Um ... no thanks, Google.

Googling for google maps api billing led to a page headed "Pricing that scales to fit your needs", which has an elaborate pricing structure listing a whole bnch of variants (I have no idea which of these I was using), of which the first $200/month is free. But since they insist on setting up a billing account, I'd probably have to give them a credit card number -- which one? My personal credit card, for a page that isn't even on my site? Does the nonprofit nature center even have a credit card? How many of these API calls is their site likely to get in a month, and what are the chances of going over the limit?

It all rubbed me the wrong way, especially when the context of "Your trail maps page that real people actually use has broken without warning, and will be held hostage until you give usa credit card number". This is what one gets for using a supposedly free (as in beer) library that's not Free open source software.

So I replaced Google with the excellent open source Leaflet library, which, as a bonus, has much better documentation than Google Maps. (It's not that Google's documentation is poorly written; it's that they keep changing their APIs, but there's no way to tell the dozen or so different APIs apart because they're all just called "Maps", so when you search for documentation you're almost guaranteed to get something that stopped working six years ago -- but the documentation is still there making it look like it's still valid.) And I was happy to discover that, in the time since I originally set up the trailmap page, some open providers of aerial/satellite map tiles have appeared. So we can use open source and have a satellite view.

Our trail map is back online with Leaflet, and with any luck, this time it will keep working. PEEC Los Alamos Area Trail Map.

Tags: mapping, web, programming, javascript
[ 16:13 May 24, 2018    More programming | permalink to this entry | ]