Shallow Thoughts : : linux

Akkana's Musings on Open Source Computing and Technology, Science, and Nature.

Sun, 26 Jun 2016

How to un-deny a host blocked by denyhosts

We had a little crisis Friday when our server suddenly stopped accepting ssh connections.

The problem turned out to be denyhosts, a program that looks for things like failed login attempts and blacklists IP addresses.

But why was our own IP blacklisted? It was apparently because I'd been experimenting with a program called mailsync, which used to be a useful program for synchronizing IMAP folders with local mail folders. But at least on Debian, it has broken in a fairly serious way, so that it makes three or four tries with the wrong password before it actually uses the right one that you've configured in .mailsync. These failed logins are a good way to get yourself blacklisted, and there doesn't seem to be any way to fix mailsync or the c-client library it uses under the covers.

Okay, so first, stop using mailsync. But then how to get our IP off the server's blacklist? Just editing /etc/hosts.deny didn't do it -- the IP reappeared there a few minutes later.

A web search found lots of solutions -- you have to edit a long list of files, but no two articles had the same file list. It appears that it's safest to remove the IP from every file in /var/lib/denyhosts.

So here are the step by step instructions.

First, shut off the denyhosts service:

service denyhosts stop

Go to /var/lib/denyhosts/ and grep for any file that includes your IP:

grep *

(If you aren't sure what your IP is as far as the outside world is concerned, Googling what's my IP will helpfully tell you, as well as giving you a list of other sites that will also tell you.)

Then edit each of these files in turn, removing your IP from them (it will probably be at the end of the file).

When you're done with that, you have one more file to edit: remove your IP from the end of /etc/hosts.deny

You may also want to add your IP to /etc/hosts.allow, but it may not make much difference, and if you're on a dynamic IP it might be a bad idea since that IP will eventually be used by someone else.

Finally, you're ready to re-start denyhosts:

service denyhosts stop

Whew, un-blocked. And stay away from mailsync. I wish I knew of a program that actually worked to keep IMAP and mbox mailboxes in sync.

[ 12:59 Jun 26, 2016    More linux | permalink to this entry | comments ]

Thu, 09 Jun 2016

Visual diffs and file merges with vimdiff

I needed to merge some changes from a development file into the file on the real website, and discovered that the program I most often use for that, meld, is in one of its all too frequent periods where its developers break it in ways that make it unusable for a few months. (Some of this is related to GTK, which is a whole separate rant.)

That led me to explore some other diff/merge alternatives. I've used tkdiff quite a bit for viewing diffs, but when I tried to use it to merge one file into another I found its merge just too hard to use. Likewise for emacs: it's a wonderful editor but I never did figure out how to get ediff to show diffs reliably, let alone merge from one file to another.

But vimdiff looked a lot easier and had a lot more documentation available, and actually works pretty well.

I normally run vim in an xterm window, but for a diff/merge tool, I want a very wide window which will show the diffs side by side. So I used gvimdiff instead of regular vimdiff: gvimdiff docs.production/filename

Configuring gvimdiff to see diffs

gvimdiff initially pops up a tiny little window, and it ignores Xdefaults. Of course you can resize it, but who wants to do that every time? You can control the initial size by setting the lines and columns variables in .vimrc. About 180 columns by 60 lines worked pretty well for my fonts on my monitor, showing two 80-column files side by side. But clearly I don't want to set that in .vimrc so that it runs every time I run vim; I only want that super-wide size when I'm running a side-by-side diff.

You can control that by checking the &diff variable in .vimrc:

if &diff
    set lines=58
    set columns=180

If you do decide to resize the window, you'll notice that the separator between the two files doesn't stay in the center: it gives you lots of space for the right file and hardly any for the left. Inside that same &diff clause, this somewhat arcane incantation tells vim to keep the separator centered:

    autocmd VimResized * exec "normal \<C-w>="

I also found that the colors, in the vim scheme I was using, made it impossible to see highlighted text. You can go in and edit the color scheme and make your own, of course, but an easy way quick fix is to set all highlighting to one color, like yellow, inside the if $diff section:

    highlight DiffAdd    cterm=bold gui=none guibg=Yellow
    highlight DiffDelete cterm=bold gui=none guibg=Yellow
    highlight DiffChange cterm=bold gui=none guibg=Yellow
    highlight DiffText   cterm=bold gui=none guibg=Yellow

Merging changes

Okay, once you can view the differences between the two files, how do you merge from one to the other? Most online sources are quite vague on that, but it's actually fairly easy:
]c jumps to the next difference
[c jumps to the previous difference
dp makes them both look like the left side (apparently stands for diff put
do makes them both look like the right side (apparently stands for diff obtain

The only difficult part is that it's not really undoable. u (the normal vim undo keystroke) works inconsistently after dp: the focus is generally in the left window, so u applies to that window, while dp modified the right window and the undo doesn't apply there. If you put this in your .vimrc

nmap du :wincmd w<cr>:normal u<cr>:wincmd w<cr>
then you can use du to undo changes in the right window, while u still undoes in the left window. So you still have to keep track of which direction your changes are going.

Worse, neither undo nor this du command restores the highlighting showing there's a difference between the two files. So, really, undoing should be reserved for emergencies; if you try to rely on it much you'll end up being unsure what has and hasn't changed.

In the end, vimdiff probably works best for straightforward diffs, and it's probably best get in the habit of always merging from right to left, using do. In other words, run vimdiff file-to-merge-to file-to-merge-from, and think about each change before doing it to make it less likely that you'll need to undo.

And hope that whatever silly transient bug in meld drove you to use vimdiff gets fixed quickly.

Tags: , ,
[ 20:10 Jun 09, 2016    More linux/editors | permalink to this entry | comments ]

Sat, 07 May 2016

Setting "Emacs" key theme in gtk3 (and Firefox 46)

I recently let Firefox upgrade itself to 46.0.1, and suddenly I couldn't type anything any more. The emacs/readline editing bindings, which I use probably thousands of times a day, no longer worked. So every time I typed a Ctrl-H to delete the previous character, or Ctrl-B to move back one character, a sidebar popped up. When I typed Ctrl-W to delete the last word, it closed the tab. Ctrl-U, to erase the contents of the urlbar, opened a new View Source tab, while Ctrl-N, to go to the next line, opened a new window. Argh!

(I know that people who don't use these bindings are rolling their eyes and wondering "What's the big deal?" But if you're a touch typist, once you've gotten used to being able to edit text without moving your hands from the home position, it's hard to imagine why everyone else seems content with key bindings that require you to move your hands and eyes way over to keys like Backspace or Home/End that aren't even in the same position on every keyboard. I map CapsLock to Ctrl for the same reason, since my hands are too small to hit the PC-positioned Ctrl key without moving my whole hand. Ctrl was to the left of the "A" key on nearly all computer keyboards until IBM's 1986 "101 Enhanced Keyboard", and it made a lot more sense than IBM's redesign since few people use Caps Lock very often.)

I found a bug filed on the broken bindings, and lots of people commenting online, but it wasn't until I found out that Firefox 46 had switched to GTK3 that I understood had actually happened. And adding gtk3 to my web searches finally put me on the track to finding the solution, after trying several other supposed fixes that weren't.

Here's what actually worked: edit ~/.config/gtk-3.0/settings.ini and add, inside the [Settings] section, this line:

gtk-key-theme-name = Emacs

I think that's all that was needed. But in case that doesn't do it, here's something I had already tried, unsuccessfully, and it's possible that you actually need it in addition to the settings.ini change (I don't know how to undo magic Gnome settings so I can't test it):

gsettings set org.gnome.desktop.interface gtk-key-theme "Emacs"

Tags: , , , ,
[ 18:11 May 07, 2016    More linux | permalink to this entry | comments ]

Sat, 26 Mar 2016

Debian: Holding packages you build from source, and rebuilding them easily

Recently I wrote about building the Debian hexchat package to correct a key binding bug.

I built my own version of the hexchat packages, then installed the ones I needed:

dpkg -i hexchat_2.10.2-1_i386.deb hexchat-common_2.10.2-1_all.deb hexchat-python_2.10.2-1_i386.deb hexchat-perl_2.10.2-1_i386.deb

That's fine, but of course, a few days later Debian had an update to the hexchat package that wiped out my changes.

The solution to that is to hold the packages so they won't be overwritten on the next apt-get upgrade:

aptitude hold hexchat hexchat-common hexchat-perl hexchat-python

If you forget which packages you've held, you can find out with aptitude:

aptitude search '~ahold'

Simplifying the rebuilding process

But now I wanted an easier way to build the package. I didn't want to have to search for my old blog post and paste the lines one by one every time there was an update -- then I'd get lazy and never update the package, and I'd never get security fixes.

I solved that with a zsh function:

newhexchat() {
    # Can't set errreturn yet, because that will cause mv and rm
    # (even with -f) to exit if there's nothing to remove.
    cd ~/outsrc/hexchat
    echo "Removing what was in old previously"
    rm -rf old
    echo "Moving everything here to old/"
    mkdir old
    mv *.* old/

    # Make sure this exits on errors from here on!
    setopt localoptions errreturn

    echo "Getting source ..."
    apt-get source hexchat
    cd hexchat-2*
    echo "Patching ..."
    patch -p0 < ~/outsrc/hexchat-2.10.2.patch
    echo "Building ..."
    debuild -b -uc -us
    echo 'Installing' ../hexchat{,-python,-perl}_2*.deb
    sudo dpkg -i ../hexchat{,-python,-perl}_2*.deb

Now I can type newhexchat and pull a new version of the source, build it, and install the new packages.

How do you know if you need to rebuild?

One more thing. How can I find out when there's a new version of hexchat, so I know I need to build new source in case there's a security fix?

One way is the Debian Package Tracking System. You can subscribe to a package and get emails when a new version is released. There's supposed to be a package tracker web interface, e.g. package tracker: hexchat with a form you can fill out to subscribe to updates -- but for some packages, including hexchat, there's no form. Clicking on the link for the new package tracker goes to a similar page that also doesn't have a form.

So I guess the only option is to subscribe by email. Send mail to containing this line:

subscribe hexchat [your-email-address]
You'll get a reply asking for confirmation.

This may turn out to generate too much mail: I've only just subscribed, so I don't know yet. There are supposedly keywords you can use to limit the subscription, such as upload-binary and upload-source, but the instructions aren't at all clear on how to include them in your subscription mail -- you say keyword, or keyword your-email, so where do you put the actual keywords you want to accept? They offer no examples.

Use apt to check whether your version is current

If you can't get the email interface to work or suspect it'll be too much email, you can use apt to check whether the current version in the repository is higher than the one you're running:

apt-cache policy hexchat

You might want to automate that, to make it easy to check on every package you've held to see if there's a new version. Here's a little shell function to do that:

# Check on status of all held packages:
check_holds() {
    for pkg in $( aptitude search '~ahold' | awk '{print $2}' ); do
        policy=$(apt-cache policy $pkg)
        installed=$(echo $policy | grep Installed: | awk '{print $2}' )
        candidate=$(echo $policy | grep Candidate: | awk '{print $2}' )
        if [[ "$installed" == "$candidate" ]]; then
            echo $pkg : nothing new
            echo $pkg : new version $candidate available

Tags: , , ,
[ 11:11 Mar 26, 2016    More linux | permalink to this entry | comments ]

Thu, 17 Mar 2016

Changing X brightness and gamma with xrandr

I switched a few weeks ago from unstable ("Sid") to testing ("Stretch") in the hope that my system, particularly X, would break less often. The very next day, I updated and discovered I couldn't use my system at night any more, because the program I use to reduce the screen brightness by tweaking X gamma no longer worked. Neither did other related programs, such as xgamma and xcalib.

The Dell monitor I use doesn't have reasonable hardware brightness controls: strangely, the brightness button works when the monitor is connected over VGA, but if I want to use the sharper HDMI connection, brightness adjustment no longer works. So I depend on software brightness adjustment in order to use my computer at night when the room is dim.

Fortunately, it turns out there's a workaround. xrandr has options for both brightness and gamma:

xrandr --output HDMI1 --brightness .5
xrandr --output HDMI1 --gamma .5:.5:.5

I've always put xbrightness on a key, so I can use a function key to adjust brightness interactively up and down according to conditions. So a command that sets brightness to .5 or .8 isn't what I need; I need to get the current brightness and set it a little brighter or a little dimmer. xrandr doesn't offer that, so I needed to script it.

You can get the current brightness with

xrandr --verbose | grep -i brightness

But I was hoping there would be a more straightforward way to get brightness from a program. I looked into Python bindings for xrandr; there are some, but with no documentation and no examples. After an hour of fiddling around, I concluded that I could waste the rest of the day poring through the source code and trying things hoping something would work; or I could spend fifteen minutes using to wrap the command-line xrandr.

So subprocesses it was. It made for a nice short script, much simpler than the old xbrightness C program that used <X11/extensions/xf86vmode.h> and XF86VidModeGetGammaRampSize(): xbright on github.

Tags: , , ,
[ 11:01 Mar 17, 2016    More linux | permalink to this entry | comments ]

Wed, 24 Feb 2016

Migrating from xchat: a couple of hexchat fixes

I decided recently to clean up my Debian "Sid" system, using apt-get autoclean, apt-get purge `deborphan`, aptitude purge ~c, and aptitude purge ~o. It gained me almost two gigabytes of space. On the other hand, it deleted several packages I had long depended on. One of them was xchat.

I installed hexchat, the fully open replacement for xchat. Mostly, it's the same program ... but a few things didn't work right.

Script fixes

The two xchat scripts I use weren't loading. Turns out hexchat wants to find its scripts in .config/hexchat/addons, so I moved them there. But still didn't work; it was looking for a widget called "xchat-inputbox". That was fairly easy to patch: I added a line to print the name of each widget it saw, determined the name had changed in the obvious way, and changed

    if( $child->get( "name" ) eq 'xchat-inputbox' ) {
    if( $child->get( "name" ) eq 'xchat-inputbox' ||
        $child->get( "name" ) eq 'hexchat-inputbox' ) {
That solved the problem.

Notifying me if someone calls me

The next problem: when someone mentioned my nick in a channel, the channel tab highlighted; but when I switched to the channel, there was no highlight on the actual line of conversation so I could find out who was talking to me. (It was turning the nick of the person addressing me to a specific color, but since every nick is a different color anyway, that doesn't make the line stand out when you're scanning for it.)

The highlighting for message lines is set in a dialog you can configure: Settings→Text events...
Scroll down to Channel Msg Hilight and click on that elaborate code on the right: %C2<%C8%B$1%B%C2>%O$t$2%O
That's the code that controls how the line will be displayed.

Some of these codes are described in Hexchat: Appearance/Theming, and most of the rest are described in the dialog itself. $t is an exception: I'm not sure what it means (maybe I just missed it in the list).

I wanted hexchat to show the nick of whoever called me name in inverse video. (Xchat always made it bold, but sometimes that's subtle; inverse video would be a lot easier to find when scrolling through a busy channel.) %R is reverse video, %B is bold, and %O removes any decorations and sets the text back to normal text, so I set the code to: %R%B<$1>%O $t$2 That seemed to work, though after I exited hexchat and started it up the next morning it had magically changed to %R%B<$1>%O$t$2%O.

Hacking hexchat source to remove hardwired keybindings

But the big problem was the hardwired keybindings. In particular, Ctrl-F -- the longstanding key sequence that moves forward one character -- in hexchat, it brings up a search window. (Xchat had this problem for a little while, many years ago, but they fixed it, or at least made it sensitive to whether the GTK key theme is "Emacs".)

Ctrl-F doesn't appear in the list under Settings→Keyboard shortcuts, so I couldn't fix it that way. I guess they should rename that dialog to Some keyboard shortcuts. Turns out Ctrl-F is compiled in. So the only solution is to rebuild from source.

I decided to use the Debian package source:

apt-get source hexchat

The search for the Ctrl-F binding turned out to be harder than it had been back in the xchat days. I was confident the binding would be in one of the files in src/fe-gtk, but grepping for key, find and search all gave way too many hits. Combining them was the key:

egrep -i 'find|search' *.c | grep -i key

That gave a bunch of spurious hits in fkeys.c -- I had already examined that file and determined that it had to do with the Settings→Keyboard shortcuts dialog, not the compiled-in key bindings. But it also gave some lines from menu.c including the one I needed:

    {N_("Search Text..."), menu_search, GTK_STOCK_FIND, M_MENUSTOCK, 0, 0, 1, GDK_KEY_f},

Inspection of nearby lines showed that the last GDK_KEY_ argument is optional -- there were quite a few lines that didn't have a key binding specified. So all I needed to do was remove that GDK_KEY_f. Here's my patch:

--- src/fe-gtk/menu.c.orig      2016-02-23 12:13:55.910549105 -0700
+++ src/fe-gtk/menu.c   2016-02-23 12:07:21.670540110 -0700
@@ -1829,7 +1829,7 @@
        {N_("Save Text..."), menu_savebuffer, GTK_STOCK_SAVE, M_MENUSTOCK, 0, 0,
 #define SEARCH_OFFSET (70)
        {N_("Search"), 0, GTK_STOCK_JUSTIFY_LEFT, M_MENUSUB, 0, 0, 1},
-               {N_("Search Text..."), menu_search, GTK_STOCK_FIND, M_MENUSTOCK,
 0, 0, 1, GDK_KEY_f},
+               {N_("Search Text..."), menu_search, GTK_STOCK_FIND, M_MENUSTOCK,
 0, 0, 1},
                {N_("Search Next"   ), menu_search_next, GTK_STOCK_FIND, M_MENUS
TOCK, 0, 0, 1, GDK_KEY_g},
                {N_("Search Previous"   ), menu_search_prev, GTK_STOCK_FIND, M_M
ENUSTOCK, 0, 0, 1, GDK_KEY_G},
                {0, 0, 0, M_END, 0, 0, 0},

After making that change, I rebuilt the hexchat package and installed it:

sudo apt-get build-dep hexchat
sudo apt-get install devscripts
cd hexchat-2.10.2/
debuild -b -uc -us
sudo dpkg -i ../hexchat_2.10.2-1_i386.deb

Update: I later wrote about how to automate this here: Debian: Holding packages you build from source, and rebuilding them easily.

And the hardwired Ctrl-F key binding was gone, and the normal forward-character binding from my GTK key theme took over.

I still have a couple of minor things I'd like to fix, like the too-large font hexchat uses for its channel tabs, but those are minor. At least I'm back to where I was before foolishly deciding to clean up my system.

Tags: , ,
[ 19:00 Feb 24, 2016    More linux | permalink to this entry | comments ]

Fri, 05 Feb 2016

Updating Debian under a chroot

Debian's Unstable ("Sid") distribution has been terrible lately. They're switching to a version of X that doesn't require root, and apparently the X transition has broken all sorts of things in ways that are hard to fix and there's no ETA for when things might get any better.

And, being Debian, there's no real bug system so you can't just CC yourself on the bug to see when new fixes might be available to try. You just have to wait, try every few days and see if the system

That's hard when the system doesn't work at all. Last week, I was booting into a shell but X wouldn't run, so at least I could pull updates. This week, X starts but the keyboard and mouse don't work at all, making it hard to run an upgrade. has been fixed.

Fortunately, I have an install of Debian stable ("Jessie") on this system as well. When I partition a large disk I always reserve several root partitions so I can try out other Linux distros, and when running the more experimental versions, like Sid, sometimes that's a life saver. So I've been running Jessie while I wait for Sid to get fixed. The only trick is: how can I upgrade my Sid partition while running Jessie, since Sid isn't usable at all?

I have an entry in /etc/fstab that lets me mount my Sid partition easily:

/dev/sda6 /sid ext4 defaults,user,noauto,exec 0 0
So I can type mount /sid as myself, without even needing to be root.

But Debian's apt upgrade tools assume everything will be on /, not on /sid. So I'll need to use chroot /sid (as root) to change the root of the filesystem to /sid. That only affects the shell where I type that command; the rest of my system will still be happily running Jessie.

Mount the special filesystems

That mostly works, but not quite, because I get a lot of errors like permission denied: /dev/null.

/dev/null is a device: you can write to it and the bytes disappear, as if into a black hole except without Hawking radiation. Since /dev is implemented by the kernel and udev, in the chroot it's just an empty directory. And if a program opens /dev/null in the chroot, it might create a regular file there and actually write to it. You wouldn't want that: it eats up disk space and can slow things down a lot.

The way to fix that is before you chroot: mount --bind /dev /sid/dev which will make /sid/dev a mirror of the real /dev. It has to be done before the chroot because inside the chroot, you no longer have access to the running system's /dev.

But there is a different syntax you can use after chrooting:

mount -t proc proc proc/
mount --rbind /sys sys/
mount --rbind /dev dev/

It's a good idea to do this for /proc and /sys as well, and Debian recommends adding /dev/pts (which must be done after you've mounted /dev), even though most of these probably won't come into play during your upgrade.

Mount /boot

Finally, on my multi-boot system, I have one shared /boot partition with kernels for Jessie, Sid and any other distros I have installed on this system. (That's somewhat hard to do using grub2 but easy on Debian though you may need to turn off auto-update and Debian is making it harder to use extlinux now.) Anyway, if you have a separate /boot partition, you'll want it mounted in the chroot, in case the update needs to add a new kernel. Since you presumably already have the same /boot mounted on the running system, use mount --bind for that as well.

So here's the final set of commands to run, as root:

mount /sid
mount --bind /proc /sid/proc
mount --bind /sys /sid/sys
mount --bind /dev /sid/dev
mount --bind /dev/pts /sid/dev/pts
mount --bind /boot /sid/boot
chroot /sid

And then you can proceed with your apt-get update, apt-get dist-upgrade etc. When you're finished, you can unmount everything with one command:

umount --recursive /sid

Some helpful background reading:

Tags: , , ,
[ 11:43 Feb 05, 2016    More linux/install | permalink to this entry | comments ]

Sun, 31 Jan 2016

Setting mouse speed in X

My mouse died recently: the middle button started bouncing, so a middle button click would show up as two clicks instead of one. What a piece of junk -- I only bought that Logitech some ten years ago! (Seriously, I'm pretty amazed how long it lasted, considering it wasn't anything fancy.)

I replaced it with another Logitech, which turned out to be quite difficult to find. Turns out most stores only sell cordless mice these days. Why would I want something that depends on batteries to use every day at my desktop?

But I finally found another basic corded Logitech mouse (at Office Depot). Brought it home and it worked fine, except that the speed was way too fast, much faster than my old mouse. So I needed to find out how to change mouse speed.

X11 has traditionally made it easy to change mouse acceleration, but that wasn't what I wanted. I like my mouse to be fairly linear, not slow to start then suddenly zippy. There's no X11 property for mouse speed; it turns out that to set mouse speed, you need to call it Deceleration.

But first, you need to get the ID for your mouse.

$ xinput list| grep -i mouse
⎜   ↳ Logitech USB Optical Mouse                id=11   [slave  pointer  (2)]

Armed with the ID of 11, we can find the current speed (deceleration) and its ID:

$ xinput list-props 11 | grep Deceleration
        Device Accel Constant Deceleration (259):       3.500000
        Device Accel Adaptive Deceleration (260):       1.000000

Constant deceleration is what I want to set, so I'll use that ID of 259 and set the new deceleration to 2:

$ xinput set-prop 11 259 2

That's fine for doing it once. But what if you want it to happen automatically when you start X? Those constants might all stay the same, but what if they don't?

So let's build a shell pipeline that should work even if the constants aren't.

First, let's get the mouse ID out of xinput list. We want to pull out the digits immediately following "id=", and nothing else.

$ xinput list | grep Mouse | sed 's/.*id=\([0-9]*\).*/\1/'

Save that in a variable (because we'll need to use it more than once) and feed it in to list-props to get the deceleration ID. Then use sed again, in the same way, to pull out just the thing in parentheses following "Deceleration":

$ mouseid=$(xinput list | grep Mouse | sed 's/.*id=\([0-9]*\).*/\1/')
$ xinput list-props $mouseid | grep 'Constant Deceleration'
        Device Accel Constant Deceleration (262):       2.000000
$ xinput list-props $mouseid | grep 'Constant Deceleration' | sed 's/.* Deceleration (\([0-9]*\)).*/\1/'

Whew! Now we have a way of getting both the mouse ID and the ID for the "Constant Deceleration" parameter, and we can pass them in to set-prop with our desired value (I'm using 2) tacked onto the end:

$ xinput set-prop $mouseid $(xinput list-props $mouseid | grep 'Constant Deceleration' | sed 's/.* Deceleration (\([0-9]*\)).*/\1/') 2

Add those two lines (setting the mouseid, then the final xinput line) wherever your window manager will run them when you start X. For me, using Openbox, they go in .config/openbox/autostart. And now my mouse will automatically be the speed I want it to be.

Tags: , ,
[ 13:42 Jan 31, 2016    More linux | permalink to this entry | comments ]