Shallow Thoughts : tags : editors

Converting C to Python with a vi regexp

I'm fiddling with a serial motor controller board, trying to get it working with a Raspberry Pi. (It works nicely with an Arduino, but one thing I'm learning is that everything hardware-related is far easier with Arduino than with RPi.)

The excellent Arduino library helpfully provided by Pololu has a list of all the commands the board understands. Since it's Arduino, they're in C++, and look something like this:

#define QIK_GET_FIRMWARE_VERSION         0x81
#define QIK_GET_ERROR_BYTE               0x82
#define QIK_GET_CONFIGURATION_PARAMETER  0x83
[ ... ]
#define QIK_CONFIG_DEVICE_ID                        0
#define QIK_CONFIG_PWM_PARAMETER                    1

and so on.

On the Arduino side, I'd prefer to use Python, so I need to get them to look more like:

    QIK_GET_FIRMWARE_VERSION = 0x81
    QIK_GET_ERROR_BYTE = 0x82
    QIK_GET_CONFIGURATION_PARAMETER = 0x83
[ ... ]
    QIK_CONFIG_DEVICE_ID = 0
    QIK_CONFIG_PWM_PARAMETER = 1

... and so on ... with an indent at the beginning of each line since I want this to be part of a class.

There are 32 #defines, so of course, I didn't want to make all those changes by hand. So I used vim. It took a little fiddling -- mostly because I'd forgotten that vim doesn't offer + to mean "one or more repetitions", so I had to use * instead. Here's the expression I ended up with:

.,$s/\#define *\([A-Z0-9_]*\) *\(.*\)/ \1 = \2/

In English, you can read this as:

From the current line to the end of the file (,.$/), look for a pattern consisting of only capital letters, digits and underscores ([A-Z0-9_]). Save that as expression #1 (\( \)). Skip over any spaces, then take the rest of the line (.*), and call it expression #2 (\( \)).

Then replace all that with a new line consisting of 4 spaces, expression 1, a spaced-out equals sign, and expression 2 ( \1 = \2).

Who knew that all you needed was a one-line regular expression to translate C into Python?

(Okay, so maybe it's not quite that simple. Too bad a regexp won't handle the logic inside the library as well, and the pin assignments.)

Tags: regexp, linux, programming, python, editors, vim
[ 20:38 Jan 19, 2013    More linux/editors | permalink to this entry | comments ]

Integrating graphics with text in Emacs

I discussed Emacs's artist-mode a few days ago as a simple, but incomplete, solution to the problem of sketching graphs while taking notes during a math class. But I've found a much better way, one that allows for including any images -- drawings, photos, or screenshots. It took a little work and some custom .emacs code, but I love the result.

Iimage mode

The key is iimage-mode, which displays inline images. In this mode, you put a line in your buffer with a reference to your image file, something like this:

file://myimage.jpg

and Emacs will replace it with the contents of that image. Marvellous!

You can use other patterns for filenames as well, but I'm fine with using URLs. Note there are only two slashes in file:// -- it's a local file in the same directory as the text file being edited.

It's a little tricky to enable it. The docs are not entirely clear on the differences between iimage-mode, turn-on-iimage-mode and iimage-mode-buffer. I found I could get a file that already had existing images to display them with:

  (turn-on-iimage-mode)
  (iimage-mode-buffer t)

Very cool! But too much to type every time. And to use it for note-taking, I needed a way to say, "Create a new image here, let me edit it, then display the image I just edited inline."

Enabling iimage-mode automatically

First, I wanted iimage mode displayed automatically on files in my note-taking directories. I normally use text-mode for these files, with spell checking and line wrapping turned on (auto-fill mode). So I defined a new minor mode based on text-mode:

(define-derived-mode text-img-mode text-mode "Image display mode"
  (auto-fill-mode)
  (turn-on-iimage-mode)
  (iimage-mode-buffer t)
  )

Then I wanted this mode to be called whenever I'm editing a file in my classes directory. So I added it to my auto-mode-alist:

(setq auto-mode-alist
   ...
      (cons '("Docs/classes/" . text-img-mode)
   ...
      auto-mode-alist) )

Inserting a new image

Next, I needed a way to insert an image URL into the buffer and call up an image editor on it. I shouldn't have to type the filename twice and keep track of it; that's what computers are for.

And I needed a drawing program. As a longtime GIMP geek, most of my computer drawing has been in GIMP. But GIMP is overkill for calling up a quick sketch window. I was tempted to use TuxPaint; it's a good sketching app even if you're not five years old, and it's fun and easy to use. But by default, TuxPaint has some features that get in the way of note-taking, like distracting sound effects. I'm sure it's possible to turn those off, and I do plan to investigate that.

I saw a reference to pinta as a lightweight drawing app, but it required a boatload of Mono libraries that I don't otherwise need, and Krita has the same problem with KDE services. So I opted for MyPaint. It works okay, though it's rather slow to start up and has some other issues, so I'm still hoping to find a more lightweight sketching app.

In any case, I fiddled around with start-process until I figured out how to use it to start a program. Then I wrote a little function that lets the user pick a filename, inserts a URL to that filename into the buffer, then calls up mypaint on the file.

(defun img ()
  "Prompt for a filename, then call up mypaint to create an image"
  (interactive)
  (let ((imgfile (read-string "Filename? " "xxx.jpg" 'my-history)))
    (insert "\nfile://" imgfile "\n" )
    (start-process "mypaint" nil "/usr/bin/mypaint" imgfile)
  ))

Worked fine! I can run M-x img, be prompted for a filename, and get a mypaint window where I can make my sketch.

Noticing that a new image has been added

But wait. I finish sketching, write the file and quit mypaint ... and the buffer still shows something like file://xxx.jpg, even if it's showing other images inline. I needed a way to tell it to refresh and load any new images. (I considered having emacs wait for mypaint to exit, but decided I might sometimes want to keep editing while mypaint was still up.)

M-x eval-expression (iimage-mode-buffer t) will do that, but that's a lot of typing to do. Obviously, I needed a key binding.

Strangely enough, C-c i wasn't taken for text buffers, so that seemed like a natural. So I added a key binding to the end of the text-img-mode. iimage-mode-buffer requires that t argument -- it gives an error without it -- so the key binding looks a little more complicated than one that just calls a simple function. I added it to the end of my text-img-mode function.

(define-derived-mode text-img-mode text-mode "Image display mode"
 ...
  (local-set-key "\C-ci" 
    (lambda () (interactive) (iimage-mode-buffer t)))
  )

But after using it a bit, I discovered that this didn't reload images if I edited them a second time. Fortunately, vwood had the answer:

(defun refresh-iimages ()
  "Only way I've found to refresh iimages (without also recentering)"
  (interactive)
  (clear-image-cache nil)
  (iimage-mode nil)
  (iimage-mode t)
  (message "Refreshed images")
 )

I added the message at the end, since otherwise the function left a distracting "Toggling iimage-mode off; better pass an explicit argument" error.

Then the key binding in my text-img-mode became

(local-set-key "\C-ci" 'refresh-iimages)

Inserting a screenshot

Wait -- one more thing. As I actually used text-img-mode to take notes, I discovered that taking screenshots would actually be much more useful than making my own drawings. Then I could copy small sections of the slides and graphs into my notes at the appropriate place, without needing to copy equations at all.

Why not write a function to allow that? The unpleasantly named scrot program fills the bill nicely, and gives me a choice of clicking in a window or dragging out an area of the screen.

(defun screenshot ()
 "Prompt for a filename, then call up scrot to create an interactive screenshot"
  (interactive)
  (let ((imgfile (read-string "Filename? " "scr.jpg" 'my-history)))
    (insert "\nfile://" imgfile "\n" )
    (start-process "scrot" nil "/usr/bin/scrot" "-s" imgfile)
  ))

This turned out to be so useful that I added a key for it in text-img-mode:

(local-set-key "\C-cs" 'screenshot)

I'm so happy with the result! Iimage mode is working great, and having text and images together is turning out to be perfect for note-taking.

My only problem now -- okay, I admit it -- is a tendency to get so excited over inserting screenshots that I get distracted and forget to actually listen to the lecture. I'm sure I'll get over that, but for now, Thank goodness vlc is good at skipping back!

Tags: editors, emacs, graphics, education
[ 12:42 Jan 12, 2013    More linux/editors | permalink to this entry | comments ]

ASCII graphics in Emacs with Artist Mode

I found a cool package in Emacs the other day: Artist Mode. It lets you draw ASCII graphics in your text file.

I was actually looking for the solution to a different problem: taking notes in a math-intensive Coursera class. I've been taking notes in Emacs, but a text editor is awkward for equations and even more awkward for graphs.

What I really wanted was something like the old Claris Works (or so I'm told; I never used it myself) -- something that's primarily a text editor but lets you drawings, equations, and tables when you need to. In theory, word processors like LibreOffice could do that, but in practice they're not very good at switching modes, nor at integrating several types of media into one document. Texmacs is great for the equations and apparently it can do tables too, but it can't do freehand drawing.

And none of these programs is very configurable -- I can't use my fast, comfortable Emacs bindings while typing, and that's a deal-breaker for me, because being able to make corrections quickly makes a huge difference in my typing speed. LibreOffice's key bindings are only partially configurable, and after you've spent half a day chasing down all the action names you need (the ones that are actually available), you upgrade to a newer version and discover you have to do it all over again because there's no way to migrate configuration files. Even Texmacs, ironically, is no better: the documentation claims it's possible to configure key bindings, but it doesn't appear anyone has ever succeeded in figuring out how.

Anyway, ASCII graphics aren't the ultimate solution to note-taking. And I've found a better solution for that, while I'll write about separately. But for now, Artist Mode is just so cool I had to share it.

Enable it by running M-x artist-mode. You can immediately start drawing in your buffer with the mouse. Whatever you draw gets turned into ASCII graphics.

For note-taking, it's fine for scribbling the rough shape of a curve. It takes no time to mouse in a little sketch like

       |                               .. 20
       |                              ..
       |                            ...
    10 |.                          ..
       |..                       ...
       | ...                   ...
       |   ..              ....
       |    .....      .....
       |    ............
       |.....        .............
       +-------------------------------------

It even has primitives (middleclick to get a.menu) for things like lines, rectangles and circles, and for filling regions. When you're done drawing, M-x artist-mode goes back to whatever mode you were using before.

I probably won't use it very much for note taking. But there are times when I've wanted to draw ASCII graphics -- a laborious process in ordinary text modes -- and other times when it would just be fun to play around with my buffer. I'm happy to know about Artist Mode. I may not need it often, but it sure is fun to use now and then.

Tags: editors, emacs, graphics, education
[ 19:02 Jan 10, 2013    More linux/editors | permalink to this entry | comments ]

Customize the Emacs modeline color

I wrote last week about how to customize syntax highlighting colors in Emacs. And as part of that, I ditched the color theme I'd been using and let Emacs go back to its default colors.

Which mostly was fine, except that when I split the window into two windows, to look at two files at once or two different parts of the same file, the separator between the two windows -- the mode line -- was the same grey as Emacs's normal background, so it wasn't very obvious where the window split was.

A web search turned out lots of different ways to set the mode line color. Many of them involve color themes and are fairly complicated. Here's the simplest method I found:

(set-face-foreground 'modeline "white")
(set-face-background 'modeline "purple")
(set-face-background 'modeline-inactive "light blue")

You can set your active mode line to a pretty color, so it stands out a bit and makes it easy to tell which of the visible windows is the one you're actually typing in, and set the inactive mode lines -- windows that are visible but you arne't actually typing in -- to a less striking color.

Tags: editors, emacs
[ 12:50 Jan 02, 2013    More linux/editors | permalink to this entry | comments ]

Customizing syntax highlighting colors in Emacs

Emacs has wonderful syntax highlighting. Words will be displayed in different colors depending on their syntax and the mode of the current file -- for instance, in C code, keywords of the language are highlighted in one color, comments in another, strings in a third.

The problem comes when the colors aren't right. Like that awful gold color that the flyspell spell checker uses for some words. Against a light background it makes the words almost impossible to read. I've struggled for years trying to set up custom color schemes to get around that problem, but I finally learned a simpler way to handle it when you see something in a color you want to change.

The trick is to find out what face you need to change. "Face" to emacs means more than a font face like Sans or Lucida; it means a collection of information about how characters are displayed, including font face, weight, slant, color and other attributes.

When you see something displayed in a color you don't want, place the cursor somewhere in the word. type C-u C-x = will get you the face used, along with all sorts of information about it and a handy Customize what to show link. Or you can go straight to the Customize screen with M-x customize-face -- hit return to customize the face at point (the cursor location).

In the customize-face screen, there's no GUI to choose colors, but you can edit color names. Emacs lists "red" as the foreground color; if you change it to "blue" you'll see a preview of how it will look. Color names come from /etc/X11/rgb.txt, and there are various programs like xcolorsel that will show them -- or better yet, see Wikipedia's X11 color names chart.

Once you've chosen a color, the Save for future sessions button will add a section to your .emacs file with the appropriate elisp code. Of course, you can move this code elsewhere as well. I have a somewhat complex .emacs setup, so I've moved the code into another file. Strangely, I found that my .Xdefaults background color setting no longer worked once I started using custom-set-faces, so I added a line for that as well.

(set-background-color "grey90")
(custom-set-faces
 '(flyspell-duplicate ((((class color)) (:foreground "red" :underline t :weight bold))))
 '(font-lock-comment-face ((((class color) (min-colors 88) (background light)) (:foreground "blue"))))
 )

Tags: emacs, editors
[ 12:59 Dec 23, 2012    More linux/editors | permalink to this entry | comments ]

Vim tip: opening lines without delay

I use vim to reply to email messages in mutt. And something I do very often is to open some new lines between quoted sections. For instance, if I'm replying to mail with a section like

> Have you noticed anything weird about squirrels lately?
> I think they're plotting something.

I might want to open up space after the first line, and say

> Have you noticed anything weird about squirrels lately?

You too? One of our local squirrels ran off with my Arduino yesterday.

> I think they're plotting something.

So I need three blank lines, with the cursor on the middle line.

In vim, typically I would place the cursor on the line before the space I want to open, and type o<RETURN><ESC>O :

o

open a line below the current line

<RETURN>

make an empty line below the quoted line

<ESC>

get out of insert mode

O

open a new line above the current line and enter insert mode there

Sounds complicated, but it's pretty fast to type o<RET><ESC>O especially when you do it many times a day.

Except vim has an odd quirk: after you type that O, you have to wait a second or two. If you start typing immediately, somehow you're not in insert mode any more, and the line you just opened, and thought you were typing in, disappears.

Well, it turns out this happens because <ESC>O is a sequence that can also happen from arrow keys. So vim has a timeout it uses to figure out whether an <ESC>O sequence was from an arrow key, or two characters typed by a real person. If the O follows too soon after the <ESC>, it figures it's an arrow key. (This is discussed in more detail in this Stack Overflow thread: Vim: Delay before 'O' opens a new line?)

It turns out there's a simple solution: set the timeout much shorter, like 100 milliseconds, in .vimrc:

set ttimeoutlen=100

Then arrow keys should still work -- unless you typically use vim over a very slow connection -- but you can type the sequence yourself (unless you type very fast) and not have it confused with an arrow key.

But really, there's a better solution. Since opening up space like this is something I do so often, why not make a vim binding for it?

map <leader>o o<CR><ESC>O

Now all I have to do is type \o (backslash, \, is the "leader" character) and I get a space opened up, with a blank line both before it and after it.

Tags: editors, vim
[ 11:20 Jul 21, 2012    More linux/editors | permalink to this entry | comments ]

Editing tab-separated files with Emacs

I wanted to keep maintenance records for my car in a local file. Things like the date and mileage of the last oil change, when the timing belt was replaced, stuff like that.

I didn't want to hassle with databases or spreadsheets -- a simple, human-readable file is a lot easier to deal with. But I also wanted it in a predictable format so it could potentially be parsed later: I might some day want to write a program that keeps track and reminds me when I need to do things. (You'd think this program would already exist, but curiously, I haven't found one.)

So, something like:

7/9/12 <TAB> 86306 <TAB> Oil change, filter
6/11/12<TAB> 84813 <TAB> Smog test

Simple, right? And super easy just to type in in a text editor.

Well ... in emacs, maybe not. As an editor oriented toward programmers, emacs tends to prefer spaces instead of tabs ... and that's normally the way I prefer it. Tabs are a bad idea in most software projects, unless everybody on the project has already agreed on a tab style and width. (The complete tabs-vs.spaces religious war is beyond the scope of this article, but take my word for it that they can be a problem.)

But this wasn't code, and I needed an easy way to get those tabs into the file. Of course, I could quote them directly: type Ctrl-Q TAB every time I wanted a tab. But I knew I wouldn't remember to do that every time.

So I needed a local variables line, sometimes called a modeline, to flag this file as different from any other file I edit -- in this one, I really do want a TAB when I hit the TAB key.

My first try put this as the first line of the file:

-*- Mode: Text; indent-tabs-mode: t -*-

(Actually, my first try omitted the -*- part and didn't work at all, but let's ignore that part.)

When I re-loaded the file, emacs loaded it in text mode, and indent-tabs-mode was set to t if I checked it with describe-variable. Great! I could add new lines at the end of the file, and when I hit TAB, emacs inserted tabs. But wait -- if I added a line at the beginning of the file, and typed something lke 7/9/12<TAB> ... it inserted a space, not a tab.

It turns out indent-tabs-mode isn't really documented. If you check the help, it says:

Indentation can insert tabs if this is non-nil

-- note that "can" insert, not "will" insert, a tab. So how could I get emacs to really insert a tab, every time I hit the tab key?

Well, of course, I could bind the TAB key to "self-insert". But I didn't want to change it for every file, only for this one. how could I make that happen as part of a local variables line? The local variables line documentation shows how to set variables or file modes, but it looks like you can't bind keys or run functions.

Well, you can't run functions, but you can define a new mode that runs functions. At the suggestion of some helpful people on the #emacs IRC channel, I defined a new mode in my .emacs, like this:

(define-derived-mode tabbed-mode text-mode "Tab separated mode"
  (local-set-key (kbd "TAB") 'self-insert-command)
  )

Then in the file, I put this line:

-*- Mode: Tabbed -*-

Now, every time I load the file, emacs turns on tabbed mode -- which is just text mode except that the TAB key always inserts a tab.

Tags: editors, emacs
[ 18:35 Jul 16, 2012    More linux/editors | permalink to this entry | comments ]

Using default-frame-alist to specify Emacs window size, position and font

Seems like every few years I need to change the way I specify my preferred emacs fonts and window sizes.

Historically this all used to happen from one file, ~/.Xdefaults, where you set up your defaults for all X programs. In a way that was nice, since you could set up defaults and see the same font everywhere. On the other hand, it made for a huge, somewhat hard to read file, and it's increasingly out of favor on modern desktops, with modern toolkits like GTK just ignoring it.

Emacs still reads Xdefaults -- but only sort of. A lot of the values I used to set there no longer work properly. Some time ago I commented out my various attempts at setting emacs font, like

Emacs*font: -*-clean-bold-*-*-*-13-*-*-*-c-*-*-*
Emacs*font: DejaVu Sans Mono-10:bold
Emacs*font: clean-13:bold
Wmacs*font: Liberation Mono-10:bold
Emacs.font: 7x13bold
Emacs.faceName: Dejavu-10:style=bold

since none of them worked, and worked out a way of setting fonts inside my .emacs file:

(set-face-attribute 'default nil :font "Terminus-12:bold")

That worked to set the font, but it had another annoying attribute: it doesn't happen at startup, so it messed up my window size. See, emacs would start up, see the size I specified in .Xdefaults:

Emacs*geometry: 80x45

and try to set that. But it hadn't read .emacs yet, so it was still using whatever its default font and size is, and that's huge -- so 45 lines made a window too tall to fit on my laptop screen. Emacs would then shrink its window to fit the screen (41 lines). Only then would it open .emacs, whereupon it would see the set-face-attribute, change the font, and resize the window again, much, smaller, still 41 lines.

What a pain!

The emacs manual, in addition to talking about these various Xdefaults properties and command-line options, does mention a couple of variables, set-screen-height and set-screen-width, that looked promising. I tried putting (set-screen-height 45) in my .emacs right after I set the font -- no dice. Apparently that doesn't work because by the time those are read, emacs has already decided that 41 lines is as big as the window can possibly be.

Here's the answer: another variable that goes inside .emacs, default-frame-alist, but this one can override that maximum-height decision that emacs has already made. Here's an example of it in some useful defaults for emacs, and based on that, I was able to come up with this:

(setq default-frame-alist
      '((top . 10) (left . 2)
        (width . 80) (height . 53)
        (font . "terminus-iso8859-1-bold-14")
        ))

Curiously, that height setting, 53, needs to be 3 more than what I actually want according to the size emacs reports to the window manager. So don't take the number too seriously; just try numbers a little bigger than what you actually want until you get the size you're after. The font setting is the X font specifier: I ran xlsfonts | grep -i terminus | grep 14 then picked one of the simpler of the lines it printed out, but you can use a full specifier like -xos4-terminus-bold-r-normal--14-140-72-72-c-80-iso8859-1 like you get from xfontsel, if you prefer.

Startup still isn't pretty -- emacs still shows a big window at one place on the screen, resizes it several times then jumps it over to the top/left coordinates I specified. Of course, I could tell my window manager to start it in the right place so the jumping-around would be minimized; but that wouldn't help the visible resizing. Just a minor irritation.

I'm sure there's lots more useful stuff buried in that sample emacs config file (it was suggested to me when I asked about this on the #emacs IRC channel), so I'll be reading it to see what else I can glean.

Tags: emacs, editors, X11
[ 11:24 Jul 14, 2011    More linux/editors | permalink to this entry | comments ]

Vim/Emacs tip: use modelines for files that need special behavior

Most of the time when I edit a text file in vim, I want lines to wrap automatically as I type, at somewhere around 70 columns wide. So I set textwidth=70 in .vimrc.

But sometimes that isn't appropriate. For instance, I have a procmail rules file where I put common spam patterns that programs like spamassassin don't seem to be able to catch. So I might have lines like:

*^Subject:.*(Ink Cartridges|Hummingbird Vine|who's who of executives|Avandia|Botox|Your Email ID|Zoosk|Best airfares on the internet|UGG Boots|police training)

... and so on -- you get the idea. I can't have lines breaking in the middle, because then the procmail rule wouldn't work. So every time I add a new phrase, I have to :set tw=0 (or one of the other umpteen ways one can tell vim not to wrap lines) first.

But you can set special behavior for one specific file by adding a special comment called a "modeline" as the first line of the file.

Procmail treats any line starting with a hash, #, as a comment, and vim recognizes # as a comment. So I can add this as the first line of the procmail file:

# vim: set tw=0:

then vim will see that and un-set that default text width I specify in .vimrc.

Vim understands most common comment styles, so it should understand lines like /* vim: set tw=0: */ and // vim: set tw=0: and ; vim: set tw=0: as well.

But to make this work I had to do one more thing: in .vimrc, I had to add

set modeline

Apparently on some versions of vim this is on by default; in others it's turned off for security reasons (someone could put an evil modeline into a file which would make your vim do something when you edited it). Definitely something to be aware of, but if you mostly edit files you created yourself on your local machine, and no one else uses your machine, it's your choice whether to worry about it.

Emacs has modelines too

Emacs has mode lines too, though it calls them Local variables lines. For instance, C++ files in Mozilla's source tend to start with:

/* -*- Mode: C++; tab-width: 2; indent-tabs-mode: nil; c-basic-offset: 2 -*- */

It's awfully handy to be able to define specific indentation style for the files within a project, making it easy for emacs users, at least, to follow your preferred coding style. If only all editors understood them!

Tags: editors, vim, emacs, tip
[ 20:26 May 25, 2011    More linux/editors | permalink to this entry | comments ]

Quick tip: Disabling version control in Emacs

For some time I've been mildly annoyed that whenever I start emacs and open a file that's under any sort of version control -- cvs, svn, git or whatever -- I can't start editing right away, because emacs has to pause for a while and load a bunch of version-control cruft I never use. Sometimes it also causes problems later, when I try to write to the file or if I update the directory.

It wasn't obvious what keywords to search for, but I finally found a combination, emacs prevent OR disable autoload vc (the vc was the important part), which led me to the solution (found on this page):

;; Disable all version control
(setq vc-handled-backends nil)

Files load much faster now!

Tags: editors, emacs, git, tip
[ 12:11 Feb 04, 2011    More linux/editors | permalink to this entry | comments ]

Escaping HTML characters in Emacs (and how to do replaces in elisp)

Last week I found myself writing another article that includes code snippets in HTML.

So what, you ask? The problem is, when you're writing articles in HTML, every time you include a code snippet inside a <pre> tag you invariably forget that special characters like < > & have special meanings in HTML, and must be escaped. Every < has to change to &lt;, and so forth, after you paste the code.

In vi/vim, replacing characters is straightforward. But I usually write longer articles in emacs, for various unimportant reasons, and although emacs has global replace, it only works from wherever you are now (called "point" in emacs lingo) to the end of the file. So if you're trying to fix something you pasted in the middle of the article, you can't do it with normal emacs replace.

Surely this is a wheel that has already been re-invented a thousand times, I thought! But googling and asking emacs experts turned up nothing. Looks like I'd have to write it.

And that turned out to be more difficult than I expected, for the same reason: emacs replace-string works the same way from a program as it does interactively, and replaces from point to the end of the file, and there's no way to restrict it to a more limited range.

Several helpful people on #emacs chimed in with ideas, but most of them didn't pan out. But ggole knew a way to do it that was both clean and reliable (thanks!).

Here's the elisp function I ended up with. It uses save-excursion to put the cursor back where it started before you ran the function, narrow-to-region to make replace-string work only on the region, and save-restriction get rid of that narrow-to-region after we're done. Nice!

(defun unhtml (start end)
  (interactive "r")
  (save-excursion
    (save-restriction
      (narrow-to-region start end)
      (goto-char (point-min))
      (replace-string "&" "&")
      (goto-char (point-min))
      (replace-string "<" "&lt;")
      (goto-char (point-min))
      (replace-string ">" "&gt;")
      )))

And yes, I used it just now on that elisp snippet.

Tags: emacs, editors, programming
[ 19:08 Dec 01, 2010    More linux/editors | permalink to this entry | comments ]

Non-greedy regular expressions to clean up crappy autogenerated HTML

I maintain the websites for several clubs. No surprise there -- anyone with even a moderate knowledge of HTML, or just a lack of fear of it, invariably gets drafted into that job in any non-computer club.

In one club, the person in charge of scheduling sends out an elaborate document every three months in various formats -- Word, RTF, Excel, it's different each time. The only regularity is that it's always full of crap that makes it hard to turn it into a nice simple HTML table.

This quarter, the formats were Word and RTF. I used unrtf to turn the RTF version into HTML -- and found a horrifying page full of lines like this:

<body><font size=3></font><font size=3><br>
</font><font size=3></font><b><font size=4></font></b><b><font size=4><table border=2>
</font></b><b><font size=4><tr><td><b><font size=4><font face="Arial">Club Schedule</font></font></b><b><font size=4></font></b><b><font size=4></font></b></td>
<font size=3></font><font size=3><td><font size=3><b><font face="Arial">April 13</font></b></font><font size=3></font><font size=3><br>
</font><font size=3></font><font size=3><b></b></font></td>

I've put the actual page content in bold; the rest is just junk, mostly doing nothing, mostly not even legal HTML, that needs to be eliminated if I want the page to load and display reasonably.

I didn't want to clean up that mess by hand! So I needed some regular expressions to clean it up in an editor. I tried emacs first, but emacs makes it hard to try an expression then modify it a little when the first try doesn't work, so I switched to vim.

The key to this sort of cleanup is non-greedy regular expressions. When you have a bad tag sandwiched in the middle of a line containing other tags, you want to remove everything from the <font through the next > -- but no farther, or else you'll delete real content. If you have a line like

<td><font size=3>Hello<font> world</td>

you only want to delete through the <font>, not through the </td>.

In general, you make a regular expression non-greedy by adding a ? after the wildcard -- e.g. <font.*?>. But that doesn't work in vim. In vim, you have to use \{M,N} which matches from M to N repetitions of whatever immediately precedes it. You can also use the shortcut \{-} to mean the same thing as *? (0 or more matches) in other programs.

Using that, I built up a series of regexp substitutes to clean up that unrtf mess in vim:

:%s/<\/\{0,1}font.\{-}>//g
:%s/<b><\/b>//g
:%s/<\/b><b>//g
:%s/<\/i><i>//g
:%s/<td><\/td>/<td><br><\/td>/g
:%s/<\/\{0,1}span.\{-}>//g
:%s/<\/\{0,1}center>//g

That took care of 90% of the work, leaving me with hardly any cleanup I needed to do by hand. I'll definitely keep that list around for the next time I need to do this.

Tags: regexp, html, editors, vim
[ 22:02 Mar 29, 2010    More linux/editors | permalink to this entry | comments ]

Make scroll lock stop beeping in Emacs 23

I haven't been using the spare machine much lately. So I hadn't noticed until last week that since upgrading to the emacs 23.1.1 on Ubuntu Karmic koala, every time I press the Scroll Lock key -- the key my KVM uses to switch to the other computer -- with focus in an emacs window, emacs beeps and complains that the key is unbound.

That was a problem I thought I'd solved long ago, an easy fix in .emacs:

(global-set-key [scroll-lock] 'ignore)

But in emacs 23, it wasn't working any more. Emacs listed the key as "<Scroll_Lock>", but using that directly in global-set-key doesn't work.

The friendly and helpful (really!) crew at #emacs found me a solution, after some fiddling around.

(global-set-key (kbd "<Scroll_Lock>") 'ignore)

Tags: emacs, editors, kvm, tips
[ 22:47 Feb 09, 2010    More linux/editors | permalink to this entry | comments ]

Taming Emacs' text mode wrapping and indenting

To wrap long lines, or not to wrap? It's always a dilemma. Automatic wrapping is great when you're hammering away typing lots of text. But it's infuriating when you're trying to format something yourself and the editor decides it wants to line-wrap a little too early.

Although of course you can set the wrapping width, Emacs has a tendency to wrap early -- especially when you hit return. All too often, I'll be typing away at a long line, get to the end of the sentence and paragraph with the last word on the same line with the rest -- then realize that as soon as I hit return, Emacs is going to move that last word to a line by itself. Drives me nuts!

And the solution turns out to be so simple. The Return key, "\C-m". was bound to the (newline) function (you can find out by typing M-x, then describe-key, then hitting Return). Apparently (newline) re-wraps the current line before inserting a line break. But I just wanted it to insert a line break.

No problem -- just bind "C-m" to (insert "\n").

But there's a second way, too, if you don't want to rebind: there's a magic internal emacs table you can change.

(set-char-table-range auto-fill-chars 10 nil)

But wait -- there's one other thing I want to fix in text mode.

Automatic indent is another one of those features that's very convenient ... except when it's not.

If I have some text like:

First point:
  - subpoint a
  - subpoint b

then it's handy if, when I hit Return after subpoint a, emacs indents to the right level for subpoint b. But what happens when I get to the end of that list?

First point:
  - subpoint a
  - subpoint b

Second point:
  - subpoint c

When I hit Return after subpoint b, Emacs quite reasonably indents two spaces. If I immediately type another Return, Emacs sensibly deletes the two spaces it just inserted, opens a new line -- but then it indents that new line another two spaces.

After a blank line, I always want to start at the beginning, not indented at all.

Here's how to fix that. Define a function that will be called whenever you hit return in text mode. That function tests whether the caret comes immediately after a blank line, or at the beginning of the file. It indents except in those two cases; and in neither case does it re-wrap the current line.

;; In text mode, I don't want it auto-indenting for the first
;; line in the file, or lines following blank lines.
;; Everywhere else is okay.
(defun newline-and-text-indent ()
  "Insert a newline, then indent the next line sensibly for text"
  (interactive)
  (cond
   ;; Beginning of buffer, or beginning of an existing line, don't indent:
   ((or (bobp) (bolp)) (newline))

   ;; If we're on a whitespace-only line,
   ((and (eolp)
         (save-excursion (re-search-backward "^\\(\\s \\)*$"
                                             (line-beginning-position) t)))
    ;; ... delete the whitespace, then add another newline:
    (kill-line 0)
    (newline))

   ;; Else (not on whitespace-only) insert a newline,
   ;; then add the appropriate indent:
   (t (insert "\n")
      (indent-according-to-mode))
   ))

Then tell emacs to call that function when it sees the Return key in text mode:

(defun text-indent-hook ()
  (local-set-key "\C-m" 'newline-and-text-indent)
  )
(setq text-mode-hook 'text-indent-hook)

Finally, this is great for HTML mode too, if you get irritated at not being able to put an <a href="longurl"> all on one line:

(defun html-hook ()
  (local-set-key "\C-m" (lambda () (interactive) (insert "\n")))
  )
(setq sgml-mode-hook 'html-hook)

Tags: emacs, editors, tips
[ 10:29 Jan 13, 2010    More linux/editors | permalink to this entry | comments ]

Emacs: Insert tags in HTML Mode

Wouldn't it be nice if Emacs HTML mode had a way to insert HTML tags, so you didn't have to type <b></b> all the time? Sort of like what's described in this page -- except that page describes an HTML mode that clearly isn't the one that's installed on Ubuntu, since none of those bindings actually work?

I've been meaning to figure out a way to do that for ages, and finally got around to it. Turns out Emacs SGML mode (which is really what Ubuntu installs and uses for HTML files) doesn't have functions for specific HTML tags like <b>, but it does have a general tag-inserting function.

Type C-c C-t -- emacs prompts you for the tag, so type b or whatever, and hit return -- and you get the tag, with the cursor correctly positioned for you to type your new bold text.

But that's four keystrokes. What if you want shorter bindings for particular tags, like C-b C-b to insert a bold?

For that, you need to use a lambda and a mode hook. In your .emacs it looks like this:

;; Define keys for inserting tags in HTML mode:
(defun html-hook ()
  (local-set-key "\C-c\C-b" (lambda () (interactive) (sgml-tag "b")))
  )
(setq sgml-mode-hook 'html-hook)

There's apparently also supposed to be a command bound to C-c / that closes the current tag, but my version of sgml-mode doesn't bind anything to that key, and the only likely-looking function name, sgml-maybe-end-tag, doesn't end the current tag. Such is life!

But one more don't-miss feature that I'd missed all along is C-c C-n: type it before a special character like < or & and emacs will insert the appropriate &lt; or &amp; for you. Nice!

(Thanks to bojohan on #emacs for the tips!)

Tags: editors, emacs, html-mode
[ 20:31 Jul 29, 2009    More linux/editors | permalink to this entry | comments ]

Emacs bookmarks -- a huge time-saver

Oh, wow. I can't believe I've used Emacs all these years without knowing about bookmarks.

I wanted something in Emacs akin to the "Open Recent" menu that a lot of GUI apps have. Except, well, I didn't want it to need a menu (I don't normally show a menubar in Emacs) and I didn't want it limited only to recently accessed files. So ... just like Open Recent, only completely different.

What I really wanted was a way to nickname files I access regularly, so I don't have to type ~/foo/bar/blaz/route-66/dufus/velociraptor/archaeopteryx/filename every time. Even with tab completion, remembering long paths gets old. Of course emacs must have a way to do that; it has everything. The trick was guessing what it might be called in order to search for it.

The answer is emacs bookmarks and they're super easy to use.

C-x r m sets a bookmark for the current location in the current file. It prompts for a bookmark name; give it a nickname, or hit return to default it to the current filename.

C-x r b bookmark-name jumps back to a bookmark, opening the file if it isn't already. Of course, tab completion works for the bookmark name.

Bookmarks are saved in ~/.emacs.bmk so they're persistent.

It's perfect. I just wish I'd thought to look for it years ago.

(Of course, Emacs can do recent files too.)

Tags: editors, emacs, bookmarks
[ 09:21 Mar 27, 2009    More linux/editors | permalink to this entry | comments ]

Vim tip: fixing the light-background color schemes

I use a light background for my X terminals (xterm and rxvt): not white, but a light grey that I find easy on the eyes. Long ago, I spent the time to set up a custom vim color scheme that works with the light background.

But sometimes I need to run vim somewhere where I don't have access to my custom scheme. It always starts up with a lot of the words displayed in yellow, completely unreadable against a light background. :set background=light doesn't help -- the default colorscheme is already intended for a light background, yet it still uses yellow characters.

I tried all the colorschemes installed with ubuntu's vim (you can get a list of them with ls /usr/share/vim/vim71/colors). The only light-background vim schemes that don't use yellow all have their primary text color as red. Do a lot of people really want to edit red text? Maybe the same people who think that yellow is a dark color?

Curiously, it turns out that if you use one of these light color schemes on a Linux console (with its black background), the yellow text isn't yellow (which would show up fine against black), but orange (which would be better on a light background).

Mikael knew the answer:

:set t_Co=88

This tells vim to use 88-color mode instead of its default of 8, and the yellow text turns light blue. Not terrifically readable but much better than yellow. Or, instead, try

:set t_Co=256

and the yellow/light blue text turns an ugly, but readable, orange (probably the same orange as the console used).

So, vim users with dark-on-light terminal schemes: add set t_Co=256 in your .vimrc (no colon) and you'll be much happier.

Update: Pádraig Brady has a great page explaining more about terminal colour highlights, including a TERM=xterm-256color setting to get vim to use 256 colors automatically. There's also a lot of good advice there on enabling colors in other console apps.

The only catch: on Ubuntu you do have to install the ncurses-term package, which will get you xterm-256color as well as 256color variants for lots of other terminal types. Here's useful page on 256-Color XTerms in Ubuntu.

Tags: editors, vim, color, tips
[ 21:29 Mar 22, 2009    More linux/editors | permalink to this entry | comments ]

More fun with regexps: Adding "[no output]" in shell logs

Someone on LinuxChix' techtalk list asked whether she could get tcsh to print "[no output]" after any command that doesn't produce output, so that when she makes logs to help her co-workers, they will seem clearer.

I don't know of a way to do that in any shell (the shell would have to capture the output of every command; emacs' shell-mode does that but I don't think any real shells do) but it seemed like it ought to be straightforward enough to do as a regular expression substitute in vi. You're looking for lines where a line beginning with a prompt is followed immediately by another line beginning with a prompt; the goal is to insert a new line consisting only of "[no output]" between the two lines.

It turned out to be pretty easy in vim. Here it is:

:%s/\(^% .*$\n\)\(% \)/\1[no results]\r\2/

Explanation:

:

starts a command

%

do the following command on every line of this file

s/

start a global substitute command

\(

start a "capture group" -- you'll see what it does soon

^

match only patterns starting at the beginning of a line

%

look for a % followed by a space (your prompt)

.*

after the prompt, match any other characters until...

$

the end of the line, after which...

\n

there should be a newline character

\)

end the capture group after the newline character

\(

start a second capture group

%

look for another prompt. In other words, this whole

expression will only match when a line starting with a prompt

is followed immediately by another line starting with a prompt.

\)

end the second capture group

/

We're finally done with the mattern to match!

Now we'll start the replacement pattern.

\1

Insert the full content of the first capture group

(this is also called a "backreference" if you want

to google for a more detailed explanation).

So insert the whole first command up to the newline

after it.

[no results]

After the newline, insert your desired string.

\r

insert a carriage return here (I thought this should be

\n for a newline, but that made vim insert a null instead)

\2

insert the second capture group (that's just the second prompt)

/

end of the substitute pattern

Of course, if you have a different prompt, substitute it for "% ". If you have a complicated prompt that includes time of day or something, you'll have to use a slightly more complicated match pattern to match it.

Tags: regexp, shell, CLI, linux, editors
[ 13:34 Oct 12, 2008    More linux/editors | permalink to this entry | comments ]

Smart Wrapping with Greedy and Non-Greedy Regular Expressions

I had a chance to spend a day at the AGU conference last week. The American Geophysical Union is a fabulous conference -- something like 14,000 different talks over the course of the week, on anything related to earth or planetary sciences -- geology, solar system astronomy, atmospheric science, geophysics, geochemistry, you name it.

I have no idea how regular attendees manage the information overload of deciding which talks to attend. I wasn't sure how I would, either, but I started by going through the schedule for the day I'd be there, picking out a (way too long) list of potentially interesting talks, and saving them as lines in a file.

Now I had a file full of lines like:

1020      U22A    MS 303  Terrestrial Impact Cratering: New Insights Into the Cratering Process From Geophysics and Geochemistry II

Fine, except that I couldn't print out something like that -- printers stop at 80 columns. I could pass it through a program like "fold" to wrap the long lines, but then it would be hard to scan through quickly to find the talk titles and room numbers. What I really wanted was to wrap it so that the above line turned into something like:

1020      U22A    MS 303  Terrestrial Impact Cratering: New Insights
                          Into the Cratering Process From Geophysics
                          and Geochemistry II

But how to do that? I stared at it for a while, trying to figure out whether there was a clever vim substitute that could handle it. I asked on a couple of IRC channels, just in case there was some amazing Linux smart-wrap utility I'd never heard of. I was on the verge of concluding that the answer was no, and that I'd have to write a python script to do the wrapping I wanted, when Mikael emitted a burst of line noise:
%s/\(.\{72\}\)\(.*\)/\1^M^I^I^I\2/

Only it wasn't line noise. Seems Mikael just happened to have been reading about some of the finer points of vim regular expressions earlier that day, and he knew exactly the trick I needed -- that .\{72\}, which matches lines that are at least 72 characters long. And amazingly, that expression did something very close to what I wanted.

Or at least the first step of it. It inserts the first line break, turning my line into

1020      U22A    MS 303  Terrestrial Impact Cratering: New Insights
                          Into the Cratering Process From Geophysics and Geochemistry II

but I still needed to wrap the second and subsequent lines.

But that was an easier problem -- just do essentially the same thing again, but limit it to only lines starting with a tab. After some tweaking, I arrived at exactly what I wanted:

%s/^\(.\{,65\}\) \(.*\)/\1^M^I^I^I\2/

%g/^^I^I^I.\{58\}/s/^\(.\{,55\}\) \(.*\)/\1^M^I^I^I\2/

I had to run the second line two or three times to wrap the very long lines.

Devdas helpfully translated the second one into English: "You have 3 tabs, followed by 58 characters, out of which you match the first 55 and put that bit in $1, and the capture the remaining in $2, and rewrite to $1 newline tab tab tab $2."

Here's a more detailed breakdown:

Line one:

%	Do this over the whole file
s/	Begin global substitute
^	Start at the beginning of the line
\(	Remember the result of the next match
.\{,65\}_	Look for up to 65 characters with a space at the end
\) \(	End of remembered pattern #1, skip a space, and start remembered pattern #2
.*\)	Pattern #2 includes everything to the end of the line
/	End of matched pattern; begin replacement pattern
\1^M	Insert saved pattern #1 (the first 65 lines ending with a space) followed by a newline
^I^I^I\2	On the second line, insert three tabs then saved pattern #2
/	End replacement pattern

Line two:

%g/	Over the whole file, only operate on lines with this pattern
^^I^I^I	Lines starting with three tabs
.\{58\}/	After the tabs, only match lines that still have at least 58 characters (this guards against wrapping already wrapped lines when it's run repeatedly)
s/	Begin global substitute
^	Start at the beginning of the line
\(	Remember the result of the next match
.\{,55\}	Up to 55 characters
\) \(	End of remembered pattern #1, skip a space, and start remembered pattern #2
.*\)	Pattern #2 includes everything to the end of the line
/	End of matched pattern; begin replacement pattern
\1^M	The first pattern (up to 55 chars) is one line
^I^I^I\2	Three tabs then the second pattern
/	End replacement pattern

Greedy and non-greedy brace matches

The real key is those curly-brace expressions, \{,65\} and \{58\} -- that's how you control how many characters vim will match and whether or not the match is "greedy". Here's how they work (thanks to Mikael for explaining).

The basic expression is {M,N} -- it means between M and N matches of whatever precedes it. (Vim requires that the first brace be escaped -- \{}. Escaping the second brace is optional.) So .{M,N} can match anything between M and N characters but "prefer" N, i.e. try to match as many as possible up to N. To make it "non-greedy" (match as few as possible, "preferring" M), use .{-M,N}

You can leave out M, N, or both; M defaults to 0 while N defaults to infinity. So {} is short for {0,∞} and is equivalent to *, while {-} means {-0,∞}, like a non-greedy version of *.

Given the string: one, two, three, four, five

,.\{},	matches , two, three, four,
,.\{-},	matches , two,
,.\{5,},	matches , two, three, four,
,.\{-5,},	matches , two, three,
,.\{,2},	matches nothing
,.\{,7},	matches , two,
,.\{5,7},	matches , three,

Of course, this syntax is purely for vim; regular expressions are unfortunately different in sed, perl and every other program. Here's a fun table of regexp terms in various programs.

Tags: linux, editors, regexp
[ 11:44 Dec 20, 2007    More linux/editors | permalink to this entry | comments ]

Emacs with Long Lines

I don't like composing text documents in word processors like Open Office. Call it a quirk if you like, but I find them intrusive: they take up a lot of CPU and memory, they take up a lot of window space for stuff I don't need while I'm writing (all those margins and rulers and toolbars and such) making it hard to compare two documents at once, and they tend to have intrusive focus behavior (like popping windows to the front when I didn't ask for it).

So when I need to write a paper (or a book), I prefer to compose in a text editor like vim or emacs, something that won't get in the way of my train of thought. When it's mostly written and ready to format, then I start up the big heavyweight word processor and import or paste the text into it.

(For those of you who think I'm insane and should just live in Open Office all day, the same problem comes up for people who do a lot of composing for web applications, such as an online blog, gmail, a web forum, or a wiki, and for people who want a choice of editor for their GUI mail app.)

Fine, but that introduces a problem. See, text editors have a fixed line width (typically 80 characters, though of course you can adjust this) and paragraphs are usually separated by blank lines (two newline characters together). Word processors expect each paragraph to be one long line for the whole paragraph, and line breaks are used as paragraph breaks (but you only want one of them, not two). How do you reconcile these two models in order to paste plaintext from an editor into a word processor?

Several years ago when I first encountered this problem, I investigated solutions in both vim and emacs (oddly enough, I'm an editor agnostic and equally happy in either one).

For vim, I never did find a solution to the problem, so that settled the editor choice for me. Perhaps some vim expert can let me know what I missed.

For emacs, I found longlines-mode, a hack which lets long lines appear to be wrapped while you're editing them even though they're really not. Apparently Wikipedia has this issue and some Wikipedia contributors use longlines-mode too. (That page also has brief notes on alternate solutions.)

I used longlines-mode for a long time, and it's more or less functional, but I was never really happy with it. It turns out to have some pretty annoying bugs which I was forever needing to work around, and it doesn't solve the blank-lines problem -- you still need to delete blank lines before or after pasting.

Yesterday I was working on an essay for a class I'm taking and decided I'd had enough of longlines-mode and wanted a better solution. I poked around and chatted with the nice folks on #emacs (hoping that someone had come up with a better solution, but no one knew of one) and based on some ideas they had, I came up with one of my own.

My new method is to edit the text file normally: line breaks where they look good, blank lines to separate paragraphs. When I'm finished writing and ready to paste, I run M-x wp-munge, which calls up a very simple function I wrote and added to my .emacs:

;; For composing in emacs then pasting into a word processor,
;; this un-fills all the paragraphs (i.e. turns each paragraph
;; into one very long line) and removes any blank lines that
;; previously separated paragraphs.
;;
(defun wp-munge () "un-fill paragraphs and remove blank lines" (interactive)
  (let ((save-fill-column fill-column))
    (set-fill-column 1000000)
    (mark-whole-buffer)
    (fill-individual-paragraphs (point-min) (point-max))
    (delete-matching-lines "^$")
    (set-fill-column save-fill-column) ))

So simple! Why didn't I think of doing it that way before?

Tags: emacs, editors
[ 20:10 Feb 19, 2007    More linux/editors | permalink to this entry | comments ]

Linkifying with Regular Expressions

I had a page of plaintext which included some URLs in it, like this:

Tour of the Hayward Fault
http://www.mcs.csuhayward.edu/~shirschf/tour-1.html

Technical Reports on Hayward Fault
http://quake.usgs.gov/research/geology/docs/lienkaemper_docs06.htm

I wanted to add links around each of the urls, so that I could make it part of a web page, more like this:

Tour of the Hayward Fault
http://www.mcs.csu hayward.edu/~shirschf/tour-1.html

Technical Reports on Hayward Fault
htt p://quake.usgs.gov/research/geology/docs/lienkaemper_docs06.htm

Surely there must be a program to do this, I thought. But I couldn't find one that was part of a standard Linux distribution.

But you can do a fair job of linkifying just using a regular expression in an editor like vim or emacs, or by using sed or perl from the commandline. You just need to specify the input pattern you want to change, then how you want to change it.

Here's a recipe for linkifying with regular expressions.

Within vim:

:%s_\(https\=\|ftp\)://\S\+_<a href="&">&</a>_

If you're new to regular expressions, it might be helpful to see a detailed breakdown of why this works:

:

Tell vim you're about to type a command.

%

The following command should be applied everywhere in the file.

s_

Do a global substitute, and everything up to the next underscore will represent the pattern to match.

\(

This will be a list of several alternate patterns.

http

If you see an "http", that counts as a match.

s\=

Zero or one esses after the http will match: so http and https are okay, but httpsssss isn't.

\|

Here comes another alternate pattern that you might see instead of http or https.

ftp

URLs starting with ftp are okay too.

\)

We're done with the list of alternate patterns.

://

After the http, https or ftp there should always be a colon-slash-slash.

\S

After the ://, there must be a character which is not whitespace.

\+

There can be any number of these non-whitespace characters as long as there's at least one. Keep matching until you see a space.

_

Finally, the underscore that says this is the end of the pattern to match. Next (until the final underscore) will be the expression which will replace the pattern.

<a href="&">

An ampersand, &, in a substitute expression means "insert everything that was in the original pattern". So the whole url will be inserted between the quotation marks.

&</a>

Now, outside the <a href="..."> tag, insert the matched url again, and follow it with a </a> to close the tag.

_

The final underscore which says "this is the end of the replacement pattern". We're done!

Linkifying from the commandline using sed

Sed is a bit trickier: it doesn't understand \S for non-whitespace, nor = for "zero or one occurrence". But this expression does the trick:
sed -e 's_\(http\|https\|ftp\)://[^ \t]\+_<a href="&">&</a>_' <infile.txt >outfile.html

Addendum: George Riley tells me about VST for Vim 7, which looks like a nice package to linkify, htmlify, and various other useful things such as creating HTML presentations. I don't have Vim 7 yet, but once I do I'll definitely check out VST.

Tags: linux, editors, pipelines, regexp, shell, CLI
[ 12:40 May 14, 2006    More linux/editors | permalink to this entry | comments ]

Emacs: Typing dashes in html mode

What to do with a few extra hours in a boring motel with no net access? How about digging into fixing one of Emacs' more annoying misfeatures?

Whenever I edit an html file using emacs, I find I have to stay away from double dashes -- I can't add a phrase such as this one. If I forget and type a phrase with a double dash, then as soon as I get to the end of that line and emacs decides it's time to wrap to the next line, it "helpfully" treats the double dashes as a comment, and indents the next line to the level where the dashes were, adding another set of dashes. I've googled, I've asked on emacs IRC help channels, but there doesn't seem to be any way out. (I guess no one else ever uses double dashes in html files?)

It's frustrating: I like using double dashes now and then. And aside from the occasional boneheaded misfeature like this one, I like using emacs. But the dash problem been driving me nuts for a long time now. So I finally dug into the code to cure it.

First, the file is sgml-mode.el, so don't bother searching anything with html in the name. On my system it's /usr/share/emacs/21.4/lisp/textmodes/sgml-model.el. Edit that file and search for "--" and the first thing you'll find (well, after the file's preamble comments) is a comment in the definition of "sgml-specials" saying that if you include ?- in the list of specials, it will hork the typing of double dashes, so that's normally left out.

A clue! Perhaps some Debian or Ubuntu site file has changed sgml-specials for me, and all I need to do is change it back! So I typed

M-x describe-variable sgml-specials

to see the current setting.

Um ... it's set to "34". That's not very helpful. I haven't a clue how that translates to the list of characters I see in sgml-mode.el. Forget about that approach for now.

Searching through the file for the string "comment" got me a few more hits, and I tried commenting out various comment handling lines until the evil behavior went away. (I had to remove sgml-mode.elc first, otherwise emacs wouldn't see any changes I made to sgml-mode.el. If you haven't done much elisp hacking, the .el is the lisp source, while the .elc is a byte-compiled version which loads quicker but isn't intended to be edited by humans. For Java programmers, the .elc is sort of like a .class file.)

Commenting out these four lines did the trick:

  (set (make-local-variable 'font-lock-syntactic-keywords)
       '(("\\(<\\)! *--.*-- *\\(>\\)" (1 "!") (2 "!"))))
  ;; This will allow existing comments within declarations to be
  ;; recognized.
  (set (make-local-variable 'comment-start-skip) "\\(?:\\)?")

To regenerate the .elc file so sgml-mode will load faster, I ran emacs as root from the directory sgml-mode.el was in, and typed:

M-x byte-compile-file sgml-mode.el

All better! And now I know where to find documentation for all those useful-looking, but seemingly undocumented, keyboard shortcuts that go along with emacs' html mode. Just search in the file for html-mode-map, and you'll find all sorts of useful stuff.

For instance, that typing Ctrl-C Ctrl-C followed by various letters: u gets you an unordered list, h gets you an href tag, i an image tag, and so on, with the cursor positioned where you want to type next.

It doesn't seem to offer any basic inline formatting (like <i> or <em>), alas; but of course that's easy to add by editing the file (or maybe even in .emacs). To add an <em> tag, add this line to html-mode-map:

    (define-key map "\C-c\C-ce" 'html-em)

then add this function somewhere near where html-headline-1 and friends are defined:

(define-skeleton html-em
  "HTML emphasis tags."
  nil
  "" _ "")

Of course, you can define any set of tags you use often, not just <em>.

HTML mode in emacs should be much more fun and less painful now!

Update: If you don't want to modify the files as root, it also works fine to copy sgml-mode.el to wherever you keep personal elisp files. For instance, put them in a directory called ~/.emacs-lisp then add this to your .emacs:
(setq load-path (cons "~/.emacs-lisp/" load-path))

Tags: emacs, editors
[ 21:48 Mar 29, 2006    More linux/editors | permalink to this entry | comments ]

Helpful Vim Tip: Finding Syntax for Colors

An upgrade from woody to sarge introduced a new problem with editing mail messages in vim: Subject lines appeared in yellow, against my light grey background, so they weren't readable any more.

Vim color files have always been a mystery to me. I have one which I adapted from one of the standard color schemes, but I've never been clear what the legal identifiers are or how to find out. But I changed both places where it said "ctermfg=Yellow" to another color, and nothing changed, so this time I had to find out.

Fortunately a nice person on #vim suggested :he synID (he is short for "help", of course) which told me all I needed to know. Put the cursor on the errant line and type: :echo synIDattr(synID(line("."), col("."), 1), "name")

That told me that the Subject line was syntax class "mailSubject". So I tried (copying other lines in my color file) adding this line:

hi mailSubject term=underline ctermfg=Red guifg=Red

and now all is happy again in vim land. I wish I'd learned that synID trick a long time ago!

Tags: vim, color, editors, tips
[ 09:59 Jun 22, 2005    More linux/editors | permalink to this entry | comments ]

Tweaking Emacs' Text Indent: Don't Indent So Aggressively

Encouraged by my success a few days ago at finally learning how to disable vim's ctrl-spacebar behavior, the next day I went back to an emacs problem that's been bugging me for a while: in text mode, newline-and-indent always wants to indent the first line of a text file (something I almost never want), and skips blank lines when calculating indent (so starting a new paragraph doesn't reset the indent back to zero).

I had already googled to no avail, and had concluded that the only way was to write a new text-indent function which could be bound to the return key in the text mode hook.

This went fairly smoothly: I got a little help in #emacs with checking the pattern immediately before the cursor (though I turned out not to need that after all) and for the function called "bobp" (beginning of buffer predicate). Here's what I ended up with:

(defun newline-and-text-indent ()
  "Insert a newline, then indent the next line sensibly for text"
  (interactive)
  (if (or (bobp)
          (looking-at "^$"))
      (newline)
      (newline-and-indent)
  ))
(defun text-indent-hook ()
  (local-set-key "\C-m" 'newline-and-text-indent)
  )
(setq text-mode-hook 'text-indent-hook)

It seems to work fine. For the curious, here's my current .emacs

Tags: emacs, editors
[ 13:03 Feb 19, 2005    More linux/editors | permalink to this entry | comments ]

Turning off Ctrl-Space in Vim

One of those niggling problems that has plagued me for a long time: in the editor vim, if I'm typing along in insert mode and instead of a space I accidentally hit control-space, vim inserts a bunch of text I didn't want, then exits insert mode. Meanwhile I'm still merrily typing away, typing what are now vim comments which invariably end up deleting the last two paragraphs I typed then doing several more operations which end up erasing the undo buffer so I can't get those paragraphs back.

Ctrl-space inserts a null character (you can verify this by putting it in a file and running od -xc on it). I've done lots of googling in the past, but it's hard to google on strings like " " or even "space" or "null", and nobody I asked had a clue what this function was called (it turns out it re-inserts whatever the last inserted characters were) so I couldn't google on the function name.

Vim's help suggests that <Nul>, <Char-0>, or <C-^V > should do it. I tried them with map, vmap, cmap, and nmap, to no avail. I also tried <C-@> since that's a historical way of referring to the null character, googling found some references to that in vim, and that's how it displays if I type it in vim.

I finally found #vim on freenode, and asked there. Last night nobody knew, but this morning, p0g found the problem: I needed to use imap, not the map/vmap/cmap/nmap I'd been using.

So here, perserved for google posterity in case other people are plagued by this problem, is the answer:

imap <Nul> <Space>

For good measure, I also mapped the character to no-op in all the other modes as well:

map  <Nul> <Nop>
vmap <Nul> <Nop>
cmap <Nul> <Nop>
nmap <Nul> <Nop>

My current .vimrc.

Tags: vim, editors, tips
[ 10:24 Feb 17, 2005    More linux/editors | permalink to this entry | comments ]

Emacs Color Themes

A nifty emacs trick I learned about today: ColorThemes.

Instead of the old hacked-together color collection I've been using in emacs, I can load color-theme.el and choose from lots of different color schemes.

I added these lines to .emacs:

(require 'font-lock)
(if (fboundp 'global-font-lock-mode) (global-font-lock-mode 1))
(load "~/.emacs-lisp/color-theme.el")
(color-theme-ramangalahy)  ;; pick a favorite theme

The disadvantage is that color-theme.el is fifteen thousand lines long! So I'll probably make a local version that strips out all but the theme I actually use (then I can customize that).

The (global-font-lock-mode 1) tells emacs to use syntax highlighting on every file, not just certain types. So now I get at least some highlighting even in html files, though it still doesn't seem to be able to highlight like vim does (e.g. different colors for text inside <b> or <b> tags).

Tags: emacs, editors
[ 17:57 Feb 03, 2005    More linux/editors | permalink to this entry | comments ]

Nifty emacs hack: how to wrap only certain files

For a long time I've wanted some, but not all, text and html files to line-wrap automatically in emacs. For instance, it drives me nuts when I edit a system configuration file and it wraps each long line, or when I edit an html file containing lots of long links and it keeps wrapping between the <a and the href=. But for files which are mostly text (such as these blog entries), I want line wrapping.

I'd been trying to do this with html-mode-hook and text-mode-hook, then checking the filename and calling (auto-fill-mode) if appropriate, but it wasn't working, because buffer-file-name isn't always defined at the time the mode hook is called. (No one seems to know why.) The buffer name seems to be defined at that point, but it doesn't contain path information so I can't say "Use wrapping for anything under ~/Docs" or "Don't wrap anything in /etc".

But with some help from sachac and the nice folks on #emacs I came up with a much better solution, and it's way simpler than the mode-hook approach: derived modes.

I set up two new modes, called html-wrap-mode and text-wrap-mode, which are the same as html-mode and text-mode except that they turn on auto-fill. Then I use the easy auto-mode-alist mechanism, which already does string matching on the filename, to call these modes, instead of the regular text and html modes, based on the extension or some other aspect of the file's pathname. Here's what I added to .emacs:

;; Want auto-fill-mode for some text and html files, but not all.
;; So define two derived modes for that, and we'll use auto-mode-alist
;; to choose them based on filename.
(define-derived-mode html-wrap-mode html-mode "HTML wrap mode"
  (auto-fill-mode))
(define-derived-mode text-wrap-mode text-mode "Text wrap mode"
  (auto-fill-mode))

(setq auto-mode-alist
      (cons '("\\.blx$" . html-wrap-mode)
      (cons '("Docs/.*.html$" . html-wrap-mode)
      (cons '("Docs/" . text-wrap-mode)
            auto-mode-alist) ) ) )

Here's my current .emacs.

I wonder if vim has a way to do this?

Tags: editors, emacs
[ 22:30 Jan 13, 2005    More linux/editors | permalink to this entry | comments ]