Shallow Thoughts : tags : editors

Akkana's Musings on Open Source Computing and Technology, Science, and Nature.

Sun, 14 Sep 2014

Global key bindings in Emacs

Global key bindings in emacs. What's hard about that, right? Just something simple like

(global-set-key "\C-m" 'newline-and-indent)
and you're all set.

Well, no. global-set-key gives you a nice key binding that works ... until the next time you load a mode that wants to redefine that key binding out from under you.

For many years I've had a huge collection of mode hooks that run when specific modes load. For instance, python-mode defines \C-c\C-r, my binding that normally runs revert-buffer, to do something called run-python. I never need to run python inside emacs -- I do that in a shell window. But I fairly frequently want to revert a python file back to the last version I saved. So I had a hook that ran whenever python-mode loaded to override that key binding and set it back to what I'd already set it to:

(defun reset-revert-buffer ()
  (define-key python-mode-map "\C-c\C-r" 'revert-buffer) )
(setq python-mode-hook 'reset-revert-buffer)

That worked fine -- but you have to do it for every mode that overrides key bindings and every binding that gets overridden. It's a constant chase, where you keep needing to stop editing whatever you wanted to edit and go add yet another mode-hook to .emacs after chasing down which mode is causing the problem. There must be a better solution.

A web search quickly led me to the StackOverflow discussion Globally override key bindings. I tried the techniques there; but they didn't work.

It took a lot of help from the kind folks on #emacs, but after an hour or so they finally found the key: emulation-mode-map-alists. It's only barely documented -- the key there is "The “active” keymaps in each alist are used before minor-mode-map-alist and minor-mode-overriding-map-alist" -- and there seem to be no examples anywhere on the web for how to use it. It's a list of alists mapping names to keymaps. Oh, clears it right up! Right?

Okay, here's what it means. First you define a new keymap and add your bindings to it:

(defvar global-keys-minor-mode-map (make-sparse-keymap)
  "global-keys-minor-mode keymap.")

(define-key global-keys-minor-mode-map "\C-c\C-r" 'revert-buffer)
(define-key global-keys-minor-mode-map (kbd "C-;") 'insert-date)

Now define a minor mode that will use that keymap. You'll use that minor mode for basically everything.

(define-minor-mode global-keys-minor-mode
  "A minor mode so that global key settings override annoying major modes."
  t "global-keys" 'global-keys-minor-mode-map)

(global-keys-minor-mode 1)

Now build an alist consisting of a list containing a single dotted pair: the name of the minor mode and the keymap.

;; A keymap that's supposed to be consulted before the first
;; minor-mode-map-alist.
(defconst global-minor-mode-alist (list (cons 'global-keys-minor-mode
                                              global-keys-minor-mode-map)))

Finally, set emulation-mode-map-alists to a list containing only the global-minor-mode-alist.

(setf emulation-mode-map-alists '(global-minor-mode-alist))

There's one final step. Even though you want these bindings to be global and work everywhere, there is one place where you might not want them: the minibuffer. To be honest, I'm not sure if this part is necessary, but it sounds like a good idea so I've kept it.

(defun my-minibuffer-setup-hook ()
  (global-keys-minor-mode 0))
(add-hook 'minibuffer-setup-hook 'my-minibuffer-setup-hook)

Whew! It's a lot of work, but it'll let me clean up my .emacs file and save me from endlessly adding new mode-hooks.

Tags: , ,
[ 16:46 Sep 14, 2014    More linux/editors | permalink to this entry | comments ]

Sun, 15 Jun 2014

Vim: Set wrapping and indentation according to file type

Although I use emacs for most of my coding, I use vim quite a lot too, for quick edits, mail messages, and anything I need to edit when logged onto a remote server. In particular, that means editing my procmail spam filter files on the mail server.

The spam rules are mostly lists of regular expression patterns, and they can include long lines, such as:
gift ?card .*(Visa|Walgreen|Applebee|Costco|Starbucks|Whitestrips|free|Wal.?mart|Arby)

My default vim settings for editing text, including line wrap, don't work if get a flood of messages offering McDonald's gift cards and decide I need to add a "|McDonald" on the end of that long line.

Of course, I can type ":set tw=0" to turn off wrapping, but who wants to have to do that every time? Surely vim has a way to adjust settings based on file type or location, like emacs has.

It didn't take long to find an example of Project specific settings on the vim wiki. Thank goodness for the example -- I definitely wouldn't have figured that syntax out just from reading manuals. From there, it was easy to make a few modifications and set textwidth=0 if I'm opening a file in my procmail directory:

" Set wrapping/textwidth according to file location and type
function! SetupEnvironment()
  let l:path = expand('%:p')
  if l:path =~ '/home/akkana/Procmail'
    " When editing spam filters, disable wrapping:
    setlocal textwidth=0
endfunction
autocmd! BufReadPost,BufNewFile * call SetupEnvironment()

Nice! But then I remembered other cases where I want to turn off wrapping. For instance, editing source code in cases where emacs doesn't work so well -- like remote logins over slow connections, or machines where emacs isn't even installed, or when I need to do a lot of global substitutes or repetitive operations. So I'd like to be able to turn off wrapping for source code.

I couldn't find any way to just say "all source code file types" in vim. But I can list the ones I use most often. While I was at it, I threw in a special wrap setting for mail files:

" Set wrapping/textwidth according to file location and type
function! SetupEnvironment()
  let l:path = expand('%:p')
  if l:path =~ '/home/akkana/Procmail'
    " When editing spam filters, disable wrapping:
    setlocal textwidth=0
  elseif (&ft == 'python' || &ft == 'c' || &ft == 'html' || &ft == 'php')
    setlocal textwidth=0
  elseif (&ft == 'mail')
    " Slightly narrower width for mail (and override mutt's override):
    setlocal textwidth=68
  else
    " default textwidth slightly narrower than the default
    setlocal textwidth=70
  endif
endfunction
autocmd! BufReadPost,BufNewFile * call SetupEnvironment()

As long as we're looking at language-specific settings, what about doing language-specific indentation like emacs does? I've always suspected vim must have a way to do that, but it doesn't enable it automatically like emacs does. You need to set three variables, assuming you prefer to use spaces rather than tabs:

" Indent specifically for the current filetype
filetype indent on
" Set indent level to 4, using spaces, not tabs
set expandtab shiftwidth=4

Then you can also use useful commands like << and >> for in- and out-denting blocks of code, or ==, for indenting to the right level. It turns out vim's language indenting isn't all that smart, at least for Python, and gets the wrong answer a lot of them time. You can't rely on it as a syntax checker the way you can with emacs. But it's a lot better than no language-specific indentation.

I will be a much happier vimmer now!

Tags: , ,
[ 11:29 Jun 15, 2014    More linux/editors | permalink to this entry | comments ]

Tue, 29 Apr 2014

The evil HTML double-dash problem in Emacs is still there

Long ago (in 2006!), I blogged on an annoying misfeature of Emacs when editing HTML files: you can't type double dashes. Emacs sees them as an SGML comment and insists on indenting all subsequent lines in strange ways.

I wrote about finding a fix for the problem, involving commenting out four lines in sgml-mode.el. That file had a comment at the very beginning suggesting that they know about the problem and had guarded against it, but obviously it didn't work and the variable that was supposed to control the behavior had been overridden by other hardwired behaviors.

That fix has worked well for eight years. But just lately, I've been getting a lot of annoying warnings when I edit HTML files: "Error: autoloading failed to define function sgml_lexical_context". Apparently the ancient copy of sgml-mode.el that I'd been using all these years was no longer compatible with ... something else somewhere inside emacs. I needed to update it.

Maybe, some time during the intervening 8 years, they'd actually fixed the problem? I was hopeful. I moved my old patched sgml-mode.el aside and edited some files. But the first time I tried typing a double dashes -- like this, with text inside that's long enough to wrap to a new line -- I saw that the problem wasn't fixed at all.

I got a copy of the latest sgml-mode.el -- on Debian, that meant:

apt-get install emacs23-el
cp /usr/share/emacs/23.4/lisp/textmodes/sgml-mode.el.gz ~/.emacs-lisp
gunzip ~/.emacs-lisp/sgml-mode.el.gz
Then I edited the file and started searching for strings like font-lock and comment.

Unfortunately, the solution I documented in my old blog post is no longer helpful. The code has changed too much, and now there are many, many different places where automatic comment handling happens. I had to comment out each of them bit by bit before I finally found the section that's now causing the problem. Commenting out these lines fixed it:

   (set (make-local-variable 'indent-line-function) 'sgml-indent-line)
   (set (make-local-variable 'comment-start) "")
   (set (make-local-variable 'comment-indent-function) 'sgml-comment-indent)
   (set (make-local-variable 'comment-line-break-function)
        'sgml-comment-indent-new-line)

I didn't have to remove any .elc files, like I did in 2006; just putting the sgml-mode.el file in my Emacs load-path was enough. I keep all my customized Emacs code in a directory called .emacs-lisp, and in my .emacs I make sure it's in my path:

(setq load-path (cons "~/.emacs-lisp/" load-path))
And now I can type double dashes again. Whew!

Tags: , ,
[ 12:42 Apr 29, 2014    More linux/editors | permalink to this entry | comments ]

Wed, 05 Jun 2013

Stop Emacs from invoking a browser

After upgrading my OS (in this case, to Debian sid), I noticed that my browser window kept being replaced with an HTML file I was editing in emacs. I'd hit Back or close the tab, and the next time I checked, there it was again, my HTML source.

I'm sure it's a nice feature that emacs can show me my HTML in a browser. But it's not cool to be replacing my current page without asking. How do I turn it off? A little searching revealed that this was html-autoview-mode, which apparently at some point started defaulting to ON instead of OFF. Running M-x html-autoview-mode toggles it back off for the current session -- but that's no help if I want it off every time I start emacs.

I couldn't find any documentation for this, and the obvious (html-autoview-mode nil) in .emacs didn't work -- first, it gives a syntax error because the function isn't defined until after you've loaded html-mode, but even if you put it in your html-mode hook, it still doesn't work.

I had to read the source of sgml-mode.el. (M-x describe-function html-autoview-mode also would have told me, if I had already loaded html-mode, but I didn't realize that until later.) Turns out html-autoview-mode turns off if its argument is negative, not nil. So I added it to my html derived mode:

(define-derived-mode html-wrap-mode html-mode "HTML wrap mode"
  (auto-fill-mode)
  ;; Don't call an external browser every time you save an html file:
  (html-autoview-mode -1)
)

Tags: , ,
[ 22:48 Jun 05, 2013    More linux/editors | permalink to this entry | comments ]

Sat, 19 Jan 2013

Converting C to Python with a vi regexp

I'm fiddling with a serial motor controller board, trying to get it working with a Raspberry Pi. (It works nicely with an Arduino, but one thing I'm learning is that everything hardware-related is far easier with Arduino than with RPi.)

The excellent Arduino library helpfully provided by Pololu has a list of all the commands the board understands. Since it's Arduino, they're in C++, and look something like this:

#define QIK_GET_FIRMWARE_VERSION         0x81
#define QIK_GET_ERROR_BYTE               0x82
#define QIK_GET_CONFIGURATION_PARAMETER  0x83
[ ... ]
#define QIK_CONFIG_DEVICE_ID                        0
#define QIK_CONFIG_PWM_PARAMETER                    1
and so on.

On the Arduino side, I'd prefer to use Python, so I need to get them to look more like:

    QIK_GET_FIRMWARE_VERSION = 0x81
    QIK_GET_ERROR_BYTE = 0x82
    QIK_GET_CONFIGURATION_PARAMETER = 0x83
[ ... ]
    QIK_CONFIG_DEVICE_ID = 0
    QIK_CONFIG_PWM_PARAMETER = 1
... and so on ... with an indent at the beginning of each line since I want this to be part of a class.

There are 32 #defines, so of course, I didn't want to make all those changes by hand. So I used vim. It took a little fiddling -- mostly because I'd forgotten that vim doesn't offer + to mean "one or more repetitions", so I had to use * instead. Here's the expression I ended up with:

.,$s/\#define *\([A-Z0-9_]*\) *\(.*\)/ \1 = \2/

In English, you can read this as:

From the current line to the end of the file (,.$/), look for a pattern consisting of only capital letters, digits and underscores ([A-Z0-9_]). Save that as expression #1 (\( \)). Skip over any spaces, then take the rest of the line (.*), and call it expression #2 (\( \)).

Then replace all that with a new line consisting of 4 spaces, expression 1, a spaced-out equals sign, and expression 2 ( \1 = \2).

Who knew that all you needed was a one-line regular expression to translate C into Python?

(Okay, so maybe it's not quite that simple. Too bad a regexp won't handle the logic inside the library as well, and the pin assignments.)

Tags: , , , , ,
[ 21:38 Jan 19, 2013    More linux/editors | permalink to this entry | comments ]

Sat, 12 Jan 2013

Integrating graphics with text in Emacs

I discussed Emacs's artist-mode a few days ago as a simple, but incomplete, solution to the problem of sketching graphs while taking notes during a math class. But I've found a much better way, one that allows for including any images -- drawings, photos, or screenshots. It took a little work and some custom .emacs code, but I love the result.

Iimage mode

[iimage-mode: images displayed inline in Emacs] The key is iimage-mode, which displays inline images. In this mode, you put a line in your buffer with a reference to your image file, something like this:

file://myimage.jpg
and Emacs will replace it with the contents of that image. Marvellous!

You can use other patterns for filenames as well, but I'm fine with using URLs. Note there are only two slashes in file:// -- it's a local file in the same directory as the text file being edited.

It's a little tricky to enable it. The docs are not entirely clear on the differences between iimage-mode, turn-on-iimage-mode and iimage-mode-buffer. I found I could get a file that already had existing images to display them with:

  (turn-on-iimage-mode)
  (iimage-mode-buffer t)

Very cool! But too much to type every time. And to use it for note-taking, I needed a way to say, "Create a new image here, let me edit it, then display the image I just edited inline."

Enabling iimage-mode automatically

First, I wanted iimage mode displayed automatically on files in my note-taking directories. I normally use text-mode for these files, with spell checking and line wrapping turned on (auto-fill mode). So I defined a new minor mode based on text-mode:

(define-derived-mode text-img-mode text-mode "Image display mode"
  (auto-fill-mode)
  (turn-on-iimage-mode)
  (iimage-mode-buffer t)
  )

Then I wanted this mode to be called whenever I'm editing a file in my classes directory. So I added it to my auto-mode-alist:

(setq auto-mode-alist
   ...
      (cons '("Docs/classes/" . text-img-mode)
   ...
      auto-mode-alist) )

Inserting a new image

Next, I needed a way to insert an image URL into the buffer and call up an image editor on it. I shouldn't have to type the filename twice and keep track of it; that's what computers are for.

And I needed a drawing program. As a longtime GIMP geek, most of my computer drawing has been in GIMP. But GIMP is overkill for calling up a quick sketch window. I was tempted to use TuxPaint; it's a good sketching app even if you're not five years old, and it's fun and easy to use. But by default, TuxPaint has some features that get in the way of note-taking, like distracting sound effects. I'm sure it's possible to turn those off, and I do plan to investigate that.

I saw a reference to pinta as a lightweight drawing app, but it required a boatload of Mono libraries that I don't otherwise need, and Krita has the same problem with KDE services. So I opted for MyPaint. It works okay, though it's rather slow to start up and has some other issues, so I'm still hoping to find a more lightweight sketching app.

In any case, I fiddled around with start-process until I figured out how to use it to start a program. Then I wrote a little function that lets the user pick a filename, inserts a URL to that filename into the buffer, then calls up mypaint on the file.

(defun img ()
  "Prompt for a filename, then call up mypaint to create an image"
  (interactive)
  (let ((imgfile (read-string "Filename? " "xxx.jpg" 'my-history)))
    (insert "\nfile://" imgfile "\n" )
    (start-process "mypaint" nil "/usr/bin/mypaint" imgfile)
  ))

Worked fine! I can run M-x img, be prompted for a filename, and get a mypaint window where I can make my sketch.

Noticing that a new image has been added

But wait. I finish sketching, write the file and quit mypaint ... and the buffer still shows something like file://xxx.jpg, even if it's showing other images inline. I needed a way to tell it to refresh and load any new images. (I considered having emacs wait for mypaint to exit, but decided I might sometimes want to keep editing while mypaint was still up.)

M-x eval-expression (iimage-mode-buffer t) will do that, but that's a lot of typing to do. Obviously, I needed a key binding.

Strangely enough, C-c i wasn't taken for text buffers, so that seemed like a natural. So I added a key binding to the end of the text-img-mode. iimage-mode-buffer requires that t argument -- it gives an error without it -- so the key binding looks a little more complicated than one that just calls a simple function. I added it to the end of my text-img-mode function.

(define-derived-mode text-img-mode text-mode "Image display mode"
 ...
  (local-set-key "\C-ci" 
    (lambda () (interactive) (iimage-mode-buffer t)))
  )

But after using it a bit, I discovered that this didn't reload images if I edited them a second time. Fortunately, vwood had the answer:

(defun refresh-iimages ()
  "Only way I've found to refresh iimages (without also recentering)"
  (interactive)
  (clear-image-cache nil)
  (iimage-mode nil)
  (iimage-mode t)
  (message "Refreshed images")
 )

I added the message at the end, since otherwise the function left a distracting "Toggling iimage-mode off; better pass an explicit argument" error.

Then the key binding in my text-img-mode became

(local-set-key "\C-ci" 'refresh-iimages)

Inserting a screenshot

Wait -- one more thing. As I actually used text-img-mode to take notes, I discovered that taking screenshots would actually be much more useful than making my own drawings. Then I could copy small sections of the slides and graphs into my notes at the appropriate place, without needing to copy equations at all.

Why not write a function to allow that? The unpleasantly named scrot program fills the bill nicely, and gives me a choice of clicking in a window or dragging out an area of the screen.

(defun screenshot ()
 "Prompt for a filename, then call up scrot to create an interactive screenshot"
  (interactive)
  (let ((imgfile (read-string "Filename? " "scr.jpg" 'my-history)))
    (insert "\nfile://" imgfile "\n" )
    (start-process "scrot" nil "/usr/bin/scrot" "-s" imgfile)
  ))

This turned out to be so useful that I added a key for it in text-img-mode:

(local-set-key "\C-cs" 'screenshot)

I'm so happy with the result! Iimage mode is working great, and having text and images together is turning out to be perfect for note-taking.

My only problem now -- okay, I admit it -- is a tendency to get so excited over inserting screenshots that I get distracted and forget to actually listen to the lecture. I'm sure I'll get over that, but for now, Thank goodness vlc is good at skipping back!

Tags: , , ,
[ 13:42 Jan 12, 2013    More linux/editors | permalink to this entry | comments ]

Thu, 10 Jan 2013

ASCII graphics in Emacs with Artist Mode

I found a cool package in Emacs the other day: Artist Mode. It lets you draw ASCII graphics in your text file.

I was actually looking for the solution to a different problem: taking notes in a math-intensive Coursera class. I've been taking notes in Emacs, but a text editor is awkward for equations and even more awkward for graphs.

What I really wanted was something like the old Claris Works (or so I'm told; I never used it myself) -- something that's primarily a text editor but lets you drawings, equations, and tables when you need to. In theory, word processors like LibreOffice could do that, but in practice they're not very good at switching modes, nor at integrating several types of media into one document. Texmacs is great for the equations and apparently it can do tables too, but it can't do freehand drawing.

And none of these programs is very configurable -- I can't use my fast, comfortable Emacs bindings while typing, and that's a deal-breaker for me, because being able to make corrections quickly makes a huge difference in my typing speed. LibreOffice's key bindings are only partially configurable, and after you've spent half a day chasing down all the action names you need (the ones that are actually available), you upgrade to a newer version and discover you have to do it all over again because there's no way to migrate configuration files. Even Texmacs, ironically, is no better: the documentation claims it's possible to configure key bindings, but it doesn't appear anyone has ever succeeded in figuring out how.

Anyway, ASCII graphics aren't the ultimate solution to note-taking. And I've found a better solution for that, while I'll write about separately. But for now, Artist Mode is just so cool I had to share it.

Enable it by running M-x artist-mode. You can immediately start drawing in your buffer with the mouse. Whatever you draw gets turned into ASCII graphics.

For note-taking, it's fine for scribbling the rough shape of a curve. It takes no time to mouse in a little sketch like

       |                               .. 20
       |                              ..
       |                            ...
    10 |.                          ..
       |..                       ...
       | ...                   ...
       |   ..              ....
       |    .....      .....
       |    ............
       |.....        .............
       +-------------------------------------

It even has primitives (middleclick to get a.menu) for things like lines, rectangles and circles, and for filling regions. When you're done drawing, M-x artist-mode goes back to whatever mode you were using before.

I probably won't use it very much for note taking. But there are times when I've wanted to draw ASCII graphics -- a laborious process in ordinary text modes -- and other times when it would just be fun to play around with my buffer. I'm happy to know about Artist Mode. I may not need it often, but it sure is fun to use now and then.

Tags: , , ,
[ 20:02 Jan 10, 2013    More linux/editors | permalink to this entry | comments ]

Wed, 02 Jan 2013

Customize the Emacs modeline color

[Emacs with colored mode lines]

I wrote last week about how to customize syntax highlighting colors in Emacs. And as part of that, I ditched the color theme I'd been using and let Emacs go back to its default colors.

Which mostly was fine, except that when I split the window into two windows, to look at two files at once or two different parts of the same file, the separator between the two windows -- the mode line -- was the same grey as Emacs's normal background, so it wasn't very obvious where the window split was.

A web search turned out lots of different ways to set the mode line color. Many of them involve color themes and are fairly complicated. Here's the simplest method I found:

(set-face-foreground 'modeline "white")
(set-face-background 'modeline "purple")
(set-face-background 'modeline-inactive "light blue")

You can set your active mode line to a pretty color, so it stands out a bit and makes it easy to tell which of the visible windows is the one you're actually typing in, and set the inactive mode lines -- windows that are visible but you arne't actually typing in -- to a less striking color.

Tags: ,
[ 13:50 Jan 02, 2013    More linux/editors | permalink to this entry | comments ]

Sun, 23 Dec 2012

Customizing syntax highlighting colors in Emacs

Emacs has wonderful syntax highlighting. Words will be displayed in different colors depending on their syntax and the mode of the current file -- for instance, in C code, keywords of the language are highlighted in one color, comments in another, strings in a third.

The problem comes when the colors aren't right. Like that awful gold color that the flyspell spell checker uses for some words. Against a light background it makes the words almost impossible to read. I've struggled for years trying to set up custom color schemes to get around that problem, but I finally learned a simpler way to handle it when you see something in a color you want to change.

The trick is to find out what face you need to change. "Face" to emacs means more than a font face like Sans or Lucida; it means a collection of information about how characters are displayed, including font face, weight, slant, color and other attributes.

[Emacs' customize-face screen] When you see something displayed in a color you don't want, place the cursor somewhere in the word. type C-u C-x = will get you the face used, along with all sorts of information about it and a handy Customize what to show link. Or you can go straight to the Customize screen with M-x customize-face -- hit return to customize the face at point (the cursor location).

In the customize-face screen, there's no GUI to choose colors, but you can edit color names. Emacs lists "red" as the foreground color; if you change it to "blue" you'll see a preview of how it will look. Color names come from /etc/X11/rgb.txt, and there are various programs like xcolorsel that will show them -- or better yet, see Wikipedia's X11 color names chart.

Once you've chosen a color, the Save for future sessions button will add a section to your .emacs file with the appropriate elisp code. Of course, you can move this code elsewhere as well. I have a somewhat complex .emacs setup, so I've moved the code into another file. Strangely, I found that my .Xdefaults background color setting no longer worked once I started using custom-set-faces, so I added a line for that as well.

(set-background-color "grey90")
(custom-set-faces
 '(flyspell-duplicate ((((class color)) (:foreground "red" :underline t :weight bold))))
 '(font-lock-comment-face ((((class color) (min-colors 88) (background light)) (:foreground "blue"))))
 )

Tags: ,
[ 13:59 Dec 23, 2012    More linux/editors | permalink to this entry | comments ]

Sat, 21 Jul 2012

Vim tip: opening lines without delay

I use vim to reply to email messages in mutt. And something I do very often is to open some new lines between quoted sections. For instance, if I'm replying to mail with a section like

> Have you noticed anything weird about squirrels lately?
> I think they're plotting something.
I might want to open up space after the first line, and say
> Have you noticed anything weird about squirrels lately?

You too? One of our local squirrels ran off with my Arduino yesterday.

> I think they're plotting something.
So I need three blank lines, with the cursor on the middle line.

In vim, typically I would place the cursor on the line before the space I want to open, and type o<RETURN><ESC>O :

o
open a line below the current line
<RETURN>
make an empty line below the quoted line
<ESC>
get out of insert mode
O
open a new line above the current line and enter insert mode there

Sounds complicated, but it's pretty fast to type o<RET><ESC>O especially when you do it many times a day.

Except vim has an odd quirk: after you type that O, you have to wait a second or two. If you start typing immediately, somehow you're not in insert mode any more, and the line you just opened, and thought you were typing in, disappears.

Well, it turns out this happens because <ESC>O is a sequence that can also happen from arrow keys. So vim has a timeout it uses to figure out whether an <ESC>O sequence was from an arrow key, or two characters typed by a real person. If the O follows too soon after the <ESC>, it figures it's an arrow key. (This is discussed in more detail in this Stack Overflow thread: Vim: Delay before 'O' opens a new line?)

It turns out there's a simple solution: set the timeout much shorter, like 100 milliseconds, in .vimrc:

set ttimeoutlen=100

Then arrow keys should still work -- unless you typically use vim over a very slow connection -- but you can type the sequence yourself (unless you type very fast) and not have it confused with an arrow key.

But really, there's a better solution. Since opening up space like this is something I do so often, why not make a vim binding for it?

map <leader>o o<CR><ESC>O

Now all I have to do is type \o (backslash, \, is the "leader" character) and I get a space opened up, with a blank line both before it and after it.

Tags: ,
[ 12:20 Jul 21, 2012    More linux/editors | permalink to this entry | comments ]

Mon, 16 Jul 2012

Editing tab-separated files with Emacs

I wanted to keep maintenance records for my car in a local file. Things like the date and mileage of the last oil change, when the timing belt was replaced, stuff like that.

I didn't want to hassle with databases or spreadsheets -- a simple, human-readable file is a lot easier to deal with. But I also wanted it in a predictable format so it could potentially be parsed later: I might some day want to write a program that keeps track and reminds me when I need to do things. (You'd think this program would already exist, but curiously, I haven't found one.)

So, something like:

7/9/12 <TAB> 86306 <TAB> Oil change, filter
6/11/12<TAB> 84813 <TAB> Smog test

Simple, right? And super easy just to type in in a text editor.

Well ... in emacs, maybe not. As an editor oriented toward programmers, emacs tends to prefer spaces instead of tabs ... and that's normally the way I prefer it. Tabs are a bad idea in most software projects, unless everybody on the project has already agreed on a tab style and width. (The complete tabs-vs.spaces religious war is beyond the scope of this article, but take my word for it that they can be a problem.)

But this wasn't code, and I needed an easy way to get those tabs into the file. Of course, I could quote them directly: type Ctrl-Q TAB every time I wanted a tab. But I knew I wouldn't remember to do that every time.

So I needed a local variables line, sometimes called a modeline, to flag this file as different from any other file I edit -- in this one, I really do want a TAB when I hit the TAB key.

My first try put this as the first line of the file:

-*- Mode: Text; indent-tabs-mode: t -*-
(Actually, my first try omitted the -*- part and didn't work at all, but let's ignore that part.)

When I re-loaded the file, emacs loaded it in text mode, and indent-tabs-mode was set to t if I checked it with describe-variable. Great! I could add new lines at the end of the file, and when I hit TAB, emacs inserted tabs. But wait -- if I added a line at the beginning of the file, and typed something lke 7/9/12<TAB> ... it inserted a space, not a tab.

It turns out indent-tabs-mode isn't really documented. If you check the help, it says:

Indentation can insert tabs if this is non-nil
-- note that "can" insert, not "will" insert, a tab. So how could I get emacs to really insert a tab, every time I hit the tab key?

Well, of course, I could bind the TAB key to "self-insert". But I didn't want to change it for every file, only for this one. how could I make that happen as part of a local variables line? The local variables line documentation shows how to set variables or file modes, but it looks like you can't bind keys or run functions.

Well, you can't run functions, but you can define a new mode that runs functions. At the suggestion of some helpful people on the #emacs IRC channel, I defined a new mode in my .emacs, like this:

(define-derived-mode tabbed-mode text-mode "Tab separated mode"
  (local-set-key (kbd "TAB") 'self-insert-command)
  )

Then in the file, I put this line:

-*- Mode: Tabbed -*-

Now, every time I load the file, emacs turns on tabbed mode -- which is just text mode except that the TAB key always inserts a tab.

Tags: ,
[ 19:35 Jul 16, 2012    More linux/editors | permalink to this entry | comments ]

Thu, 14 Jul 2011

Using default-frame-alist to specify Emacs window size, position and font

Seems like every few years I need to change the way I specify my preferred emacs fonts and window sizes.

Historically this all used to happen from one file, ~/.Xdefaults, where you set up your defaults for all X programs. In a way that was nice, since you could set up defaults and see the same font everywhere. On the other hand, it made for a huge, somewhat hard to read file, and it's increasingly out of favor on modern desktops, with modern toolkits like GTK just ignoring it.

Emacs still reads Xdefaults -- but only sort of. A lot of the values I used to set there no longer work properly. Some time ago I commented out my various attempts at setting emacs font, like

Emacs*font: -*-clean-bold-*-*-*-13-*-*-*-c-*-*-*
Emacs*font: DejaVu Sans Mono-10:bold
Emacs*font: clean-13:bold
Wmacs*font: Liberation Mono-10:bold
Emacs.font: 7x13bold
Emacs.faceName: Dejavu-10:style=bold
since none of them worked, and worked out a way of setting fonts inside my .emacs file:
(set-face-attribute 'default nil :font "Terminus-12:bold")

That worked to set the font, but it had another annoying attribute: it doesn't happen at startup, so it messed up my window size. See, emacs would start up, see the size I specified in .Xdefaults:

Emacs*geometry: 80x45
and try to set that. But it hadn't read .emacs yet, so it was still using whatever its default font and size is, and that's huge -- so 45 lines made a window too tall to fit on my laptop screen. Emacs would then shrink its window to fit the screen (41 lines). Only then would it open .emacs, whereupon it would see the set-face-attribute, change the font, and resize the window again, much, smaller, still 41 lines.

What a pain!

The emacs manual, in addition to talking about these various Xdefaults properties and command-line options, does mention a couple of variables, set-screen-height and set-screen-width, that looked promising. I tried putting (set-screen-height 45) in my .emacs right after I set the font -- no dice. Apparently that doesn't work because by the time those are read, emacs has already decided that 41 lines is as big as the window can possibly be.

Here's the answer: another variable that goes inside .emacs, default-frame-alist, but this one can override that maximum-height decision that emacs has already made. Here's an example of it in some useful defaults for emacs, and based on that, I was able to come up with this:

(setq default-frame-alist
      '((top . 10) (left . 2)
        (width . 80) (height . 53)
        (font . "terminus-iso8859-1-bold-14")
        ))

Curiously, that height setting, 53, needs to be 3 more than what I actually want according to the size emacs reports to the window manager. So don't take the number too seriously; just try numbers a little bigger than what you actually want until you get the size you're after. The font setting is the X font specifier: I ran xlsfonts | grep -i terminus | grep 14 then picked one of the simpler of the lines it printed out, but you can use a full specifier like -xos4-terminus-bold-r-normal--14-140-72-72-c-80-iso8859-1 like you get from xfontsel, if you prefer.

Startup still isn't pretty -- emacs still shows a big window at one place on the screen, resizes it several times then jumps it over to the top/left coordinates I specified. Of course, I could tell my window manager to start it in the right place so the jumping-around would be minimized; but that wouldn't help the visible resizing. Just a minor irritation.

I'm sure there's lots more useful stuff buried in that sample emacs config file (it was suggested to me when I asked about this on the #emacs IRC channel), so I'll be reading it to see what else I can glean.

Tags: , ,
[ 12:24 Jul 14, 2011    More linux/editors | permalink to this entry | comments ]

Wed, 25 May 2011

Vim/Emacs tip: use modelines for files that need special behavior

Most of the time when I edit a text file in vim, I want lines to wrap automatically as I type, at somewhere around 70 columns wide. So I set textwidth=70 in .vimrc.

But sometimes that isn't appropriate. For instance, I have a procmail rules file where I put common spam patterns that programs like spamassassin don't seem to be able to catch. So I might have lines like:

*^Subject:.*(Ink Cartridges|Hummingbird Vine|who's who of executives|Avandia|Botox|Your Email ID|Zoosk|Best airfares on the internet|UGG Boots|police training)
... and so on -- you get the idea. I can't have lines breaking in the middle, because then the procmail rule wouldn't work. So every time I add a new phrase, I have to :set tw=0 (or one of the other umpteen ways one can tell vim not to wrap lines) first.

But you can set special behavior for one specific file by adding a special comment called a "modeline" as the first line of the file.

Procmail treats any line starting with a hash, #, as a comment, and vim recognizes # as a comment. So I can add this as the first line of the procmail file:

# vim: set tw=0:
then vim will see that and un-set that default text width I specify in .vimrc.

Vim understands most common comment styles, so it should understand lines like /* vim: set tw=0: */ and // vim: set tw=0: and ; vim: set tw=0: as well.

But to make this work I had to do one more thing: in .vimrc, I had to add

set modeline

Apparently on some versions of vim this is on by default; in others it's turned off for security reasons (someone could put an evil modeline into a file which would make your vim do something when you edited it). Definitely something to be aware of, but if you mostly edit files you created yourself on your local machine, and no one else uses your machine, it's your choice whether to worry about it.

Emacs has modelines too

Emacs has mode lines too, though it calls them Local variables lines. For instance, C++ files in Mozilla's source tend to start with:

/* -*- Mode: C++; tab-width: 2; indent-tabs-mode: nil; c-basic-offset: 2 -*- */

It's awfully handy to be able to define specific indentation style for the files within a project, making it easy for emacs users, at least, to follow your preferred coding style. If only all editors understood them!

Tags: , , ,
[ 21:26 May 25, 2011    More linux/editors | permalink to this entry | comments ]

Fri, 04 Feb 2011

Quick tip: Disabling version control in Emacs

For some time I've been mildly annoyed that whenever I start emacs and open a file that's under any sort of version control -- cvs, svn, git or whatever -- I can't start editing right away, because emacs has to pause for a while and load a bunch of version-control cruft I never use. Sometimes it also causes problems later, when I try to write to the file or if I update the directory.

It wasn't obvious what keywords to search for, but I finally found a combination, emacs prevent OR disable autoload vc (the vc was the important part), which led me to the solution (found on this page):

;; Disable all version control
(setq vc-handled-backends nil)

Files load much faster now!

Tags: , , ,
[ 13:11 Feb 04, 2011    More linux/editors | permalink to this entry | comments ]

Wed, 01 Dec 2010

Escaping HTML characters in Emacs (and how to do replaces in elisp)

Last week I found myself writing another article that includes code snippets in HTML.

So what, you ask? The problem is, when you're writing articles in HTML, every time you include a code snippet inside a <pre> tag you invariably forget that special characters like < > & have special meanings in HTML, and must be escaped. Every < has to change to &lt;, and so forth, after you paste the code.

In vi/vim, replacing characters is straightforward. But I usually write longer articles in emacs, for various unimportant reasons, and although emacs has global replace, it only works from wherever you are now (called "point" in emacs lingo) to the end of the file. So if you're trying to fix something you pasted in the middle of the article, you can't do it with normal emacs replace.

Surely this is a wheel that has already been re-invented a thousand times, I thought! But googling and asking emacs experts turned up nothing. Looks like I'd have to write it.

And that turned out to be more difficult than I expected, for the same reason: emacs replace-string works the same way from a program as it does interactively, and replaces from point to the end of the file, and there's no way to restrict it to a more limited range.

Several helpful people on #emacs chimed in with ideas, but most of them didn't pan out. But ggole knew a way to do it that was both clean and reliable (thanks!).

Here's the elisp function I ended up with. It uses save-excursion to put the cursor back where it started before you ran the function, narrow-to-region to make replace-string work only on the region, and save-restriction get rid of that narrow-to-region after we're done. Nice!

(defun unhtml (start end)
  (interactive "r")
  (save-excursion
    (save-restriction
      (narrow-to-region start end)
      (goto-char (point-min))
      (replace-string "&" "&amp;")
      (goto-char (point-min))
      (replace-string "<" "&lt;")
      (goto-char (point-min))
      (replace-string ">" "&gt;")
      )))

And yes, I used it just now on that elisp snippet.

Tags: , ,
[ 20:08 Dec 01, 2010    More linux/editors | permalink to this entry | comments ]

Mon, 29 Mar 2010

Non-greedy regular expressions to clean up crappy autogenerated HTML

I maintain the websites for several clubs. No surprise there -- anyone with even a moderate knowledge of HTML, or just a lack of fear of it, invariably gets drafted into that job in any non-computer club.

In one club, the person in charge of scheduling sends out an elaborate document every three months in various formats -- Word, RTF, Excel, it's different each time. The only regularity is that it's always full of crap that makes it hard to turn it into a nice simple HTML table.

This quarter, the formats were Word and RTF. I used unrtf to turn the RTF version into HTML -- and found a horrifying page full of lines like this:

<body><font size=3></font><font size=3><br>
</font><font size=3></font><b><font size=4></font></b><b><font size=4><table border=2>
</font></b><b><font size=4><tr><td><b><font size=4><font face="Arial">Club Schedule</font></font></b><b><font size=4></font></b><b><font size=4></font></b></td>
<font size=3></font><font size=3><td><font size=3><b><font face="Arial">April 13</font></b></font><font size=3></font><font size=3><br>
</font><font size=3></font><font size=3><b></b></font></td>
I've put the actual page content in bold; the rest is just junk, mostly doing nothing, mostly not even legal HTML, that needs to be eliminated if I want the page to load and display reasonably.

I didn't want to clean up that mess by hand! So I needed some regular expressions to clean it up in an editor. I tried emacs first, but emacs makes it hard to try an expression then modify it a little when the first try doesn't work, so I switched to vim.

The key to this sort of cleanup is non-greedy regular expressions. When you have a bad tag sandwiched in the middle of a line containing other tags, you want to remove everything from the <font through the next > -- but no farther, or else you'll delete real content. If you have a line like

<td><font size=3>Hello<font> world</td>
you only want to delete through the <font>, not through the </td>.

In general, you make a regular expression non-greedy by adding a ? after the wildcard -- e.g. <font.*?>. But that doesn't work in vim. In vim, you have to use \{M,N} which matches from M to N repetitions of whatever immediately precedes it. You can also use the shortcut \{-} to mean the same thing as *? (0 or more matches) in other programs.

Using that, I built up a series of regexp substitutes to clean up that unrtf mess in vim:

:%s/<\/\{0,1}font.\{-}>//g
:%s/<b><\/b>//g
:%s/<\/b><b>//g
:%s/<\/i><i>//g
:%s/<td><\/td>/<td><br><\/td>/g
:%s/<\/\{0,1}span.\{-}>//g
:%s/<\/\{0,1}center>//g

That took care of 90% of the work, leaving me with hardly any cleanup I needed to do by hand. I'll definitely keep that list around for the next time I need to do this.

Tags: , , ,
[ 23:02 Mar 29, 2010    More linux/editors | permalink to this entry | comments ]

Tue, 09 Feb 2010

Make scroll lock stop beeping in Emacs 23

I haven't been using the spare machine much lately. So I hadn't noticed until last week that since upgrading to the emacs 23.1.1 on Ubuntu Karmic koala, every time I press the Scroll Lock key -- the key my KVM uses to switch to the other computer -- with focus in an emacs window, emacs beeps and complains that the key is unbound.

That was a problem I thought I'd solved long ago, an easy fix in .emacs:

(global-set-key [scroll-lock] 'ignore)
But in emacs 23, it wasn't working any more. Emacs listed the key as "<Scroll_Lock>", but using that directly in global-set-key doesn't work.

The friendly and helpful (really!) crew at #emacs found me a solution, after some fiddling around.

(global-set-key (kbd "<Scroll_Lock>") 'ignore)

Tags: , , ,
[ 23:47 Feb 09, 2010    More linux/editors | permalink to this entry | comments ]

Wed, 13 Jan 2010

Taming Emacs' text mode wrapping and indenting

To wrap long lines, or not to wrap? It's always a dilemma. Automatic wrapping is great when you're hammering away typing lots of text. But it's infuriating when you're trying to format something yourself and the editor decides it wants to line-wrap a little too early.

Although of course you can set the wrapping width, Emacs has a tendency to wrap early -- especially when you hit return. All too often, I'll be typing away at a long line, get to the end of the sentence and paragraph with the last word on the same line with the rest -- then realize that as soon as I hit return, Emacs is going to move that last word to a line by itself. Drives me nuts!

And the solution turns out to be so simple. The Return key, "\C-m". was bound to the (newline) function (you can find out by typing M-x, then describe-key, then hitting Return). Apparently (newline) re-wraps the current line before inserting a line break. But I just wanted it to insert a line break.

No problem -- just bind "C-m" to (insert "\n").

But there's a second way, too, if you don't want to rebind: there's a magic internal emacs table you can change.

(set-char-table-range auto-fill-chars 10 nil)

But wait -- there's one other thing I want to fix in text mode.

Automatic indent is another one of those features that's very convenient ... except when it's not.

If I have some text like:

First point:
  - subpoint a
  - subpoint b
then it's handy if, when I hit Return after subpoint a, emacs indents to the right level for subpoint b. But what happens when I get to the end of that list?
First point:
  - subpoint a
  - subpoint b

Second point:
  - subpoint c

When I hit Return after subpoint b, Emacs quite reasonably indents two spaces. If I immediately type another Return, Emacs sensibly deletes the two spaces it just inserted, opens a new line -- but then it indents that new line another two spaces.

After a blank line, I always want to start at the beginning, not indented at all.

Here's how to fix that. Define a function that will be called whenever you hit return in text mode. That function tests whether the caret comes immediately after a blank line, or at the beginning of the file. It indents except in those two cases; and in neither case does it re-wrap the current line.

;; In text mode, I don't want it auto-indenting for the first
;; line in the file, or lines following blank lines.
;; Everywhere else is okay.
(defun newline-and-text-indent ()
  "Insert a newline, then indent the next line sensibly for text"
  (interactive)
  (cond
   ;; Beginning of buffer, or beginning of an existing line, don't indent:
   ((or (bobp) (bolp)) (newline))

   ;; If we're on a whitespace-only line,
   ((and (eolp)
         (save-excursion (re-search-backward "^\\(\\s \\)*$"
                                             (line-beginning-position) t)))
    ;; ... delete the whitespace, then add another newline:
    (kill-line 0)
    (newline))

   ;; Else (not on whitespace-only) insert a newline,
   ;; then add the appropriate indent:
   (t (insert "\n")
      (indent-according-to-mode))
   ))

Then tell emacs to call that function when it sees the Return key in text mode:

(defun text-indent-hook ()
  (local-set-key "\C-m" 'newline-and-text-indent)
  )
(setq text-mode-hook 'text-indent-hook)

Finally, this is great for HTML mode too, if you get irritated at not being able to put an <a href="longurl"> all on one line:

(defun html-hook ()
  (local-set-key "\C-m" (lambda () (interactive) (insert "\n")))
  )
(setq sgml-mode-hook 'html-hook)

Tags: , ,
[ 11:29 Jan 13, 2010    More linux/editors | permalink to this entry | comments ]

Wed, 29 Jul 2009

Emacs: Insert tags in HTML Mode

Wouldn't it be nice if Emacs HTML mode had a way to insert HTML tags, so you didn't have to type <b></b> all the time? Sort of like what's described in this page -- except that page describes an HTML mode that clearly isn't the one that's installed on Ubuntu, since none of those bindings actually work?

I've been meaning to figure out a way to do that for ages, and finally got around to it. Turns out Emacs SGML mode (which is really what Ubuntu installs and uses for HTML files) doesn't have functions for specific HTML tags like <b>, but it does have a general tag-inserting function.

Type C-c C-t -- emacs prompts you for the tag, so type b or whatever, and hit return -- and you get the tag, with the cursor correctly positioned for you to type your new bold text.

But that's four keystrokes. What if you want shorter bindings for particular tags, like C-b C-b to insert a bold?

For that, you need to use a lambda and a mode hook. In your .emacs it looks like this:

;; Define keys for inserting tags in HTML mode:
(defun html-hook ()
  (local-set-key "\C-c\C-b" (lambda () (interactive) (sgml-tag "b")))
  )
(setq sgml-mode-hook 'html-hook)

There's apparently also supposed to be a command bound to C-c / that closes the current tag, but my version of sgml-mode doesn't bind anything to that key, and the only likely-looking function name, sgml-maybe-end-tag, doesn't end the current tag. Such is life!

But one more don't-miss feature that I'd missed all along is C-c C-n: type it before a special character like < or & and emacs will insert the appropriate &lt; or &amp; for you. Nice!

(Thanks to bojohan on #emacs for the tips!)

Tags: , ,
[ 21:31 Jul 29, 2009    More linux/editors | permalink to this entry | comments ]

Fri, 27 Mar 2009

Emacs bookmarks -- a huge time-saver

Oh, wow. I can't believe I've used Emacs all these years without knowing about bookmarks.

I wanted something in Emacs akin to the "Open Recent" menu that a lot of GUI apps have. Except, well, I didn't want it to need a menu (I don't normally show a menubar in Emacs) and I didn't want it limited only to recently accessed files. So ... just like Open Recent, only completely different.

What I really wanted was a way to nickname files I access regularly, so I don't have to type ~/foo/bar/blaz/route-66/dufus/velociraptor/archaeopteryx/filename every time. Even with tab completion, remembering long paths gets old. Of course emacs must have a way to do that; it has everything. The trick was guessing what it might be called in order to search for it.

The answer is emacs bookmarks and they're super easy to use.

C-x r m sets a bookmark for the current location in the current file. It prompts for a bookmark name; give it a nickname, or hit return to default it to the current filename.

C-x r b bookmark-name jumps back to a bookmark, opening the file if it isn't already. Of course, tab completion works for the bookmark name.

Bookmarks are saved in ~/.emacs.bmk so they're persistent.

It's perfect. I just wish I'd thought to look for it years ago.

(Of course, Emacs can do recent files too.)

Tags: , ,
[ 10:21 Mar 27, 2009    More linux/editors | permalink to this entry | comments ]

Sun, 22 Mar 2009

Vim tip: fixing the light-background color schemes

I use a light background for my X terminals (xterm and rxvt): not white, but a light grey that I find easy on the eyes. Long ago, I spent the time to set up a custom vim color scheme that works with the light background.

But sometimes I need to run vim somewhere where I don't have access to my custom scheme. It always starts up with a lot of the words displayed in yellow, completely unreadable against a light background. :set background=light doesn't help -- the default colorscheme is already intended for a light background, yet it still uses yellow characters.

I tried all the colorschemes installed with ubuntu's vim (you can get a list of them with ls /usr/share/vim/vim71/colors). The only light-background vim schemes that don't use yellow all have their primary text color as red. Do a lot of people really want to edit red text? Maybe the same people who think that yellow is a dark color?

Curiously, it turns out that if you use one of these light color schemes on a Linux console (with its black background), the yellow text isn't yellow (which would show up fine against black), but orange (which would be better on a light background).

Mikael knew the answer:

:set t_Co=88

This tells vim to use 88-color mode instead of its default of 8, and the yellow text turns light blue. Not terrifically readable but much better than yellow. Or, instead, try

:set t_Co=256
and the yellow/light blue text turns an ugly, but readable, orange (probably the same orange as the console used).

So, vim users with dark-on-light terminal schemes: add set t_Co=256 in your .vimrc (no colon) and you'll be much happier.

Update: Pádraig Brady has a great page explaining more about terminal colour highlights, including a TERM=xterm-256color setting to get vim to use 256 colors automatically. There's also a lot of good advice there on enabling colors in other console apps.

The only catch: on Ubuntu you do have to install the ncurses-term package, which will get you xterm-256color as well as 256color variants for lots of other terminal types. Here's useful page on 256-Color XTerms in Ubuntu.

Tags: , , ,
[ 22:29 Mar 22, 2009    More linux/editors | permalink to this entry | comments ]

Sun, 12 Oct 2008

More fun with regexps: Adding "[no output]" in shell logs

Someone on LinuxChix' techtalk list asked whether she could get tcsh to print "[no output]" after any command that doesn't produce output, so that when she makes logs to help her co-workers, they will seem clearer.

I don't know of a way to do that in any shell (the shell would have to capture the output of every command; emacs' shell-mode does that but I don't think any real shells do) but it seemed like it ought to be straightforward enough to do as a regular expression substitute in vi. You're looking for lines where a line beginning with a prompt is followed immediately by another line beginning with a prompt; the goal is to insert a new line consisting only of "[no output]" between the two lines.

It turned out to be pretty easy in vim. Here it is:

:%s/\(^% .*$\n\)\(% \)/\1[no results]\r\2/

Explanation:

:
starts a command
%
do the following command on every line of this file
s/
start a global substitute command
\(
start a "capture group" -- you'll see what it does soon
^
match only patterns starting at the beginning of a line
%
look for a % followed by a space (your prompt)
.*
after the prompt, match any other characters until...
$
the end of the line, after which...
\n
there should be a newline character
\)
end the capture group after the newline character
\(
start a second capture group
%
look for another prompt. In other words, this whole
expression will only match when a line starting with a prompt
is followed immediately by another line starting with a prompt.
\)
end the second capture group
/
We're finally done with the mattern to match!
Now we'll start the replacement pattern.
\1
Insert the full content of the first capture group
(this is also called a "backreference" if you want
to google for a more detailed explanation).
So insert the whole first command up to the newline
after it.
[no results]
After the newline, insert your desired string.
\r
insert a carriage return here (I thought this should be
\n for a newline, but that made vim insert a null instead)
\2
insert the second capture group (that's just the second prompt)
/
end of the substitute pattern

Of course, if you have a different prompt, substitute it for "% ". If you have a complicated prompt that includes time of day or something, you'll have to use a slightly more complicated match pattern to match it.

Tags: , , , ,
[ 14:34 Oct 12, 2008    More linux/editors | permalink to this entry | comments ]

Thu, 20 Dec 2007

Smart Wrapping with Greedy and Non-Greedy Regular Expressions

I had a chance to spend a day at the AGU conference last week. The American Geophysical Union is a fabulous conference -- something like 14,000 different talks over the course of the week, on anything related to earth or planetary sciences -- geology, solar system astronomy, atmospheric science, geophysics, geochemistry, you name it.

I have no idea how regular attendees manage the information overload of deciding which talks to attend. I wasn't sure how I would, either, but I started by going through the schedule for the day I'd be there, picking out a (way too long) list of potentially interesting talks, and saving them as lines in a file.

Now I had a file full of lines like:

1020      U22A    MS 303  Terrestrial Impact Cratering: New Insights Into the Cratering Process From Geophysics and Geochemistry II
Fine, except that I couldn't print out something like that -- printers stop at 80 columns. I could pass it through a program like "fold" to wrap the long lines, but then it would be hard to scan through quickly to find the talk titles and room numbers. What I really wanted was to wrap it so that the above line turned into something like:
1020      U22A    MS 303  Terrestrial Impact Cratering: New Insights
                          Into the Cratering Process From Geophysics
                          and Geochemistry II
But how to do that? I stared at it for a while, trying to figure out whether there was a clever vim substitute that could handle it. I asked on a couple of IRC channels, just in case there was some amazing Linux smart-wrap utility I'd never heard of. I was on the verge of concluding that the answer was no, and that I'd have to write a python script to do the wrapping I wanted, when Mikael emitted a burst of line noise:
%s/\(.\{72\}\)\(.*\)/\1^M^I^I^I\2/

Only it wasn't line noise. Seems Mikael just happened to have been reading about some of the finer points of vim regular expressions earlier that day, and he knew exactly the trick I needed -- that .\{72\}, which matches lines that are at least 72 characters long. And amazingly, that expression did something very close to what I wanted.

Or at least the first step of it. It inserts the first line break, turning my line into

1020      U22A    MS 303  Terrestrial Impact Cratering: New Insights
                          Into the Cratering Process From Geophysics and Geochemistry II
but I still needed to wrap the second and subsequent lines.

But that was an easier problem -- just do essentially the same thing again, but limit it to only lines starting with a tab. After some tweaking, I arrived at exactly what I wanted:

%s/^\(.\{,65\}\) \(.*\)/\1^M^I^I^I\2/

%g/^^I^I^I.\{58\}/s/^\(.\{,55\}\) \(.*\)/\1^M^I^I^I\2/
I had to run the second line two or three times to wrap the very long lines.

Devdas helpfully translated the second one into English: "You have 3 tabs, followed by 58 characters, out of which you match the first 55 and put that bit in $1, and the capture the remaining in $2, and rewrite to $1 newline tab tab tab $2."

Here's a more detailed breakdown:

Line one:
% Do this over the whole file
s/ Begin global substitute
^ Start at the beginning of the line
\( Remember the result of the next match
.\{,65\}_ Look for up to 65 characters with a space at the end
\) \( End of remembered pattern #1, skip a space, and start remembered pattern #2
.*\) Pattern #2 includes everything to the end of the line
/ End of matched pattern; begin replacement pattern
\1^M Insert saved pattern #1 (the first 65 lines ending with a space) followed by a newline
^I^I^I\2 On the second line, insert three tabs then saved pattern #2
/ End replacement pattern

Line two:
%g/ Over the whole file, only operate on lines with this pattern
^^I^I^I Lines starting with three tabs
.\{58\}/ After the tabs, only match lines that still have at least 58 characters (this guards against wrapping already wrapped lines when it's run repeatedly)
s/ Begin global substitute
^ Start at the beginning of the line
\( Remember the result of the next match
.\{,55\} Up to 55 characters
\) \( End of remembered pattern #1, skip a space, and start remembered pattern #2
.*\) Pattern #2 includes everything to the end of the line
/ End of matched pattern; begin replacement pattern
\1^M The first pattern (up to 55 chars) is one line
^I^I^I\2 Three tabs then the second pattern
/ End replacement pattern

Greedy and non-greedy brace matches

The real key is those curly-brace expressions, \{,65\} and \{58\} -- that's how you control how many characters vim will match and whether or not the match is "greedy". Here's how they work (thanks to Mikael for explaining).

The basic expression is {M,N} -- it means between M and N matches of whatever precedes it. (Vim requires that the first brace be escaped -- \{}. Escaping the second brace is optional.) So .{M,N} can match anything between M and N characters but "prefer" N, i.e. try to match as many as possible up to N. To make it "non-greedy" (match as few as possible, "preferring" M), use .{-M,N}

You can leave out M, N, or both; M defaults to 0 while N defaults to infinity. So {} is short for {0,∞} and is equivalent to *, while {-} means {-0,∞}, like a non-greedy version of *.

Given the string: one, two, three, four, five
,.\{}, matches , two, three, four,
,.\{-}, matches , two,
,.\{5,}, matches , two, three, four,
,.\{-5,}, matches , two, three,
,.\{,2}, matches nothing
,.\{,7}, matches , two,
,.\{5,7}, matches , three,

Of course, this syntax is purely for vim; regular expressions are unfortunately different in sed, perl and every other program. Here's a fun table of regexp terms in various programs.

Tags: , ,
[ 12:44 Dec 20, 2007    More linux/editors | permalink to this entry | comments ]

Mon, 19 Feb 2007

Emacs with Long Lines

I don't like composing text documents in word processors like Open Office. Call it a quirk if you like, but I find them intrusive: they take up a lot of CPU and memory, they take up a lot of window space for stuff I don't need while I'm writing (all those margins and rulers and toolbars and such) making it hard to compare two documents at once, and they tend to have intrusive focus behavior (like popping windows to the front when I didn't ask for it).

So when I need to write a paper (or a book), I prefer to compose in a text editor like vim or emacs, something that won't get in the way of my train of thought. When it's mostly written and ready to format, then I start up the big heavyweight word processor and import or paste the text into it.

(For those of you who think I'm insane and should just live in Open Office all day, the same problem comes up for people who do a lot of composing for web applications, such as an online blog, gmail, a web forum, or a wiki, and for people who want a choice of editor for their GUI mail app.)

Fine, but that introduces a problem. See, text editors have a fixed line width (typically 80 characters, though of course you can adjust this) and paragraphs are usually separated by blank lines (two newline characters together). Word processors expect each paragraph to be one long line for the whole paragraph, and line breaks are used as paragraph breaks (but you only want one of them, not two). How do you reconcile these two models in order to paste plaintext from an editor into a word processor?

Several years ago when I first encountered this problem, I investigated solutions in both vim and emacs (oddly enough, I'm an editor agnostic and equally happy in either one).

For vim, I never did find a solution to the problem, so that settled the editor choice for me. Perhaps some vim expert can let me know what I missed.

For emacs, I found longlines-mode, a hack which lets long lines appear to be wrapped while you're editing them even though they're really not. Apparently Wikipedia has this issue and some Wikipedia contributors use longlines-mode too. (That page also has brief notes on alternate solutions.)

I used longlines-mode for a long time, and it's more or less functional, but I was never really happy with it. It turns out to have some pretty annoying bugs which I was forever needing to work around, and it doesn't solve the blank-lines problem -- you still need to delete blank lines before or after pasting.

Yesterday I was working on an essay for a class I'm taking and decided I'd had enough of longlines-mode and wanted a better solution. I poked around and chatted with the nice folks on #emacs (hoping that someone had come up with a better solution, but no one knew of one) and based on some ideas they had, I came up with one of my own.

My new method is to edit the text file normally: line breaks where they look good, blank lines to separate paragraphs. When I'm finished writing and ready to paste, I run M-x wp-munge, which calls up a very simple function I wrote and added to my .emacs:

;; For composing in emacs then pasting into a word processor,
;; this un-fills all the paragraphs (i.e. turns each paragraph
;; into one very long line) and removes any blank lines that
;; previously separated paragraphs.
;;
(defun wp-munge () "un-fill paragraphs and remove blank lines" (interactive)
  (let ((save-fill-column fill-column))
    (set-fill-column 1000000)
    (mark-whole-buffer)
    (fill-individual-paragraphs (point-min) (point-max))
    (delete-matching-lines "^$")
    (set-fill-column save-fill-column) ))

So simple! Why didn't I think of doing it that way before?

Tags: ,
[ 21:10 Feb 19, 2007    More linux/editors | permalink to this entry | comments ]

Sun, 14 May 2006

Linkifying with Regular Expressions

I had a page of plaintext which included some URLs in it, like this:
Tour of the Hayward Fault
http://www.mcs.csuhayward.edu/~shirschf/tour-1.html

Technical Reports on Hayward Fault
http://quake.usgs.gov/research/geology/docs/lienkaemper_docs06.htm

I wanted to add links around each of the urls, so that I could make it part of a web page, more like this:

Tour of the Hayward Fault
http://www.mcs.csu hayward.edu/~shirschf/tour-1.html

Technical Reports on Hayward Fault
htt p://quake.usgs.gov/research/geology/docs/lienkaemper_docs06.htm

Surely there must be a program to do this, I thought. But I couldn't find one that was part of a standard Linux distribution.

But you can do a fair job of linkifying just using a regular expression in an editor like vim or emacs, or by using sed or perl from the commandline. You just need to specify the input pattern you want to change, then how you want to change it.

Here's a recipe for linkifying with regular expressions.

Within vim:

:%s_\(https\=\|ftp\)://\S\+_<a href="&">&</a>_

If you're new to regular expressions, it might be helpful to see a detailed breakdown of why this works:

:
Tell vim you're about to type a command.
%
The following command should be applied everywhere in the file.
s_
Do a global substitute, and everything up to the next underscore will represent the pattern to match.
\(
This will be a list of several alternate patterns.
http
If you see an "http", that counts as a match.
s\=
Zero or one esses after the http will match: so http and https are okay, but httpsssss isn't.
\|
Here comes another alternate pattern that you might see instead of http or https.
ftp
URLs starting with ftp are okay too.
\)
We're done with the list of alternate patterns.
://
After the http, https or ftp there should always be a colon-slash-slash.
\S
After the ://, there must be a character which is not whitespace.
\+
There can be any number of these non-whitespace characters as long as there's at least one. Keep matching until you see a space.
_
Finally, the underscore that says this is the end of the pattern to match. Next (until the final underscore) will be the expression which will replace the pattern.
<a href="&">
An ampersand, &, in a substitute expression means "insert everything that was in the original pattern". So the whole url will be inserted between the quotation marks.
&</a>
Now, outside the <a href="..."> tag, insert the matched url again, and follow it with a </a> to close the tag.
_
The final underscore which says "this is the end of the replacement pattern". We're done!

Linkifying from the commandline using sed

Sed is a bit trickier: it doesn't understand \S for non-whitespace, nor = for "zero or one occurrence". But this expression does the trick:
sed -e 's_\(http\|https\|ftp\)://[^ \t]\+_<a href="&">&</a>_' <infile.txt >outfile.html

Addendum: George Riley tells me about VST for Vim 7, which looks like a nice package to linkify, htmlify, and various other useful things such as creating HTML presentations. I don't have Vim 7 yet, but once I do I'll definitely check out VST.

Tags: , , , , ,
[ 13:40 May 14, 2006    More linux/editors | permalink to this entry | comments ]

Wed, 29 Mar 2006

Emacs: Typing dashes in html mode

What to do with a few extra hours in a boring motel with no net access? How about digging into fixing one of Emacs' more annoying misfeatures?

Whenever I edit an html file using emacs, I find I have to stay away from double dashes -- I can't add a phrase such as this one. If I forget and type a phrase with a double dash, then as soon as I get to the end of that line and emacs decides it's time to wrap to the next line, it "helpfully" treats the double dashes as a comment, and indents the next line to the level where the dashes were, adding another set of dashes. I've googled, I've asked on emacs IRC help channels, but there doesn't seem to be any way out. (I guess no one else ever uses double dashes in html files?)

It's frustrating: I like using double dashes now and then. And aside from the occasional boneheaded misfeature like this one, I like using emacs. But the dash problem been driving me nuts for a long time now. So I finally dug into the code to cure it.

First, the file is sgml-mode.el, so don't bother searching anything with html in the name. On my system it's /usr/share/emacs/21.4/lisp/textmodes/sgml-model.el. Edit that file and search for "--" and the first thing you'll find (well, after the file's preamble comments) is a comment in the definition of "sgml-specials" saying that if you include ?- in the list of specials, it will hork the typing of double dashes, so that's normally left out.

A clue! Perhaps some Debian or Ubuntu site file has changed sgml-specials for me, and all I need to do is change it back! So I typed

M-x describe-variable sgml-specials
to see the current setting.

Um ... it's set to "34". That's not very helpful. I haven't a clue how that translates to the list of characters I see in sgml-mode.el. Forget about that approach for now.

Searching through the file for the string "comment" got me a few more hits, and I tried commenting out various comment handling lines until the evil behavior went away. (I had to remove sgml-mode.elc first, otherwise emacs wouldn't see any changes I made to sgml-mode.el. If you haven't done much elisp hacking, the .el is the lisp source, while the .elc is a byte-compiled version which loads quicker but isn't intended to be edited by humans. For Java programmers, the .elc is sort of like a .class file.)

Commenting out these four lines did the trick:

  (set (make-local-variable 'font-lock-syntactic-keywords)
       '(("\\(<\\)! *--.*-- *\\(>\\)" (1 "!") (2 "!"))))
  ;; This will allow existing comments within declarations to be
  ;; recognized.
  (set (make-local-variable 'comment-start-skip) "\\(?:\\)?")

To regenerate the .elc file so sgml-mode will load faster, I ran emacs as root from the directory sgml-mode.el was in, and typed:

M-x byte-compile-file sgml-mode.el

All better! And now I know where to find documentation for all those useful-looking, but seemingly undocumented, keyboard shortcuts that go along with emacs' html mode. Just search in the file for html-mode-map, and you'll find all sorts of useful stuff.

For instance, that typing Ctrl-C Ctrl-C followed by various letters: u gets you an unordered list, h gets you an href tag, i an image tag, and so on, with the cursor positioned where you want to type next.

It doesn't seem to offer any basic inline formatting (like <i> or <em>), alas; but of course that's easy to add by editing the file (or maybe even in .emacs). To add an <em> tag, add this line to html-mode-map:

    (define-key map "\C-c\C-ce" 'html-em)
then add this function somewhere near where html-headline-1 and friends are defined:
(define-skeleton html-em
  "HTML emphasis tags."
  nil
  "" _ "")

Of course, you can define any set of tags you use often, not just <em>.

HTML mode in emacs should be much more fun and less painful now!

Update: If you don't want to modify the files as root, it also works fine to copy sgml-mode.el to wherever you keep personal elisp files. For instance, put them in a directory called ~/.emacs-lisp then add this to your .emacs:
(setq load-path (cons "~/.emacs-lisp/" load-path))

Tags: ,
[ 22:48 Mar 29, 2006    More linux/editors | permalink to this entry | comments ]

Wed, 22 Jun 2005

Helpful Vim Tip: Finding Syntax for Colors

An upgrade from woody to sarge introduced a new problem with editing mail messages in vim: Subject lines appeared in yellow, against my light grey background, so they weren't readable any more.

Vim color files have always been a mystery to me. I have one which I adapted from one of the standard color schemes, but I've never been clear what the legal identifiers are or how to find out. But I changed both places where it said "ctermfg=Yellow" to another color, and nothing changed, so this time I had to find out.

Fortunately a nice person on #vim suggested :he synID (he is short for "help", of course) which told me all I needed to know. Put the cursor on the errant line and type: :echo synIDattr(synID(line("."), col("."), 1), "name")

That told me that the Subject line was syntax class "mailSubject". So I tried (copying other lines in my color file) adding this line:

hi mailSubject term=underline ctermfg=Red guifg=Red
and now all is happy again in vim land. I wish I'd learned that synID trick a long time ago!

Tags: , , ,
[ 10:59 Jun 22, 2005    More linux/editors | permalink to this entry | comments ]

Sat, 19 Feb 2005

Tweaking Emacs' Text Indent: Don't Indent So Aggressively

Encouraged by my success a few days ago at finally learning how to disable vim's ctrl-spacebar behavior, the next day I went back to an emacs problem that's been bugging me for a while: in text mode, newline-and-indent always wants to indent the first line of a text file (something I almost never want), and skips blank lines when calculating indent (so starting a new paragraph doesn't reset the indent back to zero).

I had already googled to no avail, and had concluded that the only way was to write a new text-indent function which could be bound to the return key in the text mode hook.

This went fairly smoothly: I got a little help in #emacs with checking the pattern immediately before the cursor (though I turned out not to need that after all) and for the function called "bobp" (beginning of buffer predicate). Here's what I ended up with:

(defun newline-and-text-indent ()
  "Insert a newline, then indent the next line sensibly for text"
  (interactive)
  (if (or (bobp)
          (looking-at "^$"))
      (newline)
      (newline-and-indent)
  ))
(defun text-indent-hook ()
  (local-set-key "\C-m" 'newline-and-text-indent)
  )
(setq text-mode-hook 'text-indent-hook)

It seems to work fine. For the curious, here's my current .emacs

Tags: ,
[ 14:03 Feb 19, 2005    More linux/editors | permalink to this entry | comments ]

Thu, 17 Feb 2005

Turning off Ctrl-Space in Vim

One of those niggling problems that has plagued me for a long time: in the editor vim, if I'm typing along in insert mode and instead of a space I accidentally hit control-space, vim inserts a bunch of text I didn't want, then exits insert mode. Meanwhile I'm still merrily typing away, typing what are now vim comments which invariably end up deleting the last two paragraphs I typed then doing several more operations which end up erasing the undo buffer so I can't get those paragraphs back.

Ctrl-space inserts a null character (you can verify this by putting it in a file and running od -xc on it). I've done lots of googling in the past, but it's hard to google on strings like " " or even "space" or "null", and nobody I asked had a clue what this function was called (it turns out it re-inserts whatever the last inserted characters were) so I couldn't google on the function name.

Vim's help suggests that <Nul>, <Char-0>, or <C-^V > should do it. I tried them with map, vmap, cmap, and nmap, to no avail. I also tried <C-@> since that's a historical way of referring to the null character, googling found some references to that in vim, and that's how it displays if I type it in vim.

I finally found #vim on freenode, and asked there. Last night nobody knew, but this morning, p0g found the problem: I needed to use imap, not the map/vmap/cmap/nmap I'd been using.

So here, perserved for google posterity in case other people are plagued by this problem, is the answer:

imap <Nul> <Space>

For good measure, I also mapped the character to no-op in all the other modes as well:

map  <Nul> <Nop>
vmap <Nul> <Nop>
cmap <Nul> <Nop>
nmap <Nul> <Nop>

My current .vimrc.

Tags: , ,
[ 11:24 Feb 17, 2005    More linux/editors | permalink to this entry | comments ]

Thu, 03 Feb 2005

Emacs Color Themes

A nifty emacs trick I learned about today: ColorThemes.

Instead of the old hacked-together color collection I've been using in emacs, I can load color-theme.el and choose from lots of different color schemes.

I added these lines to .emacs:

(require 'font-lock)
(if (fboundp 'global-font-lock-mode) (global-font-lock-mode 1))
(load "~/.emacs-lisp/color-theme.el")
(color-theme-ramangalahy)  ;; pick a favorite theme

The disadvantage is that color-theme.el is fifteen thousand lines long! So I'll probably make a local version that strips out all but the theme I actually use (then I can customize that).

The (global-font-lock-mode 1) tells emacs to use syntax highlighting on every file, not just certain types. So now I get at least some highlighting even in html files, though it still doesn't seem to be able to highlight like vim does (e.g. different colors for text inside <b> or <b> tags).

Tags: ,
[ 18:57 Feb 03, 2005    More linux/editors | permalink to this entry | comments ]

Thu, 13 Jan 2005

Nifty emacs hack: how to wrap only certain files

For a long time I've wanted some, but not all, text and html files to line-wrap automatically in emacs. For instance, it drives me nuts when I edit a system configuration file and it wraps each long line, or when I edit an html file containing lots of long links and it keeps wrapping between the <a and the href=. But for files which are mostly text (such as these blog entries), I want line wrapping.

I'd been trying to do this with html-mode-hook and text-mode-hook, then checking the filename and calling (auto-fill-mode) if appropriate, but it wasn't working, because buffer-file-name isn't always defined at the time the mode hook is called. (No one seems to know why.) The buffer name seems to be defined at that point, but it doesn't contain path information so I can't say "Use wrapping for anything under ~/Docs" or "Don't wrap anything in /etc".

But with some help from sachac and the nice folks on #emacs I came up with a much better solution, and it's way simpler than the mode-hook approach: derived modes.

I set up two new modes, called html-wrap-mode and text-wrap-mode, which are the same as html-mode and text-mode except that they turn on auto-fill. Then I use the easy auto-mode-alist mechanism, which already does string matching on the filename, to call these modes, instead of the regular text and html modes, based on the extension or some other aspect of the file's pathname. Here's what I added to .emacs:

;; Want auto-fill-mode for some text and html files, but not all.
;; So define two derived modes for that, and we'll use auto-mode-alist
;; to choose them based on filename.
(define-derived-mode html-wrap-mode html-mode "HTML wrap mode"
  (auto-fill-mode))
(define-derived-mode text-wrap-mode text-mode "Text wrap mode"
  (auto-fill-mode))

(setq auto-mode-alist
      (cons '("\\.blx$" . html-wrap-mode)
      (cons '("Docs/.*.html$" . html-wrap-mode)
      (cons '("Docs/" . text-wrap-mode)
            auto-mode-alist) ) ) )

Here's my current .emacs.

I wonder if vim has a way to do this?

Tags: ,
[ 23:30 Jan 13, 2005    More linux/editors | permalink to this entry | comments ]

Syndicated on:
LinuxChix Live
Ubuntu Women
Women in Free Software
Graphics Planet
DevChix
Ubuntu California
Planet Openbox
Devchix
Planet LCA2009

Friends' Blogs:
Morris "Mojo" Jones
Jane Houston Jones
Dan Heller
Long Live the Village Green
Ups & Downs
DailyBBG

Other Blogs of Interest:
DevChix
Scott Adams
Dave Barry
BoingBoing

Powered by PyBlosxom.