Shallow Thoughts : tags : email

Gmail with an App Password

In 2022 I wrote about Sending Mail via Gmail using OAuth2.

But it turned out that Google expires OAuth2 tokens on a weekly basis. So if you use that method, once a week you'll have to bring up a browser, log in to your Google account and go through the five or so pages of re-authorizing. Which will invariably happen when you're in a hurry and just wanted to send a quick email so you can move on to other things.

However, it turns out there's an easier way, which apparently doesn't expire: App passwords. I switched to using app passwords back then (I've been using that app password since then), and I even wrote it up, and then forgot to post it. What a dingbat!

But I changed my GMail password recently, and it turns out when you change your Gmail account password, Google revokes all app passwords you've set up (and, of course, doesn't bother to tell you that, and the error message you get when you try to sign in with the old app password has nothing whatever to do with the actual problem, which is that your app password has been revoked and you need to create a new one).

So I dug out this old never-got-posted article and used it to make a new app password, and have updated the parts that were a little out of date.

Read more ...

Tags: email, mutt, gmail
[ 20:32 Mar 29, 2024    More tech/email | permalink to this entry | ]

View Mail Attachments from Mutt

Back in 2015, I wrote a script for the mutt mailer (or any plaintext mail program, really) to view MS Word documents (or other unfriendly formats) attached to emails. (This is unfortunately something that comes up constantly in email exchanges with League of Women Voters people —

Read more ...

Tags: email, mutt, programming, python, linux
[ 18:52 Dec 18, 2022    More tech/email | permalink to this entry | ]

Sending Mail via Gmail using OAuth2 (2022 Edition)

Update: Google's OAuth2 turns out to be not a good way to send mail, because passwords have to be renewed weekly. So you probably want to use a GMail App Password instead. I'm leaving this article up in case there's some reason someone would actually want to use OAuth2 with GMail.

There's been lots of talk on mailing lists for various mail programs, like Alpine and Mutt, about Google's impending dropping of password access.

Although my regular email address is on a Linux server, I subscribe to several Google Groups. I use a gmail address for those, because Google Groups doesn't work well with non-gmail addresses (you can't view the archives or temporarily turn off mail, and unsubscribing may or may not work depending on the phase of the moon).

I prefer not to have to sign on to Google and use the clunky browser interface when I have a perfectly good mailer (I use mutt) on my computer. I send mail from mutt using a program called msmtp. But to post to a Google Group, I need to use Google's SMTP server. (SMTP is the Simple Mail Transfer Protocol, the way mail gets from one computer to another across the internet.)

Up to now, I've been using an msmtp configuration that includes my Gmail password. That requires clicking through several Gmail pages to enable the "Less Secure Apps" setting. Google resets that preference every month or so and I have to go find the "Less Secure Apps" page to click through the screens again; but aside from that, it works okay.

But now Google has announced they'll be removing support for password access on May 30, 2022.

Read more ...

Tags: email, mutt, gmail, oauth
[ 00:00 May 20, 2022    More tech/email | permalink to this entry | ]

Mutt mailer: Show Messages With Attachments

A discussion on the Mutt-Users list a while back involved someone trying to configure mutt to show on the index screen which messages had attachments.

I had no idea that was possible! But it's something I've wanted for ages. Normally, mutt shows a list of attachments after the end of the email message. That was useful back in the day when people trimmed their email messages; but now, when most people append the contents of an entire message thread going back several weeks, scrolling down to the end of an email message is barely even possible.

What I'd really like is to see in the message view whether the message has attachments -- up at the top of the message, along with the headers. But showing it in the folder index would be a great start.

What Constitutes an Attachment?

First you have to define what attachments you care about. Most normal email messages have attachments just for the text.

Read more ...

Tags: mutt, email, attachments, mime
[ 10:46 May 22, 2021    More linux | permalink to this entry | ]

HTML Email from Mutt

I know, I know. We use mailers like mutt because we don't believe in HTML mail and prefer plaintext. Me, too.

But every now and then a situation comes up where it would be useful to send something with emphasis. Or maybe you need to highlight changes in something. For whatever reason, every now and then I wish I had a way to send HTML mail.

I struggled with that way back, never did find a way, and ended up writing a Python script, htmlmail.py to send an HTML page, including images, as email.

Sending HTML Email

But just recently I found a neat mutt hack. It turns out it's quite easy to send HTML mail.

First, edit the HTML source in your usual mutt message editor (or compose the HTML some other way, and insert the file). Note: if there's any quoted text, you'll have to put a <pre> around it, or otherwise turn it into something that will display nicely in HTML.

Write the file and exit the editor. Then, in the Compose menu, type Ctrl-T to edit the attachment type. Change the type from text/plain to text/html.

That's it! Send it, and it will arrive looking like a regular HTML email, just as if you'd used one of them newfangled gooey mail clients. (No inline images, though.)

Viewing HTML Email

Finding out how easy that was made me wonder why the other direction isn't easier. Of course, I have my mailcap set up so that mutt uses lynx automatically to view HTML email:

text/html; lynx -dump %s; nametemplate=%s.html; copiousoutput

Lynx handles things like paragraph breaks and does in okay job of showing links; but it completely drops all emphasis, like bold, italic, headers, and colors. My terminal can display all those styles just fine. I've also tried links, elinks, and w3m, but none of them seem to be able to handle any text styling. Some of them will do bold if you run them interactively, but none of them do italic or colors, and none of them will do bold with -dump, even if you tell them what terminal type you want to use. Why is that so hard?

I never did find a solution, but it's worth noting some useful sites I found along the way. Like tips for testing bold, italics etc. in a terminal:, and for testing whether the terminal supports italics, which gave me these useful shell functions:

echo -e "\e[1mbold\e[0m"
echo -e "\e[3mitalic\e[0m"
echo -e "\e[4munderline\e[0m"
echo -e "\e[9mstrikethrough\e[0m"
echo -e "\e[31mHello World\e[0m"
echo -e "\x1B[31mHello World\e[0m"

ansi()          { echo -e "\e[${1}m${*:2}\e[0m"; }
bold()          { ansi 1 "$@"; }
italic()        { ansi 3 "$@"; }
underline()     { ansi 4 "$@"; }
strikethrough() { ansi 9 "$@"; }
red()           { ansi 31 "$@"; }

And in testing, I found that a lot of fonts didn't offer italics. One that does is Terminus, so if your normal font doesn't, you can run a terminal with Terminus: xterm -fn '-*-terminus-bold-*-*-*-20-*-*-*-*-*-*-*'

Not that it matters since none of the text-mode browsers offer italic anyway. But maybe you'll find some other use for italic in a terminal.

Tags: mutt, email, linux
[ 18:28 Jun 05, 2017    More linux | permalink to this entry | ]

Viewer for email attachments in Office formats

Update, December 2022:
viewmailattachments has been integrated with another mutt helper, viewhtmlmail.py, which can show HTML messages complete with embedded images. It's described in the article View Mail Attachments from Mutt and the script is at viewmailattachments.py. It no longer uses the "please wait" screen described in this article, but the rest of the discussion still applies.

I seem to have fallen into a nest of Mac users whose idea of email is a text part, an HTML part, plus two or three or seven attachments (no exaggeration!) in an unholy combination of .DOC, .DOCX, .PPT and other Microsoft Office formats, plus .PDF.

Converting to text in mutt

As a mutt user who generally reads all email as plaintext, normally my reaction to a mess like that would be "Thanks, but no thanks". But this is an organization that does a lot of good work despite their file format habits, and I want to help.

In mutt, HTML mail attachments are easy. This pair of entries in ~/.mailcap takes care of them:

text/html; firefox 'file://%s'; nametemplate=%s.html
text/html; lynx -dump %s; nametemplate=%s.html; copiousoutput

Then in .muttrc, I have

auto_view text/html
alternative_order text/plain text

If a message has a text/plain part, mutt shows that. If it has text/html but no text/plain, it looks for the "copiousoutput" mailcap entry, runs the HTML part through lynx (or I could use links or w3m) and displays that automatically. If, reading the message in lynx, it looks to me like the message has complex formatting that really needs a browser, I can go to mutt's attachments screen and display the attachment in firefox using the other mailcap entry.

Word attachments are not quite so easy, especially when there are a lot of them. The straightforward way is to save each one to a file, then run LibreOffice on each file, but that's slow and tedious and leaves a lot of temporary files behind. For simple documents, converting to plaintext is usually good enough to get the gist of the attachments. These .mailcap entries can do that:

application/msword; catdoc %s; copiousoutput
application/vnd.openxmlformats-officedocument.wordprocessingml.document; docx2txt %s -; copiousoutput

Alternatives to catdoc include wvText and antiword.

But none of them work so well when you're cross-referencing five different attachments, or for documents where color and formatting make a difference, like mail from someone who doesn't know how to get their mailer to include quoted text, and instead distinguishes their comments from the text they're replying to by making their new comments green (ugh!) For those, you really do need a graphical window.

I decided what I really wanted (aside from people not sending me these crazy emails in the first place!) was to view all the attachments as tabs in a new window. And the obvious way to do that is to convert them to formats Firefox can read.

Converting to HTML

I'd used wvHtml to convert .doc files to HTML, and it does a decent job and is fairly fast, but it can't handle .docx. (People who send Office formats seem to distribute their files fairly evenly between DOC and DOCX. You'd think they'd use the same format for everything they wrote, but apparently not.) It turns out LibreOffice has a command-line conversion program, unoconv, that can handle any format LibreOffice can handle. It's a lot slower than wvHtml but it does a pretty good job, and it can handle .ppt (PowerPoint) files too.

For PDF files, I tried using pdftohtml, but it doesn't always do so well, and it's hard to get it to produce a single HTML file rather than a directory of separate page files. And about three quarters of PDF files sent through email turn out to be PDF in name only: they're actually collections of images of single pages, wrapped together as a PDF file. (Mostly, when I see a PDF like that I just skip it and try to get the information elsewhere. But I wanted my program at least to be able to show what's in the document, and let the user choose whether to skip it.) In the end, I decided to open a firefox tab and let Firefox's built-in PDF reader show the file, though popping up separate mupdf windows is also an option.

I wanted to show the HTML part of the email, too. Sometimes there's formatting there (like the aforementioned people whose idea of quoting messages is to type their replies in a different color), but there can also be embedded images. Extracting the images and showing them in a browser window is a bit tricky, but it's a problem I'd already solved a couple of years ago: Viewing HTML mail messages from Mutt (or other command-line mailers).

Showing it all in a new Firefox window

So that accounted for all the formats I needed to handle. The final trick was the firefox window. Since some of these conversions, especially unoconv, are quite slow, I wanted to pop up a window right away with a "converting, please wait..." message. Initially, I used a javascript: URL, running the command:

firefox -new-window "javascript:document.writeln('<br><h1>Translating documents, please wait ...</h1>');"

I didn't want to rely on Javascript, though. A data: URL, which I hadn't used before, can do the same thing without javascript:

firefox -new-window "data:text/html,<br><br><h1>Translating documents, please wait ...</h1>"

But I wanted the first attachment to replace the contents of that same window as soon as it was ready, and then subsequent attachments open a new tab in that window. But it turned out that firefox is inconsistent about what -new-window and -new-tab do; there's no guarantee that -new-tab will show up in the same window you recently popped up with -new-window, and running just firefox URL might open in either the new window or the old, in a new tab or not, or might not open at all. And things got even more complicated after I decided that I should use -private-window to open these attachments in private browsing mode.

In the end, the only way firefox would behave in a repeatable, predictable way was to use -private-window for everything. The first call pops up the private window, and each new call opens a new tab in the private window. If you want two separate windows for two different mail messages, you're out of luck: you can't have two different private windows. I decided I could live with that; if it eventually starts to bother me, I can always give up on Firefox and write a little python-webkit wrapper to do what I need.

Using a file redirect instead

But that still left me with no way to replace the contents of the "Please wait..." window with useful content. Someone on #firefox came up with a clever idea: write the content to a page with a meta redirect.

So initially, I create a file pleasewait.html that includes the header:

<meta http-equiv="refresh" content="2;URL=pleasewait.html">

(other HTML, charset information, etc. as needed). The meta refresh means Firefox will reload the file every two seconds. When the first converted file is ready, I just change the header to redirect to URL=first_converted_file.html. Meanwhile, I can be opening the other documents in additional tabs.

Finally, I added the command to my .muttrc. When I'm viewing a message either in the index or pager screens, F10 will call the script and decode all the attachments.

macro index <F10> "<pipe-message>~/bin/viewmailattachments\n" "View all attachments in browser"
macro pager <F10> "<pipe-message>~/bin/viewmailattachments\n" "View all attachments in browser"

Whew! It was trickier than I thought it would be. But I find I'm using it quite a bit, and it takes a lot of the pain out of those attachment-full emails.

The script is available at: viewmailattachments.py on GitHub.

Tags: email, mutt, programming, python, firefox, linux
[ 15:18 Oct 15, 2015    More linux | permalink to this entry | ]

Email is not private

Microsoft is in trouble this week -- someone discovered Microsoft read a user's Hotmail email as part of an internal leak investigation (more info here: Microsoft frisked blogger's Hotmail inbox, IM chat to hunt Windows 8 leaker, court told). And that led The Verge to publish the alarming news that it's not just Microsoft -- any company that handles your mail can also look at the contents: "Free email also means someone else is hosting it; they own the servers, and there's no legal or technical safeguard to keep them from looking at what's inside."

Well, yeah. That's true of any email system -- not just free webmail like Hotmail or Gmail. I was lucky enough to learn that lesson early.

I was a high school student in the midst of college application angst. The physics department at the local university had generously given me an account on their Unix PDP-11 since I'd taken a few physics classes there.

I had just sent off some sort of long, angst-y email message to a friend at another local college, laying my soul bare, worrying about my college applications and life choices and who I was going to be for the rest of my life. You know, all that important earth-shattering stuff you worry about when you're that age, when you're sure that any wrong choice will ruin the whole rest of your life forever.

And then, fiddling around on the Unix system after sending my angsty mail, I had some sort of technical question, something I couldn't figure out from the man pages, and I sent off a quick question to the same college friend.

A couple of minutes later, I had new mail. From root. (For non-Unix users, root is the account of the system administrator: the person in charge of running the computer.) The mail read:

Just ask root. He knows all!

followed by a clear, concise answer to my technical question.

Great! ... except I hadn't asked root. I had asked my friend at a college across town.

When I got the email from root, it shook me up. His response to the short technical question was just what I needed ... but if he'd read my question, did it mean he'd also read the long soul-baring message I'd sent just minutes earlier? Was he the sort of snoop who spent his time reading all the mail passing through the system? I wouldn't have thought so, but ...

I didn't ask; I wasn't sure I wanted to know. Lesson learned. Email isn't private. Root (or maybe anyone else with enough knowledge) can read your email.

Maybe five years later, I was a systems administrator on a Sun network, and I found out what must have happened. Turns out, when you're a sysadmin, sometimes you see things like that without intending to. Something goes wrong with the email system, and you're trying to fix it, and there's a spool directory full of files with randomized names, and you're checking on which ones are old and which are recent, and what has and hasn't gotten sent ... and some of those files have content that includes the bodies of email messages. And sometimes you see part of what's in them. You're not trying to snoop. You don't sit there and read the full content of what your users are emailing. (For one thing, you don't have time, since typically this happens when you're madly trying to fix a critical email problem.) But sometimes you do see snippets, even if you're not trying to. I suspect that's probably what happened when "root" replied to my message.

And, of course, a snoopy and unethical system administrator who really wanted to invade his users' privacy could easily read everything passing through the system. I doubt that happened on the college system where I had an account, and I certainly didn't do it when I was a sysadmin. But it could happen.

The lesson is that email, if you don't encrypt it, isn't private. Think of email as being like a postcard. You don't expect Post Office employees to read what's written on the postcard -- generally they have better things to do -- but there are dozens of people who handle your postcard as it gets delivered who could read it if they wanted to.

As the Verge article says, "Peeking into your clients' inbox is bad form, but it's perfectly legal."

Of course, none of this excuses Microsoft's deliberately reading Hotmail mailboxes. It is bad form, and amid the outcry Microsoft has changed its Hotmail snooping policies somewhat, saying they'll only snoop deliberately in certain cases).

But the lesson for users is: if you're writing anything private, anything you don't want other people to read ... don't put it on a postcard. Or in unencrypted email.

Tags: email, privacy, tech
[ 14:59 Mar 27, 2014    More tech/email | permalink to this entry | ]

Viewing HTML mail messages from Mutt (or other command-line mailers)

Update: the script described in this article has been folded into another script called viewmailattachments.py.

Command-line mailers like mutt have one disadvantage: viewing HTML mail with embedded images. Without images, HTML mail is no problem -- run it through lynx, links or w3m. But if you want to see images in place, how do you do it?

Mutt can send a message to a browser like firefox ... but only the textual part of the message. The images don't show up.

That's because mail messages include images, not as separate files, but as attachments within the same file, encoded it a format known as MIME (Multipurpose Internet Mail Extensions). An image link in the HTML, instead of looking like <img src="picture.jpg">., will instead look something like <img src="cid:0635428E-AE25-4FA0-93AC-6B8379300161">. (Apple's Mail.app) or <img src="cid:1.3631871432@web82503.mail.mud.yahoo.com">. (Yahoo's webmail).

CID stands for Content ID, and refers to the ID of the image as it is encoded in MIME inside the image. GUI mail programs, of course, know how to decode this and show the image. Mutt doesn't.

A web search finds a handful of shell scripts that use the munpack program (part of the mpack package on Debian systems) to split off the files; then they use various combinations of sed and awk to try to view those files. Except that none of the scripts I found actually work for messages sent from modern mailers -- they don't decode the CID links properly.

I wasted several hours fiddling with various shell scripts, trying to adjust sed and awk commands to figure out the problem, when I had the usual epiphany that always eventually arises from shell script fiddling: "Wouldn't this be a lot easier in Python?"

Python's email package

Python has a package called email that knows how to list and unpack MIME attachments. Starting from the example near the bottom of that page, it was easy to split off the various attachments and save them in a temp directory. The key is

import email

fp = open(msgfile)
msg = email.message_from_file(fp)
fp.close()

for part in msg.walk():

That left the problem of how to match CIDs with filenames, and rewrite the links in the HTML message accordingly.

The documentation on the email package is a bit unclear, unfortunately. For instance, they don't give any hints what object you'll get when iterating over a message with walk, and if you try it, they're just type 'instance'. So what operations can you expect are legal on them? If you run help(part) in the Python console on one of the parts you get from walk, it's generally class Message, so you can use the Message API, with functions like get_content_type(), get_filename(). and get_payload().

More useful, it has dictionary keys() for the attributes it knows about each attachment. part.keys() gets you a list like

['Content-Type', 
 'Content-Transfer-Encoding',
 'Content-ID',
 'Content-Disposition' ]

So by making a list relating part.get_filename() (with a made-up filename if it doesn't have one already) to part['Content-ID'], I'd have enough information to rewrite those links.

Case-insensitive dictionary matching

But wait! Not so simple. That list is from a Yahoo mail message, but if you try keys() on a part sent by Apple mail, instead if will be 'Content-Id'. Note the lower-case d, Id, instead of the ID that Yahoo used.

Unfortunately, Python doesn't have a way of looking up items in a dictionary with the key being case-sensitive. So I used a loop:

    for k in part.keys():
        if k.lower() == 'content-id':
            print "Content ID is", part[k]

Most mailers seem to put angle brackets around the content id, so that would print things like "Content ID is <14.3631871432@web82503.mail.mud.yahoo.com>". Those angle brackets have to be removed, since the CID links in the HTML file don't have them.

for k in part.keys():
    if k.lower() == 'content-id':
        if part[k].startswith('<') and part[k].endswith('>'):
            part[k] = part[k][1:-1]

But that didn't work -- the angle brackets were still there, even though if I printed part[k][1:-1] it printed without angle brackets. What was up?

Unmutable parts inside email.Message

It turned out that the parts inside an email Message (and maybe the Message itself) are unmutable -- you can't change them. Python doesn't throw an exception; it just doesn't change anything. So I had to make a local copy:

for k in part.keys():
    if k.lower() == 'content-id':
        content_id = part[k]
        if content_id.startswith('<') and content_id.endswith('>'):
            content_id = content_id[1:-1]

and then save content_id, not part[k], in my list of filenames and CIDs.

Then the rest is easy. Assuming I've built up a list called subfiles containing dictionaries with 'filename' and 'Content-Id', I can do the substitution in the HTML source:

    htmlsrc = html_part.get_payload(decode=True)
    for sf in subfiles:
        htmlsrc = re.sub('cid: ?' + sf['Content-Id'],
                         'file://' + sf['filename'],
                         htmlsrc, flags=re.IGNORECASE)

Then all I have to do is hook it up to a key in my .muttrc:

# macro  index  <F10>  "<copy-message>/tmp/mutttmpbox\n<enter><shell-escape>~/bin/viewhtmlmail.py\n" "View HTML in browser"
# macro  pager  <F10>  "<copy-message>/tmp/mutttmpbox\n<enter><shell-escape>~/bin/viewhtmlmail.py\n" "View HTML in browser"

Works nicely! Here's the complete script: viewhtmlmail.

Tags: email, mutt, programming, python, mime, cmdline
[ 11:49 Oct 07, 2013    More tech/email | permalink to this entry | ]

Mutt: Show HTML for sites that send broken HTML mail

Increasingly I'm seeing broken sites that send automated HTML mail with headers claiming it's plain text.

To understand what's happening, you have to know about something called MIME multipart/alternative.

MIME stands for Multipurpose Internet Mail Extensions: it's the way mail encodes different types of attachments, so you can attach images, music, PDF documents or whatever with your email.

If you send a normal plain text mail message, you don't need MIME. But as soon as you send anything else -- like an HTML message where you've made a word bold, changed color or inserted images -- you need it. MIME adds a Content-Type to the message saying "This is HTML mail, so you need to display it as HTML when you receive it" or "Here's a PDF attachment, so you need to display it in a PDF viewer". The headers for these two cases would look like this:

Content-Type: text/html
Content-Type: application/pdf

A lot of mail programs, for reasons that have never been particularly clear, like to send two copies of every mail message: one in plain text, one in HTML. They're two copies of the same message -- it's just that one version has fancier formatting than the other. The MIME header that announces this is

Content-Type: multipart/alternative

because the two versions, text and HTML, are alternative versions of the same message. The recipient need only read one, not both. Inside the multipart/alternative section there will be further MIME headers, one saying Content-Type: text/plain, where it puts the text of your message, and one Content-Type: text/html, where it puts HTML source code.

This mostly works fine for real mail programs (though it's a rather silly waste of bandwidth, sending double copies of everything for no particularly good reason, and personally I always configure the mailers I use to send only one copy at a time). But increasingly I'm seeing automated mail robots that send multipart/alternative mail, but do it wrong: they send HTML for both parts, or they send a valid HTML part and a blank text part.

Why don't the site owners notice the problem?

You wouldn't ever notice a problem if you use the default configuration on most mailers, to show the HTML part if at all possible. But most mail programs give you an option to show the text part if there is one. That way, you don't have to worry about those people who like to send messages in pink blinking text on a plaid background -- all you see is the text.

If your mailer is configured to show plain text, for most messages you'll see just text -- no colors, no blinking, no annoyances. But for mail sent by these misconfigured mail robots, what you'll see is HTML source code.

I've seen this in several places -- lots of spammers do it (who cares? I was going to delete the message anyway), and one of the local astronomy clubs does it so I've long since stopped trying to read their announcements. But the latest place I've seen this is one that ought to know better: Coursera. They apparently reconfigured their notification system recently, and I started getting course notifications that look like this:

/* Client-specific Styles */
#outlook a{padding:0;} /* Force Outlook to provide a "view in browser" button.
*/
body{width:100% !important;} .ReadMsgBody{width:100%;}
.ExternalClass{width:100%;} /* Force Hotmail to display emails at full width */
body{-webkit-text-size-adjust:none;} /* Prevent Webkit platforms from changing
default text sizes. */
/* Reset Styles */
body{margin:0; padding:0;}
img{border:0; height:auto; line-height:100%; outline:none;
text-decoration:none;}
table td{border-collapse:collapse;}
#backgroundTable{height:100% !important; margin:0; padding:0; width:100%
!important;}
p {margin-top: 14px; margin-bottom: 14px;}
/* /\/\/\/\/\/\/\/\/\/\ STANDARD STYLING: PREHEADER /\/\/\/\/\/\/\/\/\/\ */
.preheaderContent div a:link, .preheaderContent div a:visited, /* Yahoo! Mail
Override */ .preheaderContent div a .yshortcuts /* Yahoo! Mail Override */{
  color: #3b6e8f;

... and on and on like that. You get the idea. It's unreadable, even by a geek who knows HTML pretty well. It would be fine in the HTML part of the message -- but this is what they're sending in the text/plain part.

I filed a bug, but Coursera doesn't have a lot of staff to respond to bug reports and it might be quite some time before they fix this. Meanwhile, I don't want to miss notifications for the algorithms course I'm currently taking. So I needed a workaround.

How to work around the problem in mutt

I found one for mutt at alternative_order and folder-hook. When in my "classes" folder, I use a folder hook to tell mutt to prefer text/html format over text/plain, even though my default is text/plain. Then you also need to add a default folder hook to set the default back for every other folder -- mutt folder hooks are frustrating in that way.

The two folder hooks look like this:

folder-hook . 'set unalternative_order *; alternative_order text/plain text'

# Prefer the HTML part but only for Coursera,
# since it sends HTML in the text part.
folder-hook =in/coursera 'unalternative_order *; alternative_order text/html'

alternative_order specifies which types you'd most like to read. unalternative_order is a lot less clear; the documentation says it "removes a mime type from the alternative_order list", but doesn't say anything more than that. What's the syntax? What's the difference between using unalternative_order or just re-setting alternative_order? Why do I have to specify it with * in both places? No one seems to know.

So it's a little unsatisfying, and perhaps not the cleanest way. But it does work around the bug for sites where you really need a way to read the mail.

Update: I also found this discussion of alternative_order which gives a nice set of key bindings to toggle interactively between the various formats. It was missing some backslashes, so I had to fiddle with it slightly to get it to work. Put this in .muttrc:

macro pager ,@aoh= "\
<enter-command> unalternative_order *; \
alternative_order text/enriched text/html text/plain text;\
macro pager A ,@aot= 'toggle alternative order'<enter>\
<exit><display-message>"

macro pager ,@aot= "\
<enter-command> unalternative_order *; \
alternative_order text/enriched text/plain text/html text;\
macro pager A ,@aoh= 'toggle alternative order'<enter>\
<exit><display-message>"

macro pager A ,@aot= "toggle alternative order"

Then just type A (capital A) to toggle between formats. If it doesn't change the first time you type A, type another one and it should redisplay. I've found it quite handy.

Tags: mutt, email, html
[ 15:13 Jul 29, 2013    More tech/email | permalink to this entry | ]

Decoding RFC 2047 email headers (like spam Subjects in other charsets)

Having not had much luck with spam filtering solutions like SpamAssassin, I'm forever having to add new spam filters by hand. For instance, after about the sixth time I get "President Waives Refi Requirement" or "Melt your fat! MUST WATCH this video now!" within a couple of hours, I'm pretty tired of it and don't want to see any more of them.

With mail filtering programs like procmail or maildrop, it's easy enough to match a pattern like "Subject:.*Refi Requirement" or "Subject:.*Melt your fat" and filter that message to a spam folder (or /dev/null).

But increasingly, I add patterns I'm seeing in spam messages, and yet the messages with those patterns keep coming in. Why? Because the spammers are using RFC 2047 to encode the subject into some other character set.

Here's how it works. A spammer sends a subject line that looks something like this:

Subject: =?utf-8?B?U3RvcCBPdmVycGF5aW5nIGZvciBQcmludGVyIEluaw==?=

Mail programs are smart enough to decode this into:

Subject: Stop Overpaying for Printer Ink

but spam filtering programs often aren't, so your "printer ink" filter won't catch it. And if you look through your spam folder with tools like grep to see why it didn't get caught, or to find particularly spammy subjects that might call for a filter (grep Subject spamfolder | sort is pretty handy), these encoded subjects will be incognito.

I briefly tried setting up a filter that spam-filed anything with =? in the Subject line. But that's way too broad a brush -- not all people there are legitimate reasons for using other charsets even in English language email. It's relatively rare, but it happens. And some bots, notably the Adafruit forum notification bot and the bot that sends out announcements from my alma mater, unaccountably encode the charset even when they're sending mail entirely in US ASCII.

So what's really needed is not to filter out all messages that specify a charset, but to decode the Subject so the spam filter can see it and filter it accordingly.

How? I couldn't find any ready-made tool available for Linux that could decode RFC 2047 headers; but the Python email package makes decoding a one-line task. In the Python interpreter:

$ python
Python 2.7.3 (default, Aug  1 2012, 05:16:07) 
Type "help", "copyright", "credits" or "license" for more information.
>>> import email
>>> email.Header.decode_header("Subject: =?utf-8?B?U3RvcCBPdmVycGF5aW5nIGZvciBQcmludGVyIEluaw==?=")
[('Subject:', None), ('Stop Overpaying for Printer Ink', 'utf-8')]
>>>

So it's easy to write a script that can pull headers out of email messages (files) and decode them. Just look for the line starting with the header you want to match -- e.g. "Subject:" -- and pass that line to email.Header.decode_header().

Only one snag. If the subject is longer than about 20 characters, spammers will often opt to split it up into multiple groups, sometimes even in different character sets. So for example, you might see something like this, spread over multiple lines:

Subject: =?windows-1252?Q?Earn_your_degree_=97_on_your_time?=
        =?windows-1252?Q?_and_terms?=

The script has to handle that too. If it's reading a header, it has to check the next line, and if that line begins with whitespace, treat it as more of the header.

The resulting script, decodemail.py (on github), seems pretty handy and should be able to be plugged in to a mail filtering program.

Tags: email, spam
[ 21:45 Dec 08, 2012    More programming | permalink to this entry | ]

Deleting email from a mail server with Python

How do you delete email from a mail server without downloading or reading it all?

Why? Maybe you got a huge load of spam and you need to delete it. Maybe you have your laptop set up to keep a copy of your mail on the server so you can get it on your desktop later ... but after a while you realize it's not worth downloading all that mail again. In my case, I use an ISP that keeps copies of all mail forwarded from one alias to another, so I periodically need to clean out the copies.

There are quite a few reasons you might want to delete mail without reading it ... so I was surprised to find that there didn't seem to be any easy way to do so.

But POP3 is a fairly simple protocol. How hard could it be to write a Python script to do what I needed?

Not hard at all, in fact. The poplib package does most of the work for you, encapsulating both the networking and the POP3 protocol. It even does SSL, so you don't have to send your password in the clear.

Once you've authenticated, you can list() messages, which gives you a status and a list of message numbers and sizes, separated by a space. Just loop through them and delete each one.

Here's a skeleton program to delete messages:

server = "mail.example.com"
port = 995
user = "myname"
passwd = "seekrit"

pop = poplib.POP3_SSL(server, port)
pop.user(user)
pop.pass_(passwd)

poplist = pop.list()
if poplist[0].startswith('+OK') :
    msglist = poplist[1]
    for msgspec in msglist :
        # msgspec is something like "3 3941", 
        # msg number and size in octets
        msgnum = int(msgspec.split(' ')[0])
        print "Deleting msg %d\r" % msgnum,
        pop.dele(msgnum)
    else :
        print "No messages for", user
else :
    print "Couldn't list messages: status", poplist[0]
pop.quit()

Of course, you might want to add more error checking, loop through a list of users, etc. Here's the full script: deletemail.

Tags: email, python, programming
[ 17:41 Aug 25, 2011    More programming | permalink to this entry | ]

Changing your email address in Yahoo Groups

A while ago I switched ISPs, and maintaining a lot of email addresses got more complicated. So I decided to consolidate.

But changing your email address turns out to be tricky on some sites. For example, on Amazon it apparently requires a phone call to customer support (I haven't gotten around to it yet, but that's what their email support people told me to do).

Then there's Yahoo groups. I'm in quite a few groups, so when I made the switch, I went to groups.yahoo.com, added a valid address and made it my primary address. Great -- thought I was done.

Weeks later, it occurred to me that I hadn't been getting any mail from a bunch of groups I used to get mail from. I went to Yahoo groups and clicked around for five minutes trying to find something that would show me my email addresses. Eventually I gave up on that, went to one of the groups I hadn't been getting, and saw a notice at the top:

The email address you are using for this group is currently bouncing. More info here.

So naturally, I clicked on the More info here link, and got taken to a page that said:

Groups Error: No Permission

No Permission
You do not have permission to access this page.

Gosh, that's some helpful info, Yahoo!

So how do you really change it?

There are lots of ways to get to the Yahoo Groups "Manage your email addresses" page -- but it shows only the new address, listed as primary, as primary, and doesn't show the old address where it's actually trying to send all the mail. No way to delete it from there.

Now, you can Edit membership in any particular group: that shows both the old nonworking address (with the box checked) and the new one (check the box to change it). Great -- so I'm supposed to do that for all 25 or so groups I'm in? Seriously?

After much searching, I finally found an old discussion thread with a link to the Edit my groups page. Scroll down to the bottom and look for "Set all of the above to".

It's still not a one-step operation -- my groups are spread across three pages and there's no "View all on one page", and each time you submit a page, it takes you back to "View groups" mode so you have to click on the next page, then click on "Edit groups" again. Still, it's a heck of a lot faster than going through all the groups one by one.

In theory it's all changed now. But then, I thought that last time ... time will tell whether the mail actually starts flowing again.

Meanwhile, Yahoo developers: you might want to take a look at that "More info" page that just gives a permission error.

Tags: email, web
[ 18:58 Aug 09, 2011    More tech | permalink to this entry | ]

Automated mail: check the plaintext part (or don't send one)

Funny thing happened last week.

I'm on the mailing list for a volunteer group. Round about last December, I started getting emails every few weeks congratulating me on RSVPing for the annual picnic meeting on October 17.

This being well past October, when the meeting apparently occurred -- and considering I'd never heard of the meeting before, let alone RSVPed for it -- I couldn't figure out why I kept getting these notices.

After about the third time I got the same notice, I tried replying, telling them there must be something wrong with their mailer. I never got a reply, and a few weeks later I got another copy of the message about the October meeting.

I continued sending replies, getting nothing in return -- until last week, when I got a nice apologetic note from someone in the organization, and an explanation of what had happened. And the explanation made me laugh.

Seems their automated email system sends messages as multipart, both HTML and plaintext. Many user mailers do that; if you haven't explicitly set it to do otherwise, you yourself are probably sending out two copies of every mail you send, one in HTML and one in plain text.

But in this automated system, the plaintext part was broken. When it sent out new messages in HTML format, apparently for the plaintext part it was always attaching the same old message, this message from October. Apparently no one in the organization had ever bothered to check the configuration, or looked at the plaintext part, to realize it was broken. They probably didn't even know it was sending out multiple formats.

I have my mailer configured to show me plaintext in preference to HTML. Even if I didn't use a text mailer (mutt), I'd still use that setting -- Thunderbird, Apple Mail, Claws and many other mailers offer it. It protects you from lots of scams and phishing attacks, "web bugs" to track you,, and people who think it's the height of style to send mail in blinking yellow comic sans on a red plaid background.

And reading the plaintext messages from this organization, I'd never noticed that the message had an HTML part, or thought to look at it to see if it was different.

It's not the first time I've seen automated mailers send multipart mail with the text part broken. An astronomy club I used to belong to set up a new website last year, and now all their meeting notices, which used to come in plaintext over a Yahoo groups mailing list, have a text part that looks like this actual example from a few days ago:

Subject: Members' Night at the Monthly Meeting
<p>&#60;&#115;&#116;&#121;&#108;&#101;&#32;&#116;&#121;&#112;&#101;&#61;&#34;&#1
16;&#101;&#120;&#116;&#47;&#99;&#115;&#115;&#34;&#62;@font-face {
  font-family: "ＭＳ 明朝";
}@font-face {
  font-family: "ＭＳ 明朝";
}@font-face {
  font-family: "Cambria";
}p.MsoNormal, li.MsoNormal, div.MsoNormal { margin: 0in 0in 0.0001pt; font-size:
12pt; font-family: Cambria; }a:link, span.MsoHyperlink { color: blue;
text-decoration: underline; }a:visited, span.MsoHyperlinkFollowed { color:
purple; text-decoration: underline; }.MsoChpDefault { font-family: Cambria;
}div.WordSection1 { page: WordSection1;
}&#60;&#47;&#115;&#116;&#121;&#108;&#101;&#62;
<p class="MsoNormal">Friday April 8<sup>th</sup> is members’ night at the
monthly meeting of the PAS.<span style="">&#160; </span>We are asking for
anyone, who has astronomical photographs that they would like to share, to
present them at the meeting.<span style="">&#160; </span>Each presenter will
have about 15 minutes to present and discuss his pictures.<span style=""> We
already have some presenters. &#160; </span></p>
<p class="MsoNormal">&#160;</p>

... on and on for pages full of HTML tags and no line breaks. I contacted the webmaster, but he was just using packaged software and didn't seem to grok that the software was broken and was sending HTML for the plaintext part as well as for the HTML part. His response was fairly typical: "It looks fine to me". I eventually gave up even trying to read their meeting announcements, and now I just delete them.

The silly thing about this is that I can read HTML mail just fine, if they'd just send HTML mail. What causes the problem is these automated systems that insist on sending both HTML and plaintext, but then the plaintext part is wrong. You'll see it on a lot of spam, too, where the plaintext portion says something like "Get a better mailer" (why? so I can see your phishing attack in all its glory?)

Folks, if you're setting up an automated email system, just pick one format and send it. Don't configure it to send multiple formats unless you're willing to test that all the formats actually work.

And developers, if you're writing an automated email system: don't use MIME multipart/alternative by default unless you're actually sending the same message in different formats. And if you must use multipart ... test it. Because your users, the administrators deploying your system for their organizations, won't know how to.

Tags: tech, email
[ 14:19 Mar 27, 2011    More tech/email | permalink to this entry | ]

"Joe-job" spam (forged From addresses)

I'm in a Yahoo group where a spammer just posted a message that looked like it was coming from someone in the group, so Yahoo allowed it.

The list owner posted a message about using good passwords so your account isn't hacked since that causes problems for everyone.

Of course, that's good advice and using good passwords is always a good idea. But I though this sounded more like a Joe-job spam, in which the spammer forges the From address to look like it's coming from someone else.

Normal users encounter this in two ways:

1. You start getting tons of bounce messages that look as though you sent spam to hundreds of people and they're refusing it.
2. You see spam that looks like it came from a friend of yours, or spam on a mailing list that looks like it came from a legitimate member of that list.

Since this sort of attack is so common, I felt the victim didn't deserve being harangued about not having set up a good password. So I posted a short note to the list explaining about Joe-jobs. But to make the point, I forged the From address of the list owner. Indeed, it got through Yahoo and out to the list just fine:

[ ... ] the spam probably wasn't from a bad password. It was probably just a spammer forging the header to look like it's from a legitimate user. It's called a "joe-job": http://en.wikipedia.org/wiki/Joe-job

To illustrate, I've changed the From address on this message to look like it's coming from Adam. I have not hacked [listowner]'s account or guessed his password or anything else. If this works, and looks like it came from [listowner], then the spam could have been done the same way -- and there's no need to blame the owner of the account, or accuse them of having a bad password.

Why does this work? Why doesn't Yahoo just block messages from user@isp.com if the mail doesn't come from isp.com?

They can't! Many, many people don't send mail from the domains in their email addresses. In effect, people forge their From header all the time. Here are some examples:

• You're using you@gmail.com, but you're using Thunderbird or Eudora or Evolution or something to read and send mail from home.
• You're on your computer at home, but you're sending work-related email from your work account.
• You're on your laptop, using Thunderbird or whatever, mailing from you@isp.com, but you're at a friend's house or a hotel or conference or somewhere.
• You're sending mail from a public terminal somewhere (eek, do people really type their mail info in to these things?)
• You're reading and sending mail from a mobile phone.
• You're sending mail from your own domain, me@mydomain.com, but you're at home or somewhere else other than wherever mydomain.com is hosted.

If mailing lists rejected posts in all these cases, people would be pretty annoyed. So they don't. But that means that now and then, some Joe-job spam gets through to mailing lists. Unfortunately.

(Update: The message that inspired this may very well have been a hacked password after all case, based on the mail headers. But I found that a lot of people didn't know about Joe-jobbing, so I thought this was worth writing up anyway.)

Tags: security, email, joe-job, spam
[ 22:28 Apr 13, 2010    More tech/email | permalink to this entry | ]

Fetchmail without Postfix

I've been using fetchmail for a couple of years to get mail from the mail server to my local machine. But it had one disadvantage: it meant that I had to have postfix (or a similar large and complex MTA) configured and running on every machine I use, even the lightweight laptop.

I run procmail to filter my mail into folders -- Linuxchix mail into one folder, GIMP mailing lists into another, and so forth -- and it seemed like it ought to be possible for fetchmail to call procmail directly, without going through postfix.

I found several suggestions on the web -- for instance, fetchmail-procmail-sendmail -- but they didn't work for me. fetchmail downloaded each message, passed it to procmail, and procmail appended it to the relevant mailbox without the appropriate "From " header that mail programs need to tell when each new message starts.

Finally, on a tip from bma on #linuxchix and after a little experimentation, I added this line to ~/.fetchmailrc:

mda /usr/bin/procmail -f %F -m /home/username/.procmailrc

Works great! And it's a lot faster than going through postfix.

Tags: linux, email, fetchmail, procmail
[ 15:07 Dec 15, 2009    More tech/email | permalink to this entry | ]

Pruning those huge Spamassassin files

During a server backup, Dave complained that my .spamassasin directory was taking up 87Mb. I had to agree, that seemed a bit excessive.

The only two large files were auto-whitelist at 42M and bayes_seen at 41M. Apparently these never get pruned by spamassassin.

Unfortunately, these are binary files, so you can't just edit them and remove the early stuff, and spamassassin doesn't seem to have any documentation on how to prune their data files. A thread on the Spamassassin Users list on managing Spamassassin data says it's okay to delete bayes_seen and it will be regenerated.

For pruning auto-whitelist, that same post suggests a program called check-whitelist that is only available in a spamassassin source tarball -- it's not installed as part of distro packages. Run this with --clean. But a search on the spamassassin.com wiki turns up an entry on AutoWhitelist that says you should use tools/sa-awlUtil instead (it doesn't say how to run it or where to get it -- presumably download a source tarball and then RTFSC -- read the source code?)

Really, I'm not sure auto whitelisting is such a good idea anyway, especially auto whitelist entries from several years ago, so I opted for a simpler solution: removing the auto-whitelist file at the same time that I removed bayes_seen. Indeed, both files were immediately generated as new mail came in, but they were now much smaller.

I've run for a few weeks since doing that, and I'm not noticing any difference in either the number of false positives or false negatives. (Both are, unfortuantely, large enough to be noticable, but that was true before the change as well.)

Tags: spam, spamassassin, email
[ 20:38 May 07, 2009    More tech/email | permalink to this entry | ]