Shallow Thoughts : : email

Automated mail: check the plaintext part (or don't send one)

Funny thing happened last week.

I'm on the mailing list for a volunteer group. Round about last December, I started getting emails every few weeks congratulating me on RSVPing for the annual picnic meeting on October 17.

This being well past October, when the meeting apparently occurred -- and considering I'd never heard of the meeting before, let alone RSVPed for it -- I couldn't figure out why I kept getting these notices.

After about the third time I got the same notice, I tried replying, telling them there must be something wrong with their mailer. I never got a reply, and a few weeks later I got another copy of the message about the October meeting.

I continued sending replies, getting nothing in return -- until last week, when I got a nice apologetic note from someone in the organization, and an explanation of what had happened. And the explanation made me laugh.

Seems their automated email system sends messages as multipart, both HTML and plaintext. Many user mailers do that; if you haven't explicitly set it to do otherwise, you yourself are probably sending out two copies of every mail you send, one in HTML and one in plain text.

But in this automated system, the plaintext part was broken. When it sent out new messages in HTML format, apparently for the plaintext part it was always attaching the same old message, this message from October. Apparently no one in the organization had ever bothered to check the configuration, or looked at the plaintext part, to realize it was broken. They probably didn't even know it was sending out multiple formats.

I have my mailer configured to show me plaintext in preference to HTML. Even if I didn't use a text mailer (mutt), I'd still use that setting -- Thunderbird, Apple Mail, Claws and many other mailers offer it. It protects you from lots of scams and phishing attacks, "web bugs" to track you,, and people who think it's the height of style to send mail in blinking yellow comic sans on a red plaid background.

And reading the plaintext messages from this organization, I'd never noticed that the message had an HTML part, or thought to look at it to see if it was different.

It's not the first time I've seen automated mailers send multipart mail with the text part broken. An astronomy club I used to belong to set up a new website last year, and now all their meeting notices, which used to come in plaintext over a Yahoo groups mailing list, have a text part that looks like this actual example from a few days ago:

Subject: Members' Night at the Monthly Meeting
<p>&#60;&#115;&#116;&#121;&#108;&#101;&#32;&#116;&#121;&#112;&#101;&#61;&#34;&#1
16;&#101;&#120;&#116;&#47;&#99;&#115;&#115;&#34;&#62;@font-face {
  font-family: "ＭＳ 明朝";
}@font-face {
  font-family: "ＭＳ 明朝";
}@font-face {
  font-family: "Cambria";
}p.MsoNormal, li.MsoNormal, div.MsoNormal { margin: 0in 0in 0.0001pt; font-size:
12pt; font-family: Cambria; }a:link, span.MsoHyperlink { color: blue;
text-decoration: underline; }a:visited, span.MsoHyperlinkFollowed { color:
purple; text-decoration: underline; }.MsoChpDefault { font-family: Cambria;
}div.WordSection1 { page: WordSection1;
}&#60;&#47;&#115;&#116;&#121;&#108;&#101;&#62;
<p class="MsoNormal">Friday April 8<sup>th</sup> is members’ night at the
monthly meeting of the PAS.<span style="">&#160; </span>We are asking for
anyone, who has astronomical photographs that they would like to share, to
present them at the meeting.<span style="">&#160; </span>Each presenter will
have about 15 minutes to present and discuss his pictures.<span style=""> We
already have some presenters. &#160; </span></p>
<p class="MsoNormal">&#160;</p>

... on and on for pages full of HTML tags and no line breaks. I contacted the webmaster, but he was just using packaged software and didn't seem to grok that the software was broken and was sending HTML for the plaintext part as well as for the HTML part. His response was fairly typical: "It looks fine to me". I eventually gave up even trying to read their meeting announcements, and now I just delete them.

The silly thing about this is that I can read HTML mail just fine, if they'd just send HTML mail. What causes the problem is these automated systems that insist on sending both HTML and plaintext, but then the plaintext part is wrong. You'll see it on a lot of spam, too, where the plaintext portion says something like "Get a better mailer" (why? so I can see your phishing attack in all its glory?)

Folks, if you're setting up an automated email system, just pick one format and send it. Don't configure it to send multiple formats unless you're willing to test that all the formats actually work.

And developers, if you're writing an automated email system: don't use MIME multipart/alternative by default unless you're actually sending the same message in different formats. And if you must use multipart ... test it. Because your users, the administrators deploying your system for their organizations, won't know how to.

Tags: tech, email
[ 13:19 Mar 27, 2011    More tech/email | permalink to this entry | comments ]

"Joe-job" spam (forged From addresses)

I'm in a Yahoo group where a spammer just posted a message that looked like it was coming from someone in the group, so Yahoo allowed it.

The list owner posted a message about using good passwords so your account isn't hacked since that causes problems for everyone.

Of course, that's good advice and using good passwords is always a good idea. But I though this sounded more like a Joe-job spam, in which the spammer forges the From address to look like it's coming from someone else.

Normal users encounter this in two ways:

1. You start getting tons of bounce messages that look as though you sent spam to hundreds of people and they're refusing it.
2. You see spam that looks like it came from a friend of yours, or spam on a mailing list that looks like it came from a legitimate member of that list.

Since this sort of attack is so common, I felt the victim didn't deserve being harangued about not having set up a good password. So I posted a short note to the list explaining about Joe-jobs. But to make the point, I forged the From address of the list owner. Indeed, it got through Yahoo and out to the list just fine:

[ ... ] the spam probably wasn't from a bad password. It was probably just a spammer forging the header to look like it's from a legitimate user. It's called a "joe-job": http://en.wikipedia.org/wiki/Joe-job

To illustrate, I've changed the From address on this message to look like it's coming from Adam. I have not hacked [listowner]'s account or guessed his password or anything else. If this works, and looks like it came from [listowner], then the spam could have been done the same way -- and there's no need to blame the owner of the account, or accuse them of having a bad password.

Why does this work? Why doesn't Yahoo just block messages from user@isp.com if the mail doesn't come from isp.com?

They can't! Many, many people don't send mail from the domains in their email addresses. In effect, people forge their From header all the time. Here are some examples:

• You're using you@gmail.com, but you're using Thunderbird or Eudora or Evolution or something to read and send mail from home.
• You're on your computer at home, but you're sending work-related email from your work account.
• You're on your laptop, using Thunderbird or whatever, mailing from you@isp.com, but you're at a friend's house or a hotel or conference or somewhere.
• You're sending mail from a public terminal somewhere (eek, do people really type their mail info in to these things?)
• You're reading and sending mail from a mobile phone.
• You're sending mail from your own domain, me@mydomain.com, but you're at home or somewhere else other than wherever mydomain.com is hosted.

If mailing lists rejected posts in all these cases, people would be pretty annoyed. So they don't. But that means that now and then, some Joe-job spam gets through to mailing lists. Unfortunately.

(Update: The message that inspired this may very well have been a hacked password after all case, based on the mail headers. But I found that a lot of people didn't know about Joe-jobbing, so I thought this was worth writing up anyway.)

Tags: security, email, joe-job, spam
[ 21:28 Apr 13, 2010    More tech/email | permalink to this entry | comments ]

Fetchmail without Postfix

I've been using fetchmail for a couple of years to get mail from the mail server to my local machine. But it had one disadvantage: it meant that I had to have postfix (or a similar large and complex MTA) configured and running on every machine I use, even the lightweight laptop.

I run procmail to filter my mail into folders -- Linuxchix mail into one folder, GIMP mailing lists into another, and so forth -- and it seemed like it ought to be possible for fetchmail to call procmail directly, without going through postfix.

I found several suggestions on the web -- for instance, fetchmail-procmail-sendmail -- but they didn't work for me. fetchmail downloaded each message, passed it to procmail, and procmail appended it to the relevant mailbox without the appropriate "From " header that mail programs need to tell when each new message starts.

Finally, on a tip from bma on #linuxchix and after a little experimentation, I added this line to ~/.fetchmailrc:

mda /usr/bin/procmail -f %F -m /home/username/.procmailrc

Works great! And it's a lot faster than going through postfix.

Tags: linux, email, fetchmail, procmail
[ 14:07 Dec 15, 2009    More tech/email | permalink to this entry | comments ]

Pruning those huge Spamassassin files

During a server backup, Dave complained that my .spamassasin directory was taking up 87Mb. I had to agree, that seemed a bit excessive.

The only two large files were auto-whitelist at 42M and bayes_seen at 41M. Apparently these never get pruned by spamassassin.

Unfortunately, these are binary files, so you can't just edit them and remove the early stuff, and spamassassin doesn't seem to have any documentation on how to prune their data files. A thread on the Spamassassin Users list on managing Spamassassin data says it's okay to delete bayes_seen and it will be regenerated.

For pruning auto-whitelist, that same post suggests a program called check-whitelist that is only available in a spamassassin source tarball -- it's not installed as part of distro packages. Run this with --clean. But a search on the spamassassin.com wiki turns up an entry on AutoWhitelist that says you should use tools/sa-awlUtil instead (it doesn't say how to run it or where to get it -- presumably download a source tarball and then RTFSC -- read the source code?)

Really, I'm not sure auto whitelisting is such a good idea anyway, especially auto whitelist entries from several years ago, so I opted for a simpler solution: removing the auto-whitelist file at the same time that I removed bayes_seen. Indeed, both files were immediately generated as new mail came in, but they were now much smaller.

I've run for a few weeks since doing that, and I'm not noticing any difference in either the number of false positives or false negatives. (Both are, unfortuantely, large enough to be noticable, but that was true before the change as well.)

Tags: spam, spamassassin, email
[ 19:38 May 07, 2009    More tech/email | permalink to this entry | comments ]