Sending Mail via Gmail using OAuth2 (2022 Edition)

Update: Google's OAuth2 turns out to be not a good way to send mail, because passwords have to be renewed weekly. So you probably want to use a GMail App Password instead. I'm leaving this article up in case there's some reason someone would actually want to use OAuth2 with GMail.

There's been lots of talk on mailing lists for various mail programs, like Alpine and Mutt, about Google's impending dropping of password access.

Although my regular email address is on a Linux server, I subscribe to several Google Groups. I use a gmail address for those, because Google Groups doesn't work well with non-gmail addresses (you can't view the archives or temporarily turn off mail, and unsubscribing may or may not work depending on the phase of the moon).

I prefer not to have to sign on to Google and use the clunky browser interface when I have a perfectly good mailer (I use mutt) on my computer. I send mail from mutt using a program called msmtp. But to post to a Google Group, I need to use Google's SMTP server. (SMTP is the Simple Mail Transfer Protocol, the way mail gets from one computer to another across the internet.)

Up to now, I've been using an msmtp configuration that includes my Gmail password. That requires clicking through several Gmail pages to enable the "Less Secure Apps" setting. Google resets that preference every month or so and I have to go find the "Less Secure Apps" page to click through the screens again; but aside from that, it works okay.

But now Google has announced they'll be removing support for password access on May 30, 2022. Or maybe it's actually May 22, the date they show on the "less secure apps" page if you're logged in). In any case, passwords are going to stop working very soon.

So anyone who's been using passwords and the "Less Secure Apps" setting to talk to gmail's SMTP server will need to switch to the XOAuth2 authentication mechanism instead. (App passwords apparently also work as long as you have 2-step verification turned on, though Google recommends against using them.)

There's been some worry that Google is going to start charging a fee to access Gmail via an API. But that doesn't seem to apply to individuals who just want to check their own mail; see details later in this article.

Enabling OAuth2

At first, I tried to use this article on configuring MSMTP for Gmail which uses the same steps as this article on Google's GitHub page. But they're both outdated, and use a step that no longer work. After flailing away for several days without success, I asked on the Mutt mailing list and was pointed to this collection of getmail6 recipes, which have the correct instructions and a working script. Here are the steps I followed.

Create and Configure a Gmail App

Log in to your google/gmail account, then go to some variant of https://code.google.com/apis/console . Most pages say to create a project, but I found it had already created a default project, so I just used that. Set a name for your new desktop app.

In Enabled APIs and Services, click on + ENABLE APIS AND SERVICES and enable the gmail API.

In Credentials, click on + CREATE CREDENTIALS and create an OAuth client ID. Click the Download button to download a JSON file containing your client_id and client_secret.

While defining your app, if you see an option to add a "scope", add https://mail.google.com/ . It's not clear if the scope is strictly necessary, but it's hard to add a scope later after you've already created your app, so do it if you see the opportunity. (I did find a way once to get back to a page that let me define a scope, but it took a lot of clicking around, and I neglected to write down the click path and haven't been able to find it again.)

Also, on the OAuth Consent screen, add your gmail address as a Test user.

Get Your OAuth2 Tokens

The goal is to get an OAuth2 "access token" and "refresh token". This was the hard part, because both the msmtp instructions and the Google ones use a script that no longer works.

The script in question is on Google's GitHub account: oauth2.py. It's a python2 script written in 2012 and last updated in 2018. Now its --generate_oauth2_token step 1 uses URLs that no longer work, so it takes you to a Google page, you click through and get a code to paste back into the script, and the script dies with a python traceback for: urllib.error.HTTPError: HTTP Error 401: Unauthorized. If you browse the GitHub Issues for that repository, you'll find lots of rewrites and Python 3 updates for that script, but none of them worked for me in 2022.

Fortunately there's another way, courtesy of getmail6. Following the getmail6 instructions, download getmail-gmail-xoauth-tokens, which is an updated and heavily rewritten version of that same Python script. (It works on its own; it doesn't require anything else from getmail6.)

Create a filename.json file starting with the JSON at line 299 of the getmail6 instructions, replacing your-gmail-user with your own gmail address, and client_id and client_secret with the appropriate values from your Google app (they're in the JSON you downloaded earlier, or you can get them by clicking around Google's API pages)

Authenticate in a Browser

Run:

getmail-gmail-xoauth-tokens --init /path/to/filename.json

This will spit out a long URL; visit that URL in a browser where you're logged in to your Google account. Click through a long series of "THIS IS DANGEROUS, ARE YOU SURE YOU TRUST THIS APP AND DON'T WANT TO CANCEL?" screens until it finally gives you a code, which you copy and paste back into the terminal where you ran getmail-gmail-xoauth-tokens. Then hit Enter in that terminal, and getmail-gmail-xoauth-tokens submits that code to Google, gets the tokens, and updates your filename.json file accordingly.

Now you can use getmail-gmail-xoauth-tokens /path/to/filename.json (without the --init) from your mail transfer program whenever you need to fetch from or send via gmail. For instance, in msmtp, the section looks like this:

account gmail
host smtp.gmail.com
port 587
protocol smtp
tls on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
auth oauthbearer
from username@gmail.com
user username@gmail.com
passwordeval /path/to/getmail-gmail-xoauth-tokens /path/to/filename.json

The getmail script apparently also works with Microsoft Outlook, though I can't vouch for that personally.

Publishing, Testing, and What About Charging for Domains?

While you're fiddling with the various Google Cloud pages for your app, you may see a Publish button and think you should click it once you have everything working. I recommend you not do this.

Remember that Alpine thread I mentioned at the beginning? Apparently Google may charge up to $75,000 per year to "verify" an app (apparently lawyers are involved). There's been similar discussion on the fetchmail list.

That's obviously a dealbreaker for a free software community project. But the good news is that it only comes into play if you Publish your app. You can keep it in Testing mode indefinitely, as long as it's used only by yourself and a small number of users you personally know (add users in the Test Users list on the OAuth Consent screen). As far as I've been able to tell, Google has no plans to change that any time soon.

Expiring and Refreshing Tokens

Some people say that an app in Testing mode needs its tokens updated every seven days (requiring the user to click through the OAuth screens again). I haven't found anything detailing this, but I did find that after first getting Oauth running, it worked for the rest of that day, and then I got busy with other things and didn't have a need to post to a Google group that week, so it was ten days before I tried it again. At which point it died with a urllib.error.HTTPError: HTTP Error 400: Bad Request. But I was able to run getmail-gmail-xoauth-tokens --init /path/to/filename.json again and go through the authentication steps in the browser, and that got things working again.

So will I have to do that once a week? Or does it only come into play when I go a week without using the script at all? I'll have to wait at least another week to find out the answer to the question. In the meantime, I have my own fork for the getmail6 script that tries to handle re-initializing if it sees a 400 error: gmail-oauth-tokens.py. I haven't actually tested the re-initializing code yet (I'll have to wait another week before I can test it), so no guarantees.

Update, May 24 2022: Thomas Klausner has a good article on how to Use multiple gmail accounts via mutt and (offline) IMAP as an alternative to OAuth2.