mutt/mutt_oauth2.py.README

mutt_oauth.py README by Alexander Perlis, 2020-07-15
====================================================


Background on plain passwords, app passwords, OAuth2 bearer tokens
------------------------------------------------------------------

An auth stage occurs near the start of the IMAP/POP/SMTP protocol
conversation. Various SASL methods can be used (depends on what the
server offers, and what the client supports). The PLAIN method, also
known as "basic auth", involves simply sending the username and
password (this occurs over an encrypted connection), and used to be
common; but, for large cloud mail providers, basic auth is a security
hole. User passwords often have low entropy (humans generally choose
passwords that can be produced from human memory), thus are targets
for various types of exhaustive attacks.  Older attacks try different
passwords against one user, whereas newer spray attacks try one
password against different users. General mitigation efforts such as
rate-limiting, or detection and outright blocking efforts, lead to
degraded or outright denied services for legitimate users. The
security weakness is two-fold: the low entropy of the user password,
together with the alarming consequence that the password often unlocks
many disparate systems in a typical enterprise single-sign-on
environment. Also, humans type passwords or copy/paste them from
elsewhere on the screen, so they can also be grabbed via keyloggers or
screen capture (or a human bystander). Two ways to solve these
conundrums:

 - app passwords
 - bearer tokens

App passwords are simply high-entropy protocol-specific passwords, in
other words a long computer-generated random string, you use one for
your mail system, a different one for your payroll system, and so
on. With app passwords in use, brute-force attacks become useless. App
passwords require no modifications to client software, and only minor
changes on the server side.  One way to think about app passwords is
that they essentially impose on you the use of a password manager. Any
user can go to the trouble of using a password manager but most users
don't bother. App passwords put the password manager inside the server
and force you to use it.

Bearer tokens take the idea of app passwords to the next level. Much
like app passwords, they too are just long computer-generated random
strings, knowledge of which simply "lets you in". But unlike an app
password which the user must manually copy from a server password
screen and then paste into their client account config screen (a
process the user doesn't want to follow too often), bearer tokens get
swapped out approximately once an hour without user interaction. For
this to work, both clients and servers must be modified to speak a
separate out-of-band protocol (the "OAuth2" protocol) to swap out
tokens. More precisely, from start to finish, the process goes like
this: the client and server must once-and-for-all be informed about
each other (this is called "app registration" and might be done by the
client developer or left to each end user), then the client informs
the server that it wants to connect, then the user is informed to
independently use a web browser to visit a server destination to
approve this request (at this stage the server will require the user
to authenticate using say their password and perhaps additional
factors such as an SMS verification or crypto device), then the client
will have a long-term "refresh token" as well as an "access token"
good for about an hour. The access token can now be used with
IMAP/POP/SMTP to access the account. When it expires, the refresh
token is used to get a new access token and perhaps a new refresh
token.  After several months of such usage, even the refresh token may
expire and the human user will have to go back and re-authenticate
(password, SMS, crypto device, etc) for things to start anew.

Since app passwords and tokens are high-entropy and their compromise
should compromise only a particular system (rather than all systems in
a single-sign-on environment), they have similar security strength
when compared to stark weakness of traditional human passwords. But if
compared only to each other, tokens provide more security. App
passwords must be short enough for humans to easily copy/paste them,
might get written down or snooped during that process, and anyhow are
long-lived and thus could get compromised by other means. The main
drawback to tokens is that their support requires significant changes
to clients and servers, but once such support exists, they are
superior and easier to use.

Many cloud providers are eliminating support for human passwords. Some are
allowing app passwords in addition to tokens. Some allow only tokens.


OAuth2 token support in mutt
----------------------------

Mutt supports the two SASL methods OAUTHBEARER and XOAUTH2 for presenting an
OAuth2 access token near the start of the IMAP/POP/SMTP connection.

(Two different SASL methods exist for historical reasons. While OAuth2
was under development, the experimental offering by servers was called
XOAUTH2, later fleshed out into a standard named OAUTHBEARER, but not
all servers have been updated to offer OAUTHBEARER. Once the major
cloud providers all support OAUTHBEARER, clients like mutt might be
modified to no longer know about XOAUTH2.)

Mutt can present a token inside IMAP/POP/SMTP, but by design mutt itself
does not know how to have a separate conversation (outside of IMAP/POP/SMTP)
with the server to authorize the user and obtain refresh and access tokens.
Mutt just needs an access token, and has a hook for an external script to
somehow obtain one.

mutt_oauth2.py is an example of such an external script. It likely can be
adapted to work with OAuth2 on many different cloud mail providers, and has
been tested against:

 - Google consumer account (@gmail.com)
 - Google work/school account (G Suite tenant)
 - Microsoft consumer account (e.g., @live.com, @outlook.com, ...)
 - Microsoft work/school account (Azure tenant)
   (Note that Microsoft uses the marketing term "Modern Auth" in lieu of
   "OAuth2". In that terminology, mutt indeed supports "Modern Auth".)


Configure script's token file encryption
----------------------------------------

The script remembers tokens between invocations by keeping them in a
token file. This file is encrypted. Inside the script are two lines
  ENCRYPTION_PIPE
  DECRYPTION_PIPE
that must be edited to specify your choice of encryption system. A
popular choice is gpg. To use this:

 - Install gpg. For example, "sudo apt install gpg".
 - "gpg --gen-key". Answer the questions. Instead of your email
   address you could choose say "My mutt_oauth2 token store", then
   choose a passphrase. You will need to produce that same passphrase
   whenever mutt_oauth2 needs to unlock the token store.
 - Edit mutt_oauth2.py and put your GPG identity (your email address or
   whatever you picked above) in the ENCRYPTION_PIPE line.
 - For the gpg-agent to be able to ask you the unlock passphrase,
   the environment variable GPG_TTY must be set to the current tty.
   Typically you would put the following inside your .bashrc or equivalent:
     export GPG_TTY=$(tty)


Create an app registration
--------------------------

Before you can connect the script to an account, you need an
"app registration" for that service. Cloud entities (like Google and
Microsoft) and/or the tenant admins (the central technology admins at
your school or place of work) might be restrictive in who can create
app registrations, as well as who can subsequently use them. For
personal/consumer accounts, you can generally create your own
registration and then use it with a limited number of different personal
accounts. But for work/school accounts, the tenant admins might approve an
app registration that you created with a personal/consumer account, or
might want an official app registration from a developer (the creation of
which and blessing by the cloud provider might require payment and/or arduous
review), or might perhaps be willing to roll their own "in-house" registration.

What you ultimately need is the "client_id" (and "client_secret" if
one was set) for this registration. Those values must be edited into
the mutt_oauth2.py script.  If your work or school environment has a
knowledge base that provides the client_id, then someone already took
care of the app registration, and you can skip the step of creating
your own registration.


-- How to create a Google registration --

Go to console.developers.google.com, and create a new project. The name doesn't
matter and could be "mutt registration project".

 - Go to Library, choose Gmail API, and enable it
 - Hit left arrow icon to get back to console.developers.google.com
 - Choose OAuth Consent Screen
   - Choose Internal for an organizational G Suite
   - Choose External if that's your only choice
   - For Application Name, put for example "Mutt"
   - Under scopes, choose Add scope, scroll all the way down, enable the "https://mail.google.com/" scope
   - Fill out additional fields (application logo, etc) if you feel like it (will make the consent screen look nicer)
 - Back at console.developers.google.com, choose Credentials
 - At top, choose Create Credentials / OAuth2 client iD
   - Application type is "Desktop app"

Edit the client_id (and client_secret if there is one) into the
mutt_oauth2.py script.


-- How to create a Microsoft registration --

Go to portal.azure.com, log in with a Microsoft account (get a free
one at outlook.com), then search for "app registration", and add a
new registration. On the initial form that appears, put a name like
"Mutt", allow any type of account, and put "http://localhost/" as
the redirect URI, then more carefully go through each
screen:

Branding
 - Leave fields blank or put in reasonable values
 - For official registration, verify your choice of publisher domain
Authentication:
 - Platform "Mobile and desktop"
 - Redirect URI "http://localhost/"
 - Any kind of account
 - Enable public client (allow device code flow)
API permissions:
 - Microsoft Graph, Delegated, "offline_access"
 - Microsoft Graph, Delegated, "IMAP.AccessAsUser.All"
 - Microsoft Graph, Delegated, "POP.AccessAsUser.All"
 - Microsoft Graph, Delegated, "SMTP.Send"
 - Microsoft Graph, Delegated, "User.Read"
Overview:
 - Take note of the Application ID (a.k.a. Client ID), you'll need it shortly

End users who aren't able to get to the app registration screen within
portal.azure.com for their work/school account can temporarily use an
incognito browser window to create a free outlook.com account and use that
to create the app registration.

Edit the client_id (and client_secret if there is one) into the
mutt_oauth2.py script.


Running the script manually to authorize tokens
-----------------------------------------------

Run "mutt_oauth2.py --help" to learn script usage. To obtain the
initial set of tokens, run the script specifying a name for a
disposable token storage file, as well as "--authorize", for example
using this naming scheme:

 mutt_oauth2.py userid@myschool.edu.tokens --verbose --authorize

The script will ask questions and provide some instructions. For the
flow question:

 - "authcode": you paste a complicated URL into a browser, then
manually extract a "code" parameter from a subsequent URL in the
browser address bar and paste that back to the script.

- "localhostauthcode": you again paste the complicated URL into a browser
but that's it --- the code is automatically extracted from the response
relying on a localhost redirect and temporarily listening on a localhost
port. This flow can only be used if the web browser opening the redirect
URL sits on the same machine as where mutt is running, in other words can not
be used if you ssh to a remote machine and run mutt on that remote machine
while your web browser remains on your local machine.

 - "devicecode": you go to a simple URL and just enter a short code.

Your answer here determines the default flow, but on any invocation of
the script you can override the default with the optional "--authflow"
parameter. To change the default, delete your token file and start over.

To figure out which flow to use, I suggest trying all three.
Depending on the OAuth2 provider and how the app registration was
configured, some flows might not work, so simply trying them is the
best way to figure out what works and which one you prefer. Personally
I prefer the "localhostauthcode" flow when I can use it.


Once you attempt an actual authorization, you might get stuck because
the web browser step might indicate your institution admins must grant
approval.  Indeed engage them in a conversation about approving the
use of mutt to access mail. If that fails, an alternative is to
identify some other well-known IMAP/POP/SMTP client that they might
have already approved, or might be willing to approve, and first go
configure it for OAuth2 and see whether it will work to reach your
mail, and then you could dig into the source code for that client and
extract its client_id, client_secret, and redirect_uri and put those
into the mutt_oauth2.py script. This would be a temporary punt for
end-user experimentation, but not an approach for configuring systems
to be used by other people. Engaging your institution admins to create
a mutt registration is the better way to go.

Once you've succeeded authorizing mutt_oauth2.py to obtain tokens, try
one of the following to see whether IMAP/POP/SMTP are working:

 mutt_oauth2.py userid@myschool.edu.tokens --verbose --test
 mutt_oauth2.py userid@myschool.edu.tokens --verbose --debug --test

Without optional parameters, the script simply returns an access token
(possibly first conducting a behind-the-scenes URL retrieval using a
stored refresh token to obtain an updated access token). Calling the
script without optional parameters is how it will be used by
mutt. Your .muttrc would look something like:

 set imap_user="userid@myschool.edu"
 set folder="imap://outlook.office365.com/"
 set smtp_url="smtp://${imap_user}@smtp.office365.com:587/"
 set imap_authenticators="oauthbearer:xoauth2"
 set imap_oauth_refresh_command="/path/to/script/mutt_oauth2.py ${imap_user}.tokens"
 set smtp_authenticators=${imap_authenticators}
 set smtp_oauth_refresh_command=${imap_oauth_refresh_command}