Dave's solution for disconnected mail handling

2006 Update: Moved to Mail.app

I've mostly switched to using Apple's Mail.app these days, with Thunderbird at home on the windows box I keep around for no good reason. Mail.app handles disconnected operation better than anything I've found; I haven't really stress-tested thunderbird in this regard. From the times my Internet connection at home has gone down, it doesn't seem quite as offline-happy, but it's not unusable. Mail.app integrates with the Mac Keychain, as does ssh-agent, so I've got most of my single-sign-on desires solved. Now if only Firefox would join the keychain game...

Goal

What do I want in an email system?

Solution

Now that you know where I'm coming from, what's the solution I use to achieve this? Run Mutt as my mailreader. When on the local mail host, use uw-imap executed locally to access my mail. When remote, use isync over an SSH tunnel to synchronize my mail to a full mirror on my remote host, and read mail locally from there using mutt. Mairix provides searching locally and remotely.

The tools:

Step 1: Install UW-IMAP, carefully

If you already have uw-imap installed, note the location of your imapd binary (/usr/local/libexec/imapd on FreeBSD), and skip this step.

UW-IMAP is the de-facto reference IMAP server. It's had more remote security exploits than I'm comfortable with, and the code is almost impossible to read -- thus making it extremely difficult to audit, or trust.

Install uw-imap-2001a or later. Do not put an entry for it in inetd.conf or in any other way make it accessible on the network. Just note the location of the installed imapd.

You'll use the rsh mode of imapd to access it instead of the network.

Install mutt version 1.3 or later

I'm using 1.3.99 as we speak. Works nicely. Reportedly a couple of bugs with threading; this is development software, so be careful. I've moved to 1.4 on my laptop and it's also happy. You need 1.3 or later for the tunnel command, and for its improved IMAP handling, and for the maildir_trash command.

Mutt configuration: Spool machine

On the spool machine, you'll read mail using IMAP. I grabbed most of this information from the Mutt and Imap webpage. It's good, read it. Assuming you have your local mail folders in the Mail folder in your homedir, set these options in your .muttrc:

set tunnel="/usr/local/libexec/imapd"
set spoolfile=imap://localhost/INBOX
set folder=imap://localhost/Mail
set timeout=5

Set the tunnel to the location of your imapd server. Instead of connecting over the network, you'll just run it locally. Fast and happy. Set the spool and the default mail save folders to use the IMAP space instead of the local dirs. Setting timeout to 5 is, perhaps, excessive - it causes mutt to check every 5 seconds to see if the current mailbox has been updated. Pick whatever value makes you happy.

Why use IMAP locally? If you refile messages between mail folders directly using Mutt, you'll store invalid X-UID headers into the wrong mail folders. When you then check your mail with IMAP, the IMAP server will decide that you've goofed up your mailbox, and renumber it for you, and change the IMAP uidvalidity identifier. Isync will then barf. You could accomplish this by having Mutt strip out the X-UID header on a refile, but this way is much more clean.

On your remote hosts

Set up Mutt to use the local copy of the files, and set:
set maildir_trash
in your .muttrc. If you want to autosynchronize your folders when you type $, you can add something like these lines, that I grabbed from The Isync website:
folder-hook ~A bind index $ <sync-mailbox>

folder-hook +INBOX 'macro index $ "<sync-mailbox>!isync -e INBOX\n"'
I haven't bothered to come up with a better way of specifying this automatically for all mailboxes. If you do, drop me a note.

Install isync

Grab isync version .9.1 or later (it has built in support for SSH tunnels, and my earlier patch is no longer necessary). Install.

Configure your .isyncrc:

Tunnel "ssh -C -q your.mail.server /usr/local/libexec/imapd"
MailDir Mail
Mailbox INBOX
Box INBOX
Mailbox mailbox1
Box Mail/mailbox1
Mailbox mailbox2
Box Mail/mailbox2
...
where mailbox1 ... mailboxN are the various mail folders that you have in your Mail directory. Hint: autocreate this list with a little shell script:
#!/bin/csh
set IS="~/.isyncrc"
set LIMIT=500
echo 'Tunnel "ssh -C -q eep.lcs.mit.edu /usr/local/libexec/imapd"' >> $IS
echo "MailDir Mail" >> $IS
echo "Mailbox INBOX" >> $IS
echo "Box INBOX" >> $IS

foreach i (ls -t ~/Mail/ | grep -v \.gz$' | head -${LIMIT})
  echo "Mailbox $i" >> $IS
  echo "Box Mail/$i" >> $IS
end

Run isync -L -a to populate your mail spool. When you want to synchronize, run isync -C -a -e -d, or synchronize a particular folder (MUCH FASTER) with isync -C -e -d foldername.

Install Mairix

Once youve installed it, create a .mairixrc. I have mutt and procmail record all of my sent and received mail:
.muttrc contains:
set record=+sent_mail

.procmailrc contains:
:0 c
$MD/received_mail
and I then simply configure mairix to search these two files via its .mairixrc:
base=/home/dga/Mail
mfolder=search
database=/home/dga/.mairix_database
mbox=received_mail:sent_mail:received_mail.*
I find this tcsh alias useful for searching with mairix from the shell:
alias mm 'mairix \!* && mutt -f ~/Mail/search'
and a cron entry to reindex frequently. (Run this on both your laptop and your base machine):
17,37,57 * * * * /usr/local/bin/mairix > /dev/null 2>/dev/null

Some related sites

Other things that're wonderful for making life good in the email front:

Appendix 2: Thoughts about other mailreaders

A note: This is now getting a bit old (2 years), so take the remarks about other mail readers with a grain of salt -- they may have improved. (--dga, 2003). While on the journey for the perfect mail system, I tried out several mailreaders over the course of an exhausting night or two. Here are my brief impressions of some other clients. Disclaimer: This reflects my taste, not yours. YMMV.
elm
Just switch to mutt. I did extremely recently, and haven't looked back. The transition is trivial, and painless. Elm development is at a glacial pace, and the elm IMAP patches are questionable at best, with no disconnected support. Mutt has a vibrant user and developer community, and is constantly getting better.
VM
No offline IMAP (but I might be wrong; i was wrong on the previous IMAP support). VM is a beautiful mail reader, and I used it for years, but switched because I didn't think it supported offline IMAP. It does now, (and maybe did then), but I'm now pretty entrenched in Mutt.
Pine
Kick-ass IMAP support... unless you want disconnected operation. Would make a great way to read your mail remotely while the Internet is functional, though. Except that, well, it's pine, and I just kind of dislike pine. No real reason. If you're a pine fan, don't complain, just accept that I'm weird. Support for disconnected operation in pine has been "planned" for many years now. I'm not holding my breath.
Mulberry
Reportedly the best IMAP client out there. Coredumped on me, very frustrating to configure, no emacs/vi, huge GUI, doesn't seem to like my 301 mailfolders. May need more experimentation, but I'm not a fan thus far.
Ximian Evolution
Requires more extra installed packages than I have hard disk space for right now. Reportedly supports disconnected operation, though one of my colleagues reported that it coredumped on him after (correctly) resynchronizing after a disconnection. May try some day.
Netscape and Outlook both support IMAP. I haven't tried them out; I don't believe in having my email reader coredump as frequently as my browser, and I don't like viruses very much. Sorry. Apparently Outlook's IMAP support is quite good, though. One of my officemates uses Netscape for his mail, and is happy with its IMAP support. The 2004 update is that Thunderbird is getting quite good. If I weren't happy with what I have now, I'd probably look in to it.

Thanks...

The couple of hours I spent figuring all this gunk out is nothing compared to the work put in to developing the tools that I used here:


Last updated: Mon Nov 6 20:25:32 GMT 2006 [validate xhtml]

dga - at - pobox dot com.