When I type the beginning of a name in Mutt, pressing CTRL-T will give me a list of addresses whose real name matches my entry, and I just have to choose the one I want. As a result I seldom type more than a few letters to enter a complete address. This trick is called an external address query which is the mechanism by which Mutt supports connecting to external directory databases through a wrapper script.

The wrapper script I used is called lbdbq and it is part of lbdb. Lbdb provides an abstraction layer to a wide variety of address sources. The one I use most is the m_inmail database built by lbdb-fetchaddr. lbdb-fetchaddr reads a mail on stdin and extracts adresses from the mail header to append them to $HOME/.lbdb/m_inmail.list. Thanks to lbdb-fetchaddr, if we ever exchanged mail I can completion your name to you address.

There are two ways to feed the m_inmail database. The first one is to do it as you go using a MDA filter that pipes a copy of incoming messages to lbdb-fetchaddr. I copied the maildrop recipe from Mark Weinem’s .mailfilter example :

if ( $SIZE < 10000 )
cc "|lbdb-fetchaddr"

As you may have guessed the filter excludes messages larger than 10 KB because lbdb-fetchaddr is not supposed to work with “big messages”. But Mark Weinem’s .mailfilter example does not provide any further explanation and I found none myself. What I did find is that this rule does from time to time cause deferals after maildrop delivery failures for a reason I have yet to discover. Until I find that reason I will refrain from using it.

So I found another way to do it : feed lbdb-fetchaddr my 4 GB of mail dating back from 2001 (the moment when a disk crash taught me the value of good backups). This method has the added value of taking advantage of historical data, not just future traffic. On mutt-users, Jason Helfman asked “if anyone out there has any success in piping their inbox to lbdb-fetchaddr” and got no answer. Indeed, lbdb-fetchaddr does not work like that : each message must be fed individually. Of course I had to automate the process so that all my maildirs can be processed in one go, so I wrote a couple of small scripts :

If you do not have an existing lbdb database, you should run lbdb-fetchalladdresses-firsttime.sh first : it scours a maildir hierarchy to records all addresses from the headers in the lbdb m_inmail database. Once you have done that you can from time to time update your database with lbdb-fetchalladdresses-daily.sh who does the same thing as lbdb-fetchalladdresses-firsttime.sh but only for messages not older than a certain user-set age.

For performance reasons lbdb-fetchaddr appends new addresses to the database without removing duplicates – duplicates are only removed at query time. For 4 GB of mail, lbdb-fetchaddr produces a 6 MB m_inmail file which piping through ‘uniq‘ reduces to a mere 300 KB. Since the process is normally a one-off, I believe it is well worth the transient load. The update script only deals with a few additional entries, so the ‘uniq‘ load is negligible especially since this is likely to be a nightly cron job.

Don’t forget to install the lbdb package before and to add the following line to your muttrc :

'set query_command = "lbdbq '%s'

So there you go : happy CTRL-T completion !

I use Maildrop to dispatch incoming mail to the appropriate maildir. Such server side filtering is nice because my mail is always tidily waiting for me in the right folder whichever tool I use to read it. But it has a major drawback : each message is present in one and only one folder. For those messages about playing paintball with my grandmother I must choose between the “paintball” folder and the “family” folder – they cannot be in both unless I make a copy which would be a gross hack.

This problem is common to all hierarchical storage of physical items. But it has already been solved many times before with the use of tags. Whether you want to go hardcore with a full thesaurus or just use your little folksonomy the required technical foundation is the same : tags.

Of course, using tags is nothing new to Gmail users : Gmail allows users to categorize their e-mails with “labels.” Labels give users a flexible method of categorizing e-mails, since an e-mail may have any number of labels (in contrast to a system in which an e-mail may belong to only one folder). Users can display all e-mails having a particular label and can use labels as a search criterion. Gmail also allows users to set up filters which label incoming e-mail automatically. Users can simulate the functionality of folder-based filtering by applying labels and archiving mail as it arrives.

But most of the rest of the world has not caught up yet. So let’s do it : it seems to me that we have everything we need to build powerful functionnality on top of our fine toolset. As usual we will not settle for anything that makes us dependant on non-free products or on client side contraptions. So let’s review the available tools :

Server side filtering requires a competent mail delivery agent. A draft RFC describes the “IMAP flag extension” to the Sieve filtering language, but I have not found anything else (a reader later supplemented my lacking research by pointing out that “works on the Sieve language specification is still very much alive and well” – see the comments). Procmail can set keywords and Maildrop supports keywords too. The Maildrop package features a command-line utility (maildirkw) that allows other applications to manually set or clear custom keywords on messages and IMAP keywords may also be set with maildrop itself.

So we potentially have a bunch of maildirs containing appropriately tagged messages. And to serve them we luckily have a whole bunch of IMAP servers supporting IMAP keywords. It came to me as a surprise to see both IMAP servers and MDA providing apparently mature support for IMAP keywords. That is a good surprise. So while we are at it here are a few details about Courier-IMAP’s IMAP keywords implementation.

The less good surprise is the state of client support :

• A couple of weeks ago, Nancy McGough herself posted to evolution-list@gnome.org to enquire about IMAP keywords support in Ximian Evolution – I also failed to find any information about it and I do not take that as a good omen.
• A mutt user asked about IMAP keywords but no useful answer was produced – research yields no further information.
• Squirrelmail has a “Message Flags & Icons” plugin. It could be worse, but development does not seem very lively.
• Thunderbird supports IMAP keywords but only allows five of them. This is a known bug but it has been open for more than four year so I’m not holding my breath for it. The amount of comments made to that bug show that user expectations about IMAP keywords are slowly growing.

So IMAP keywords are as far as I am concerned not ready for production use yet but we are now just some client support away from that stage. I am eagerly looking forward to it ! I will certainly still use physical folders for the two top levels of my classification. But below that I am going to replace a lot of them with tags.