I work as a project manager for a very large ISP that Dilbert readers would sometimes find strangely familiar. But nothing prepared me for the shock and disbelief I experienced when some of my co-workers in the information systems division asked me why I kept sending them mail written in 10-point Courier New font whereas I was sending them plain text.

Following on their remark I soon found that many people found likewise that my messages are difficult to read because of that poor choice of font. Apparently, no one realized that plain text is rendered as whatever you want it to render as, including Fette Fraktur or even Zapf Dingbats if you fancy hieroglyphic form.

Sometimes I wonder if I am really working for an ISP. If you think that such company is an oasis of Internet culture, then if you joined one nowadays you would be sadly disappointed.

Anyway here is a tip for them, straight from the horse’s mouth :

1. From the main Microsoft Outlook window, on the “Tools” menu, click “Options” and then click the “Mail Format” tab.
2. Click “Fonts“.
3. Next to the “When composing and reading plain text” box, click “Choose Font“.
4. Select the fonts you want, and then click “OK“.
5. Enjoy my plain text messages in your favorite font and size.

Plain text grants the recipient freedom to render as he sees fit, including in Braille or as audio speech - plain text is that flexible.

I do not have a fetish for the spartan aesthetics of plain text in green monospace font on a black text console. What I appreciate is universal portability : I can read plain text on any device in any situation and process it with any tool including old or underpowered ones - and I actually do. That is the power of plain text.

Of course, as an interpreted markup language, HTML can also be rendered in a variety of ways and I could probably use it, but plain provides even more freedom. HTML being a standard I nevertheless welcome it in my mailbox even if I seldom send any HTML mail. A good maxim to live by when you are writing anything which has to interoperate with other systems is : “Be liberal in what you receive, and conservative in what you send”. It is as old as the Internet but it’s a great way to make things highly compatible and interoperable.

Writing this rant probably dates me as a unixian dinosaur… But it could be worse : just wait until someones gets me started about proper mail quoting !

Spam from Gmail accounts has appeared at least three years ago. Two years ago it caused Gmail servers to hit Spamcop’s database. But only now am I starting to notice it on my systems. My existing antispam arsenal may have protected me so far, but now I receive three or four illegitimate messages a day from that source, which is more than I tolerate.

Those messages have valid domain key signatures which lower their Spamassassin score enough to pass them as ham. If Google does not police its mail service better, then domain key signatures will become more than useless. A signature is only as good as its authority.

Meanwhile I will probably have to weight down the negative scores added through Mail::SpamAssassin::Plugin::DomainKeys.

Yes, we do hate stupid email disclaimers that much : they are a completely meaningless waste of electrons, they waste display real estate and they make proper quoting more awkward. And most of all they are unilateral clauses whose acceptation is most dubious.

Lerouge dug out an article by Russel Coker about striking back at worthless email disclaimers by configuring Postfix’s smtpd_banner in /etc/postfix/main.cf with the appropriate legalese of its own. Lerouge suggested that we do likewise and I jumped on the case and implemented it immediately…

Looking for something really outrageous and fit to point out how ridiculous those things are, I found inspiration at ReasonnablAgreement.com and modified it for my purposes…

So there you go :

13:52 jim@kivu ~% telnet localhost 25
Trying 127.0.0.1…
Connected to localhost.
Escape character is ‘^]’.
220 kivu.grabeuh.com ESMTP (M Sexchange) - READ CAREFULLY. By transmitting email to this server you agree personally and on behalf of your employer or organization, to release the recipients from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (hereafter BOGUS AGREEMENTS) that the recipients have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to the recipients ongoing rights and privileges. You further represent that you have the authority to release the recipients from any BOGUS AGREEMENTS on behalf of your employer or organization.

So in case you wondered, here is proof : the humongous disclaimer from hell does fit into smtpd_banner’s limits !

Most social tools come and go. The ones that stay share a common feature : openness. For example, email is open : chose any technology, chose any provider or be your own provider, chose any client, any platform - any way you do it you are still connected to the whole world.

If you have the slightest understanding of your own interest, then there is no way you should even consider using a closed platform as your primary mean of communication. Why would you willingly chose to put your most critical asset outside of your control ?

Many users will object that they gladly surrender control to closed social networking platforms because plain email does not meet their sophisticated communications needs and they are not willing to invest in developing the skills currently required to participated efficiently in the blogging sphere. That is a tragedy because the social graph is quickly becoming the glue of the connected services.

And even if the functionality was sufficient, we would still have a huge mindshare gap to bridge. XMPP provides nice basic instant messenging and presence management in an open environment, but most users still prefer proprietary centralized networks and happily trade freedom for webcam compatibility.

But similar battles have been fought and won in the past : Compuserve, AOL, The Source, Prodigy and their ilk have all dissolved in the Internet. The forces of openness now have a new crusade to embark upon : we must take the best use cases of the closed social networking world and port them in the open !

Open is everything - the rest is details. That is what drew me to the Internet fifteen years ago.

I upgraded the Sympa mailing list manager to 5.2.3-2 using the Debian package from the “Testing” repository. The database part of the upgrade procedure was a bit fussy so instead of solving its problems I simply backed up the tables, dropped them, ran the upgrade procedure and restored them. That workaround worked fine for making the Debian packaging system happy.

But Sympa itself was definitely not happy. On starting Sympa I got the following logs in /var/log/sympa.log :

Apr 25 17:02:39 kivu sympa[657]: Could not create table admin_table in
database sympa : Table ‘admin_table’ already exists
Apr 25 17:02:39 kivu sympa[657]: Could not create table user_table in
database sympa : Table ‘user_table’ already exists
Apr 25 17:02:39 kivu sympa[657]: Could not create table subscriber_table
in database sympa : Table ’subscriber_table’ already exists
Apr 25 17:02:39 kivu sympa[657]: Could not create table netidmap_table
in database sympa : Table ‘netidmap_table’ already exists
Apr 25 17:02:39 kivu sympa[657]: Unable to execute SQL query : You have
an error in your SQL syntax; check the manual that corresponds to your
MySQL server version for the right syntax to use near ‘.`admin_table’ at
line 1
Apr 25 17:02:39 kivu sympa[657]: Database sympa defined in sympa.conf
has not the right structure or is unreachable. If you don’t use any
database, comment db_xxx parameters in sympa.conf
Apr 25 17:02:39 kivu sympa[657]: Exiting.
Apr 25 17:02:39 kivu sympa[657]: Sympa not setup to use DBI

With no database access, Sympa was not operational. Double plus ungood !

The very strange thing is that the database is fine : the right tables with the right fields and the right records are all present. It even worked with the preceding version of Sympa. It looked like Sympa itself was unable to recognize that my database setup was correct, subsequently reported those errors and thereafter refused to run with it at all.

With a little rummaging inside the Sympa-users mailing list I quickly found a report of something looking suspiciously like my problem. It is probably a bug and Olivier Berger proposed a patch that looked to me like a workable solution : according to Olivier, a faulty regex was the cause of Sympa’s failure to recognize it’s own.

After making a backup copy of /usr/lib/sympa/bin/List.pm I promptly applied his patch :

17:04 root@kivu /usr/lib/sympa/bin# diff List.pm.dist List.pm
10750a10751
> $t =~ s/^([^.]+\.)?(.+)$/\2/;

I restarted Sympa and it worked fine ever after. Thank you Olivier !

The only problem is that while Sympa was down, people wondered why the messages did not go through and resent some of their messages. None of those messages were lost - they were just piling up in a queue. So when Sympa restarted many duplicates were sent.

But at least now it’s working. So for now I’m going to use dselect to freeze the Sympa Debian package at its current version so that it is kept back next time I upgrade my system.

Last year when I reviewed the IMAP keyword tagging scene, I said :

Thunderbird supports IMAP keywords but only allows five of them. This is a known bug but it has been open for more than four year so I’m not holding my breath for it. The amount of comments made to that bug show that user expectations about IMAP keywords are slowly growing.

Eager taggers rejoice : your wishes have been heard… Among the feature list of the just released Thunderbird 2 I read :

Thunderbird 2 allows you to tag messages with descriptors such as “To Do” or “Done” or even create your own tags that are specific to your needs. Tags can be combined with saved searches and mail views to make it easier to organize email.

This is exciting because last year I was pleasantly surprised to see both IMAP servers and MDA providing apparently mature support for IMAP keywords, but I concluded that we were just some client support away from being able to use them.

So the the last missing piece of the IMAP mail tagging puzzle may have been delivered and I shall soon play with it to see if it lives up to its potential.

This is certainly a classic bit of regex wizardry but since it took me a few minutes of searching and can be valuable in a variety of contexts, it might be valuable to you too…

grep -o ‘[[:alnum:]+\.\_\-]*@[[:alnum:]+\.\_\-]*’

I needed it for extracting the adresses returning a 550 from my Postfix logs. But then I found that Sympa, my mailing list management system, handles bounces automatically very well using a scoring algorithm that the list administrator can optionally override.

We shall call this process “serendipitous ignorance“…

While we are trying to make sense of regular expressions, those curious about them and wishing for an introduction geared toward audiences other than the beard and sandals systems administration crew may appreciate the examples provided in “Egrep for Linguists“.

And yes, I do indulge in sandals and facial pilosity in the hope of mastering regexes one day…

Blue Frog automated the complaint process for each user as they receive spam. It worked so well that spammers considered it a very serious threat to their livelyhood. Blue Security CEO Eran Reshef quoted a spammer as writing “Blue found the right solution to stop spam, and I can’t let this continue“. Under heavy attacks from the spammers, Blue Security called quits in May 2006.

Following the demise of Blue Frog, the Okopipi project aimed to become a distributed replacement of Blue Security’s anti-spam software, based on a P2P network. For now there is only an Okopipi FAQ and a seminal functional overview of the Okopipi system. The official Okopipi forums are quite dead and it is not the only bad sign for the Okopipi project. But Journeyman recently loudly stated that the Okopipi project is still moving forward. So maybe you can still either keep hoping or offer your help…

Whereas Okopipi has a slight rank of Second System Effect, Knujon looks like a bold attempt to take spam control from the technical to the social dimension. Filtering works well but it is only treating the symptom of the spam problem. Knujon vows to bring businesses, governments, law enforcement, security professionals and other users together in collaboration. Filtering is a selfish associal device whereas systemic salvation lies in a multidimensional cooperative approach. As Knujon puts it :

“Organizations and Personal Email users are blocking/filtering millions of junk emails every day. This is to the advantage of spammers as it allows them to target the most vulnerable users who do not have filtering software or technical savvy. Besides helping the junk mailers and identity thieves find their target audience, we are restricting our own use of email.
[..]
Blocking and filtering are not proper solutions for law enforcement or computer security professionals since it they only serve to hide the problem and force the activity to an underground network. Ordinary users must sift through hundreds of quarantined junk emails everyday to search for legitimate messages”.

So help save those clueless “ordinary users” who do not enjoy all the spam filtering goodness ! You can do your bit by simply forwarding your spam mail to yourjunk@knujon.com. There is also a Knujon Thunderbird plugin, or you can automate that process using my script that feeds the content of any maildir to various spam reporting services. Recycle spam, save the planet !

I thought I had spam pretty much under control, with only about one getting though every few days. And then came image spam. No suspicious words, just a load of bayes poison and an image to carry the actual message. Half of my antispam arsenal was suddenly rendered useless. I was back to suffering one or two spam messages every day.

“The level of image spam has increased dramatically this year,” says Carole Theriault, a senior consultant at Sophos cited by New Scientist. According to New Scientist, Sophos estimates that, at the beginning of the year, image spam accounted for only 18% of unsolicited mail but that this has since risen to 40%.

Less impressive but much more useful than statistical FUD from a biased source, were the few articles about using optical character recognition to fight image spam, from Debian Administration and Linux Weekly News among others.

But with a production server on my hands and precious little time to maintain it I wished to stick to packages distributed by Debian so I waited a little longer for packaging while my users suffered. To my great relief, Christmas came a few days earlier this year - FuzzyOcr hit Debian unstable yesterday ! Mail server administrators rejoice ! Somebody must have been even more pissed off than me about image spam and decided to make the Debian packaging work…

So here is the FuzzyOcr Debian package blurb, straight from the horse’s mouth :

This Spamassassin plugin checks for specific keywords in image/gif, image/jpeg or image/png attachments, using gocr (an optical character recognition program). This plugin can be used to detect spam that puts all the real spam content in an attached image, while the mail itself is only random text and random html, without any URL’s or identifiable information. Additionally to the normal OcrPlugin, it can do approximate matches on words, so errors in recognition or attempts to obfuscate the text inside the image will not cause the detection to fail.

But a debug log is worth a thousand words, so here is a choice output :

[2006-12-19 12:03:39] Debug mode: Starting FuzzyOcr…
[2006-12-19 12:03:39] Debug mode: Attempting to load personal wordlist…
[2006-12-19 12:03:39] Debug mode: No personal wordlist found, skipping…
[2006-12-19 12:03:39] Debug mode: Analyzing file with content-type “image/gif”
[2006-12-19 12:03:39] Debug mode: Image is single non-interlaced…
[2006-12-19 12:03:39] Debug mode: Recognized file type: 1
[2006-12-19 12:03:39] Debug mode: Image hashing disabled in configuration, skipping…
[2006-12-19 12:03:40] Debug mode: Found word “price” in line
“lmatpriceguaranteeftdenrey”
with fuzz of 0 scanned with scanset /usr/bin/gocr -i -
[2006-12-19 12:03:40] Debug mode: Found word “price” in line
“lmatpriceguaranteeftdenrey”
with fuzz of 0 scanned with scanset /usr/bin/gocr -l 180 -d 2 -i -
[2006-12-19 12:03:40] Debug mode: Found word “viagra” in line
“viaraloomgaooomaoo”
with fuzz of 0.166666666666667 scanned with scanset /usr/bin/gocr -i -
[2006-12-19 12:03:40] Debug mode: Found word “viagra” in line
“viaqrastloomaaomaa”
with fuzz of 0.166666666666667 scanned with scanset /usr/bin/gocr -i -
[2006-12-19 12:03:40] Debug mode: Found word “viagra” in line
“viaqrastloomaarialisnomaa”
with fuzz of 0.166666666666667 scanned with scanset /usr/bin/gocr -l 180 -d 2 -i -
[2006-12-19 12:03:40] Debug mode: Found word “cialis” in line
“viaqrastloomaarialisnomaa”
with fuzz of 0.166666666666667 scanned with scanset /usr/bin/gocr -l 180 -d 2 -i -
[2006-12-19 12:03:40] Debug mode: Found word “valium” in line
“valiumlomgaooantivanmgalo”
with fuzz of 0 scanned with scanset /usr/bin/gocr -l 180 -d 2 -i -
[2006-12-19 12:03:40] Debug mode: Found word “legal” in line
“vlaraloomgaoorlalisomaoo”
with fuzz of 0.2 scanned with scanset /usr/bin/gocr -l 180 -d 2 -i -
[2006-12-19 12:03:40] Debug mode: Starting FuzzyOcr…
[2006-12-19 12:03:40] Debug mode: Attempting to load personal wordlist…
[2006-12-19 12:03:40] Debug mode: No personal wordlist found, skipping…
[2006-12-19 12:03:40] Debug mode: FuzzyOcr ending successfully…
[2006-12-19 12:03:40] Debug mode: Message is spam (score 10)…
[2006-12-19 12:03:40] Debug mode: Words found:
“price” in 1 lines
“viagra” in 2 lines
“cialis” in 1 lines
“valium” in 1 lines
“legal” in 1 lines
(6 word occurrences found)
[2006-12-19 12:03:40] Debug mode: FuzzyOcr ending successfully…

Sweet isn’t it ? And that antispam OCR goodness is just an ‘apt-get install fuzzyocr’ away !

The only parameters I changed in /etc/FuzzyOcr.cf are the following :

focr_verbose 2.0
focr_logfile /var/log/fuzzyocr.log
focr_timeout 16

The two first are self-explanatory : I just want to know what is going on. The original timeout was 12 seconds and I found that it was often too short for my puny server - apparently 16 seconds are more than enough. I restarted Amavisd-new who handles calling SpamAssassin and I was done !

I was afraid that FuzzyOcr would load my host too much but I found my fears unfounded : FuzzyOcr only scan messages which where not recognized yet as ham or spam by other SpamAssassin rules or plugins. So the additional load was not noticeable among all the heavy antispam and antiviral machinery that already operated. FuzzyOcr is full of nice surprises !

As a conclusion I must say that, on our mail server, FuzzyOcr is a complete success. I recommend that you install it as soon as possible !

Versamail, the default mail application on my Treo 650 is good for casually checking a handful of new messages in the inbox but it comes short for about anything else. For searching accross my 5 GB of mail in a hundred folders and accessing them accross a wet string, Mutt is by far the best remote solution.

Although twenty-four years younger than Vi, Mutt was apparently also designed to thrive in an environment made of appaling latency, tiny bitrates and screens with an even lower definition than the Treo’s screen. GPRS really feels like my first 2400 baud modem, so to me a command line with Lbdb, Mairix, Mutt and Vim definitely seems to be the perfect match for GPRS on the Treo.

A while ago I published lbdb-fetchalladdresses-firsttime.sh and lbdb-fetchalladdresses-daily.sh, two scripts that scour a maildir hierarchy to records all addresses from the headers in the lbdb m_inmail database in order to feed whole maildirs to LBDB for Mutt’s address completion.

A few days ago, for some unknown reasons this script ceased to work correctly. I traced the problem to an incorrect argument to ‘uniq‘. While I was at it I decided to let ‘sort‘ handle the job of ‘uniq‘ and in one fell swoop enhanced the sort operation with some better criterias (-u -d -f -i -k 1,1). So here is version 0.3 for you :

• lbdb-fetchalladdresses-firsttime.sh
• lbdb-fetchalladdresses-daily.sh

As I mentioned earlier, if you do not have an existing lbdb database, you should run lbdb-fetchalladdresses-firsttime.sh first. Once you have done that you can from time to time update your database with lbdb-fetchalladdresses-daily.sh who does the same thing as lbdb-fetchalladdresses-firsttime.sh but only for messages not older than a certain user-set age.

A month ago I published lbdb-fetchalladdresses-firsttime.sh and lbdb-fetchalladdresses-daily.sh, two scripts that scour a maildir hierarchy to records all addresses from the headers in the lbdb m_inmail database in order to feed whole maildirs to LBDB for Mutt’s address completion.

A month of running these scripts has revealed a few problems wich are now corrected with a speed optimization thrown in for good measure :

• lbdb-fetchalladdresses-firsttime.sh
• lbdb-fetchalladdresses-daily.sh

As I mentioned earlier, if you do not have an existing lbdb database, you should run lbdb-fetchalladdresses-firsttime.sh first. Once you have done that you can from time to time update your database with lbdb-fetchalladdresses-daily.sh who does the same thing as lbdb-fetchalladdresses-firsttime.sh but only for messages not older than a certain user-set age.

As many have said before, Jabber is to instant messaging what SMTP is to email:

Today most people understand instant messaging almost entirely in terms provided by AOL, Microsoft and Yahoo. As with on-line services and operating systems, closed IM systems became familiar to the general public long before the open ones. In terms of architecture and control, AOL’s, Microsoft’s and Yahoo’s instant messaging systems are as closed and non-interoperable as Prodigy, Compuserve and AOL were back in the [early] nineties.

In other words, AOL, Microsoft and Yahoo all want to keep private something that should be no less public than mail and web services. All of those companies’ IM systems use the Internet and offer free clients, but they offer no internet service anybody can build on.

This quote dates back from 2001… After all these years I at last consider the Jabber ecosystem to reach both reasonnable usability and critical mass. I believe we can thank Google for tipping the scale. So what’s next ?

I have said it privately since 2002 : the buddy list will become the center of user interaction with the mobile “phone”. Presence management holds every synchronous media together. Through presence management, messaging has moved from the periphery of the desktop toward its core. It changes the environment in which other services interoperate. For example it makes the answering machine obsolete : no one is going to knowingly initiate a call when there is on one on the other end. And the buddy list is where convergence happens : any communication can be initiated from there, not just what the operator deems reasonnable to provide.

As usual the operators are busy fighting rear-guard action, but they cannot keep the customer from craving the ideal of the Stupid Network. GPRS and UMTS enable innovation at the edge and that is the end of the walled gardens. Of course capital expenditure and spectrum scarcity still provide them with a fairly entrenched position, but we are now firmly in Internet land and there is no going back.

So as we all have been thinking all along, the open protocol takes over the closed ones. It takes time, but it is inexorable. The mobile networks are next and they know it… But that does not mean they can avoid it.

Mobile network operators will not like it but instant messaging is going to be priced at bulk data prices, not anywhere near the EUR 1000 a MB that they can get away with for SMS. Earlier attempts to sell services instead of selling raw data have met mixed responses - it will only get worse and I do not believe this to be only my wishful thinking.

When I type the beginning of a name in Mutt, pressing CTRL-T will give me a list of addresses whose real name matches my entry, and I just have to choose the one I want. As a result I seldom type more than a few letters to enter a complete address. This trick is called an external address query which is the mechanism by which Mutt supports connecting to external directory databases through a wrapper script.

The wrapper script I used is called lbdbq and it is part of lbdb. Lbdb provides an abstraction layer to a wide variety of address sources. The one I use most is the m_inmail database built by lbdb-fetchaddr. lbdb-fetchaddr reads a mail on stdin and extracts adresses from the mail header to append them to $HOME/.lbdb/m_inmail.list. Thanks to lbdb-fetchaddr, if we ever exchanged mail I can completion your name to you address.

There are two ways to feed the m_inmail database. The first one is to do it as you go using a MDA filter that pipes a copy of incoming messages to lbdb-fetchaddr. I copied the maildrop recipe from Mark Weinem’s .mailfilter example :

if ( $SIZE < 10000 )
cc "|lbdb-fetchaddr"

As you may have guessed the filter excludes messages larger than 10 KB because lbdb-fetchaddr is not supposed to work with “big messages”. But Mark Weinem’s .mailfilter example does not provide any further explanation and I found none myself. What I did find is that this rule does from time to time cause deferals after maildrop delivery failures for a reason I have yet to discover. Until I find that reason I will refrain from using it.

So I found another way to do it : feed lbdb-fetchaddr my 4 GB of mail dating back from 2001 (the moment when a disk crash taught me the value of good backups). This method has the added value of taking advantage of historical data, not just future traffic. On mutt-users, Jason Helfman asked “if anyone out there has any success in piping their inbox to lbdb-fetchaddr” and got no answer. Indeed, lbdb-fetchaddr does not work like that : each message must be fed individually. Of course I had to automate the process so that all my maildirs can be processed in one go, so I wrote a couple of small scripts :

If you do not have an existing lbdb database, you should run lbdb-fetchalladdresses-firsttime.sh first : it scours a maildir hierarchy to records all addresses from the headers in the lbdb m_inmail database. Once you have done that you can from time to time update your database with lbdb-fetchalladdresses-daily.sh who does the same thing as lbdb-fetchalladdresses-firsttime.sh but only for messages not older than a certain user-set age.

For performance reasons lbdb-fetchaddr appends new addresses to the database without removing duplicates - duplicates are only removed at query time. For 4 GB of mail, lbdb-fetchaddr produces a 6 MB m_inmail file which piping through ‘uniq‘ reduces to a mere 300 KB. Since the process is normally a one-off, I believe it is well worth the transient load. The update script only deals with a few additional entries, so the ‘uniq‘ load is negligible especially since this is likely to be a nightly cron job.

Don’t forget to install the lbdb package before and to add the following line to your muttrc :

'set query_command = "lbdbq '%s'

So there you go : happy CTRL-T completion !

I use Maildrop to dispatch incoming mail to the appropriate maildir. Such server side filtering is nice because my mail is always tidily waiting for me in the right folder whichever tool I use to read it. But it has a major drawback : each message is present in one and only one folder. For those messages about playing paintball with my grandmother I must choose between the “paintball” folder and the “family” folder - they cannot be in both unless I make a copy which would be a gross hack.

This problem is common to all hierarchical storage of physical items. But it has already been solved many times before with the use of tags. Whether you want to go hardcore with a full thesaurus or just use your little folksonomy the required technical foundation is the same : tags.

Of course, using tags is nothing new to Gmail users : Gmail allows users to categorize their e-mails with “labels.” Labels give users a flexible method of categorizing e-mails, since an e-mail may have any number of labels (in contrast to a system in which an e-mail may belong to only one folder). Users can display all e-mails having a particular label and can use labels as a search criterion. Gmail also allows users to set up filters which label incoming e-mail automatically. Users can simulate the functionality of folder-based filtering by applying labels and archiving mail as it arrives.

But most of the rest of the world has not caught up yet. So let’s do it : it seems to me that we have everything we need to build powerful functionnality on top of our fine toolset. As usual we will not settle for anything that makes us dependant on non-free products or on client side contraptions. So let’s review the available tools :

Server side filtering requires a competent mail delivery agent. A draft RFC describes the “IMAP flag extension” to the Sieve filtering language, but I have not found anything else (a reader later supplemented my lacking research by pointing out that “works on the Sieve language specification is still very much alive and well” - see the comments). Procmail can set keywords and Maildrop supports keywords too. The Maildrop package features a command-line utility (maildirkw) that allows other applications to manually set or clear custom keywords on messages and IMAP keywords may also be set with maildrop itself.

So we potentially have a bunch of maildirs containing appropriately tagged messages. And to serve them we luckily have a whole bunch of IMAP servers supporting IMAP keywords. It came to me as a surprise to see both IMAP servers and MDA providing apparently mature support for IMAP keywords. That is a good surprise. So while we are at it here are a few details about Courier-IMAP’s IMAP keywords implementation.

The less good surprise is the state of client support :

• A couple of weeks ago, Nancy McGough herself posted to evolution-list@gnome.org to enquire about IMAP keywords support in Ximian Evolution - I also failed to find any information about it and I do not take that as a good omen.
• A mutt user asked about IMAP keywords but no useful answer was produced - research yields no further information.
• Squirrelmail has a “Message Flags & Icons” plugin. It could be worse, but development does not seem very lively.
• Thunderbird supports IMAP keywords but only allows five of them. This is a known bug but it has been open for more than four year so I’m not holding my breath for it. The amount of comments made to that bug show that user expectations about IMAP keywords are slowly growing.

So IMAP keywords are as far as I am concerned not ready for production use yet but we are now just some client support away from that stage. I am eagerly looking forward to it ! I will certainly still use physical folders for the two top levels of my classification. But below that I am going to replace a lot of them with tags.