Serendipitous altruism

One of the problems with my Jabber presence indicator on a web page was that the page’s presence was being sent to the polled account instead of being hidden. I fixed that and the updated code is available. The fix was trivial :

29c29
< $JABBER->SendPresence();
—
> $JABBER->SendPresence(“invisible”);

Although widely implemented (even Edgar does it that way), the “invisible” presence type is not XMPP compliant. JEP 126 explains how to provide selective visibility in a XMPP compliant manner. Of course this is less trivial. But since JEP 126 provides the appropriate XML snippets, I could probably do it the right way using SendPacket($xml) to send the raw XML. Maybe next time… Meanwhile I reported the standard compliance issue on the Edgar bug tracker.

Ever found yourself painstakingly trying many combinations of dates to find the cheapest flight ? TechCrunch reports on FlySpy, a tool that will make that process painless :

The way it works is that I give it a departure city and a destination city and optionally a departure date and length of stay. The search result, which returns very quickly, will present me with a graph of flight prices over the next 30 days so that I can quickly look at which days are the cheapest to fly. To book a flight I just click on the point in the graph. Simple.

Jason Kottke envisions taking the idea even further by using historical data to predict the optimal date of purchase. He also suggest applying it to all industries where yield management is relevant.

Is transparent pricing coming to the air travel market ? Probably not even in your wildest dreams as Keith Devlin explained in 2002 :

Faced with all this confusion, with computers constantly monitoring sales and adjusting fares as often as ten times a day, the only real option for the fare conscious air traveler is to use a Web service to try to locate the best deal. [..] But just how well do those search engines do ? Not very, is the answer. And with good reason. Airline pricing has grown so complex that it is now practically impossible to design an algorithm that will find the cheapest fare. In mathematical terms, the (idealized) problem of finding [..] the lowest fare is NP hard [..] This is the perhaps surprising result obtained recently by mathematician Carl de Marcken.

Although depressing, this piece of research highlights an interesting aspect of FlySpy : it does not try to find the solution to the problem. Instead it follows the patch of scientific visualisation : when confronted with overwhelming amounts of data, the best way to understand it is to draw a nice picture.

Total Annihilation introduced unparalleled innovation and ambition for its time. I consider it the greatest real-time strategy game I ever played – and I played many. Its legacy still endures ten years later through a robust community of gamers and modders : ever since the demise of the company that created it, Total Annihilation has been enhanced, extended and corrected by its users. My favorite modification is Uberhack : in my opinion Uberhack is the most robust, most complete and best balanced incarnation of the game.

Total Annihilation have kept alive the dream of a game deep enough for hardcore gamers while retaining a strong appeal to the casual crowd. It seems that this baby is in the process of being developped by Gas Powered Games and expected for 2007. It is named Supreme Commander.

Gamespy has a few articles, interviews and screenshots about SupCom. It is a good place to get yourself acquainted to the beast. The Supreme Commander official forums have a nice sticky thread gathering the known public facts. Once you have read all that you can come and join us at the Supreme Commander official forums : 20k posts in six months for a strategy game that will not be out for at least eighteen months is very impressive. If the product holds up to its promises it will surely have a massively commited community of gamers and modders behind it.

My bet is that it is going to be even better than all that it is hyped to be. See you in 18 months !

I now rely on Jabber to publish my presence. Not everyone has a Jabber client at hand but Web access is about ubiquitous so it makes sense to me to make my presence available on a Web page. To me this is a personnal itch and an excuse to practice some PHP and learn a little about the exciting world of XMPP.

The source code for my presence indicator is now available for whoever wants it. It is made possible by class.jabber.php, a Jabber library for PHP. I customized one of the examples provided with class.jabber.php. It has many rough edges such as the user’s name sprinkled across the markup code instead of being a proper parameter, or the page’s presence being sent to the polled account instead of being hidden. And it is quite slow because it queries at load time. But I like it better than the example packaged with class.jabber.php.

It basically does the job and I am therefore quite sure that there will be interested users. So this is the “early” part of “Release Early, Release Often”. Take that hack, customize it, tweak it and generally improve it… And most of all please report back somewhere so that everyone can benefit. Here would be a good start.

As many have said before, Jabber is to instant messaging what SMTP is to email:

Today most people understand instant messaging almost entirely in terms provided by AOL, Microsoft and Yahoo. As with on-line services and operating systems, closed IM systems became familiar to the general public long before the open ones. In terms of architecture and control, AOL’s, Microsoft’s and Yahoo’s instant messaging systems are as closed and non-interoperable as Prodigy, Compuserve and AOL were back in the [early] nineties.

In other words, AOL, Microsoft and Yahoo all want to keep private something that should be no less public than mail and web services. All of those companies’ IM systems use the Internet and offer free clients, but they offer no internet service anybody can build on.

This quote dates back from 2001… After all these years I at last consider the Jabber ecosystem to reach both reasonnable usability and critical mass. I believe we can thank Google for tipping the scale. So what’s next ?

I have said it privately since 2002 : the buddy list will become the center of user interaction with the mobile “phone”. Presence management holds every synchronous media together. Through presence management, messaging has moved from the periphery of the desktop toward its core. It changes the environment in which other services interoperate. For example it makes the answering machine obsolete : no one is going to knowingly initiate a call when there is on one on the other end. And the buddy list is where convergence happens : any communication can be initiated from there, not just what the operator deems reasonnable to provide.

As usual the operators are busy fighting rear-guard action, but they cannot keep the customer from craving the ideal of the Stupid Network. GPRS and UMTS enable innovation at the edge and that is the end of the walled gardens. Of course capital expenditure and spectrum scarcity still provide them with a fairly entrenched position, but we are now firmly in Internet land and there is no going back.

So as we all have been thinking all along, the open protocol takes over the closed ones. It takes time, but it is inexorable. The mobile networks are next and they know it… But that does not mean they can avoid it.

Mobile network operators will not like it but instant messaging is going to be priced at bulk data prices, not anywhere near the EUR 1000 a MB that they can get away with for SMS. Earlier attempts to sell services instead of selling raw data have met mixed responses – it will only get worse and I do not believe this to be only my wishful thinking.

It has been a few days since I have begin using Chatopus as a Jabber client on my Treo 650. Overall I am very pleased. Chatopus is very simple and quite polished. It works as advertised with no effort and no undue feature creep.

The only major functionnality that I miss in Chatopus is background operation during an SSL secured connexion. It works fine with a cleartext Jabber connexion, but not with SSL. According to the author, Tony Yat-Tung Cheung it “is an issue with Palm OS’ built-in SSL library. Unfortunately, PalmSource is no longer updating the Palm OS Garnet“.

I also found a few minor annoyances. The first one is that the roster can only display at any one time the contacts from one categories or all catefories together, but not a choice of selected categories. Although that Palm OS UI convention is normally simple and practical this is not the first time I find it annoying… This illustrate the fact that UI conventions are guidelines that sometime can be overriden by specific needs. But by sticking to the convention Chatopus does a good job of remaining simple and easy to use.

Chatopus also lacks a custom status in addition to the standard standard Presence Type (Available, Away etc.). For exemple, when the Exodus Jabber client sets the “Extended Away” Presence Type it transmits the standard “xa” flag and an additional customizable and more informative status details such as “Extended away as a result of idle”. It would be nice if Chatopus transmitted this too in addition to the standard Presence Type.

Last minor gripe : if for some reason the GPRS link goes down Chatopus does not reconnect and restart it. Maybe this should be a user-configurable option.

Overall Chatopus makes the mobile use of Jabber a breeze. On my Treo 650 it now has its own button so that my roster is only a keypress away… That tells something about how useful I find it !

When I type the beginning of a name in Mutt, pressing CTRL-T will give me a list of addresses whose real name matches my entry, and I just have to choose the one I want. As a result I seldom type more than a few letters to enter a complete address. This trick is called an external address query which is the mechanism by which Mutt supports connecting to external directory databases through a wrapper script.

The wrapper script I used is called lbdbq and it is part of lbdb. Lbdb provides an abstraction layer to a wide variety of address sources. The one I use most is the m_inmail database built by lbdb-fetchaddr. lbdb-fetchaddr reads a mail on stdin and extracts adresses from the mail header to append them to $HOME/.lbdb/m_inmail.list. Thanks to lbdb-fetchaddr, if we ever exchanged mail I can completion your name to you address.

There are two ways to feed the m_inmail database. The first one is to do it as you go using a MDA filter that pipes a copy of incoming messages to lbdb-fetchaddr. I copied the maildrop recipe from Mark Weinem’s .mailfilter example :

if ( $SIZE < 10000 )
cc "|lbdb-fetchaddr"

As you may have guessed the filter excludes messages larger than 10 KB because lbdb-fetchaddr is not supposed to work with “big messages”. But Mark Weinem’s .mailfilter example does not provide any further explanation and I found none myself. What I did find is that this rule does from time to time cause deferals after maildrop delivery failures for a reason I have yet to discover. Until I find that reason I will refrain from using it.

So I found another way to do it : feed lbdb-fetchaddr my 4 GB of mail dating back from 2001 (the moment when a disk crash taught me the value of good backups). This method has the added value of taking advantage of historical data, not just future traffic. On mutt-users, Jason Helfman asked “if anyone out there has any success in piping their inbox to lbdb-fetchaddr” and got no answer. Indeed, lbdb-fetchaddr does not work like that : each message must be fed individually. Of course I had to automate the process so that all my maildirs can be processed in one go, so I wrote a couple of small scripts :

If you do not have an existing lbdb database, you should run lbdb-fetchalladdresses-firsttime.sh first : it scours a maildir hierarchy to records all addresses from the headers in the lbdb m_inmail database. Once you have done that you can from time to time update your database with lbdb-fetchalladdresses-daily.sh who does the same thing as lbdb-fetchalladdresses-firsttime.sh but only for messages not older than a certain user-set age.

For performance reasons lbdb-fetchaddr appends new addresses to the database without removing duplicates – duplicates are only removed at query time. For 4 GB of mail, lbdb-fetchaddr produces a 6 MB m_inmail file which piping through ‘uniq‘ reduces to a mere 300 KB. Since the process is normally a one-off, I believe it is well worth the transient load. The update script only deals with a few additional entries, so the ‘uniq‘ load is negligible especially since this is likely to be a nightly cron job.

Don’t forget to install the lbdb package before and to add the following line to your muttrc :

'set query_command = "lbdbq '%s'

So there you go : happy CTRL-T completion !

I use Maildrop to dispatch incoming mail to the appropriate maildir. Such server side filtering is nice because my mail is always tidily waiting for me in the right folder whichever tool I use to read it. But it has a major drawback : each message is present in one and only one folder. For those messages about playing paintball with my grandmother I must choose between the “paintball” folder and the “family” folder – they cannot be in both unless I make a copy which would be a gross hack.

This problem is common to all hierarchical storage of physical items. But it has already been solved many times before with the use of tags. Whether you want to go hardcore with a full thesaurus or just use your little folksonomy the required technical foundation is the same : tags.

Of course, using tags is nothing new to Gmail users : Gmail allows users to categorize their e-mails with “labels.” Labels give users a flexible method of categorizing e-mails, since an e-mail may have any number of labels (in contrast to a system in which an e-mail may belong to only one folder). Users can display all e-mails having a particular label and can use labels as a search criterion. Gmail also allows users to set up filters which label incoming e-mail automatically. Users can simulate the functionality of folder-based filtering by applying labels and archiving mail as it arrives.

But most of the rest of the world has not caught up yet. So let’s do it : it seems to me that we have everything we need to build powerful functionnality on top of our fine toolset. As usual we will not settle for anything that makes us dependant on non-free products or on client side contraptions. So let’s review the available tools :

Server side filtering requires a competent mail delivery agent. A draft RFC describes the “IMAP flag extension” to the Sieve filtering language, but I have not found anything else (a reader later supplemented my lacking research by pointing out that “works on the Sieve language specification is still very much alive and well” – see the comments). Procmail can set keywords and Maildrop supports keywords too. The Maildrop package features a command-line utility (maildirkw) that allows other applications to manually set or clear custom keywords on messages and IMAP keywords may also be set with maildrop itself.

So we potentially have a bunch of maildirs containing appropriately tagged messages. And to serve them we luckily have a whole bunch of IMAP servers supporting IMAP keywords. It came to me as a surprise to see both IMAP servers and MDA providing apparently mature support for IMAP keywords. That is a good surprise. So while we are at it here are a few details about Courier-IMAP’s IMAP keywords implementation.

The less good surprise is the state of client support :

• A couple of weeks ago, Nancy McGough herself posted to evolution-list@gnome.org to enquire about IMAP keywords support in Ximian Evolution – I also failed to find any information about it and I do not take that as a good omen.
• A mutt user asked about IMAP keywords but no useful answer was produced – research yields no further information.
• Squirrelmail has a “Message Flags & Icons” plugin. It could be worse, but development does not seem very lively.
• Thunderbird supports IMAP keywords but only allows five of them. This is a known bug but it has been open for more than four year so I’m not holding my breath for it. The amount of comments made to that bug show that user expectations about IMAP keywords are slowly growing.

So IMAP keywords are as far as I am concerned not ready for production use yet but we are now just some client support away from that stage. I am eagerly looking forward to it ! I will certainly still use physical folders for the two top levels of my classification. But below that I am going to replace a lot of them with tags.

If you install the new 2.3.3 version of amavisd-new via the Debian package, you are in for a few (possibly unpleasant) surprises. Among other changes, /etc/amavis/amavisd.conf no longer works.

The new configuration files in /etc/amavis/conf.d there are:

01-debian
05-node_id
15-av_scanners
15-content_filter_mode
20-debian_defaults
30-template_localization
50-user

To my dismay I also found that virus and spam filtering were also rendered inoperative although mail kept passing through Amavis just fine…

The header of 50-user says : “Place your configuration directives here. They will override those in earlier files“. It should have said “Place most of your configuration directives here” because there is another place where a couple of essential configuration parameters must be set, and it is not made obvious nor even pointed to by the documentation or anything encountered on forums and mailing list. The header of /etc/amavis/conf.d/15-content_filter_mode mentions : “You can modify this file to re-enable SPAM checking through spamassassin and to re-enable antivirus checking“. And indeed, commenting out the couple of lines that obviously needed being commented out re-enabled my favorite content filters.

And the lesson of the day : just porting the old configuration does not cut it, you really need to read every new configuration file – RTFC (Read The Fine Configuration)…

“War upon rebellion is messy and slow, like eating soup with a knife” – TE Lawrence.

In his book “Counterinsurgency Lessons from Malaya and Vietnam: Learning to Eat Soup with a Knife” John Nagl examines how armies learn during the course of conflicts for which they are initially unprepared in organization, training, and mindset. Like “The Seven Pillars of Wisdom” from which TE Lawrence’s quote is excerpted, it is a very useful read for understanding the current situation in Iraq. Written well before Operation Iraqi Freedom it nethertheless shows lessons very relevant to it.

Voicing publicly the private views of many of his peers, Brigadier Nigel Aylwin-Foster, a senior officer in the British Army has published a critical analysis of American military attitudes in Iraq. I do not believe his paper to be the scathing account that the press presents. On the contrary, it confirms the traditional strengths of the US Army and the ability for learning that individuals have demonstrated. But is does point out how heavy institutional inertia prevents widespread adaptation to the nature of counterinsurgency. Compounded doctrinal and cultural factors are shown to be the root cause of the American difficulties in Iraq.

Alwyn-Foster cites Nagl noting that “The American Army’s role from its very origins was the eradication of threats to national survival’, in contrast to the British Army’s history as ‘an instrument of limited war, designed to achieve limited goals at limited cost”. The USA possesses nearly irresistible powers in conventional wars against nation-states. But when confronted with counterinsurgency operations, this strength induces such a bias that it becomes a root of weakness : when the only tool you have is a hammer, everything looks like a nail. For example, Alwyn-Foster underlines that the US Army’s ‘Soldier’s creed’ “enjoins the soldier to have just the one type of interaction with his enemy – ‘to engage and destroy him:’ not defeat, which could permit a number of other politically attuned options, but destroy”. Being a warrior is part, but not all, of being a soldier.

Alwyn-Foster’s paper leads the reader to “the realisation that all military activity is subordinate to political intent, and must be attuned accordingly: mere destruction of the enemy is not the answer”. It is no suprise to me that he reminds me of what my aikido master used to teach us : the goal of martial arts is not the destruction of the opponent, it is the destruction of the conflict. Adversary and partner are one and there can be no winning of the hearts and minds unless this has been realized.

Nowadays, apt has built-in support for GPG verification of package signature. It was silently doing its business of verifying the integrity of Debian packages until a couple of days ago when an ‘apt-get update’ yielded the following error :

W: GPG error: http://ftp2.fr.debian.org testing Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 010908312D230C5F
W: GPG error: http://ftp2.fr.debian.org unstable Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 010908312D230C5F
W: You may want to run apt-get update to correct these problems


Apt is complaining about a package signed with a key it does not know about. We need to import it into the GPG keyring and then tell apt about it. So let’s issue the two following commands as root :

# gpg --keyserver wwwkeys.eu.pgp.net --recv-keys 010908312D230C5F
# gpg --armor --export 2D230C5F | apt-key add -


And that’s it… We’re now back to our regularly scheduled programming…

Can you believe that as of today the string “collaborative adblock” yields no hits on Google ? For now there is the centraly managed list of blockable advertisement at yoyo.org. Conversion scripts make the list available in a large number of popular ad blocking formats and there is even an Adblock filterset updater available for it as a Mozilla extension. You can submit your own items, that is a start. But wouldn’t a truly distributed collaborative peer to peer sharing of Adblock items be staggeringly more powerful ? Something made robust with a reputation system… Let’s seed the meme !

Slugishness and random soft resets… The banes of the Treo 650… I first attributed the problem to Palm OS being a bit out of its depth on such a complex device. But the randomness was too high – I could not correlate the problem with any specific application nor could my favorite inside source at Orange nor apparently any of the people involved with debugging the Treo there. It had to be a hardware problem.

With time and aggravation came the insight that all these problems might be correlated to SIM accesses. My engineering friend at Orange suggested using a piece of paper stuck in the tray to hold the SIM steadier. It did improve the situation somewhat. Later I found that during a sluggishness event, pushing hard on the tray sometimes instantly gets the system back to normal speed.

So it definitely looks like the system barfing on SIM access problems. These problems should not take place and they are caused by a bad SIM tray design. But the operating system should handle them more gracefuly, for example with a “SIM not available” dialog.

Meanwhile I have found another solution that seems to support the SIM access hypothesis : Cingular Wireless proposes a Treo 650 SIM Tray Replacement that works somewhat like the piece of paper jammed in the original tray. So get that and meanwhile remember to push hard on the tray when sluggishness occurs. I am now looking for a way to get that new SIM tray in Europe…

The Treo 650 is the synthesis of all that I tried to achieve ever since I first connected my Palm Pilot 1000 (with the Palm III upgrade) to the Ericsson SH-888 I got after breaking my beloved Ericsson GH-388. It is my best defense against chaos and my best ally in Getting Things Done. Without it, keeping pace with an increasingly complex and accelerating life would be a hard problem.

By the way, taking inspiration in the the methods related to David Allen’s Getting Things Done really does improve productivity while requiring less energy and generating less stress. My favorite arsenal for implenting it includes :

• A smartphone
• A bunch of wikis – some collaborative, some personnal
• Ticket tracking systems

The “work smarter not harder” mantra may be overrused but it does actually apply here… I love my Treo 650 !

The integrated keyboard is surprisingly pleasing to use for small messages and quick notes, but be sure to use the Treo with a foldable keyboard with full-size keys so that you can input large quantity of text painlessly. Unless you need to access heavy documents on the move a laptop suddendly does not make much sense anymore…

OneTouchPhotoDumpToJournal.sh is a trivial script I wrote that saves me much manipulation each time I come back to my workstation with removable media full of photos. It copies all images from a removable media to the directory of the day (created on the fly if not existing), autorotates them and sets the permissions right. It is what I use prior to putting the pictures in an appropriately named directory and running dir_date_serial_rename_all.sh to name them according to my standard.

Canon Eos digital cameras name their files “img_XXXX.jpg” where ‘XXXX’ is a number beween 0001 and 9999. Considering how many pictures we take, duplicate names come fast. In addition, that name is desperately lacking any meta-information that may help in making the file manageable.

So as part of my workflow I have defined a photo file naming scheme that includes the date, the time and the session that picture belongs to. What I define as a session is a group of pictures with a unity of time, place and/or action. I believe I have found a good choice of minimal meta-information to pack in the file name to make it both unique and useful to humans.

The datetime of acquisition is found in an EXIF tag embedded in the file. I deduce the session from the parent directory name because putting all pictures belonging to a single session into an approprietely named directory is what immediately follows my automated download and lossless rotation routine.

Because I am a very lazy person I have of course automated all that. So here is dir_date_serial_rename_all.sh – a small shell scripts that automates my habit of picture files renaming.

This script takes all files in the current directory and renames them using the following pattern :
current_directory_name.date.time.camera_serial.extension

For example :
"my current directory/img_6051.jpg" taken the 2nd of December 2005 at 9:07:59 AM (according to the embedded EXIF metadata)
becomes
my current directory/my_current_directory.20051202.090759.6051.jpg