Serendipitous altruism

Ever found yourself painstakingly trying many combinations of dates to find the cheapest flight ? TechCrunch reports on FlySpy, a tool that will make that process painless :

The way it works is that I give it a departure city and a destination city and optionally a departure date and length of stay. The search result, which returns very quickly, will present me with a graph of flight prices over the next 30 days so that I can quickly look at which days are the cheapest to fly. To book a flight I just click on the point in the graph. Simple.

Jason Kottke envisions taking the idea even further by using historical data to predict the optimal date of purchase. He also suggest applying it to all industries where yield management is relevant.

Is transparent pricing coming to the air travel market ? Probably not even in your wildest dreams as Keith Devlin explained in 2002 :

Faced with all this confusion, with computers constantly monitoring sales and adjusting fares as often as ten times a day, the only real option for the fare conscious air traveler is to use a Web service to try to locate the best deal. [..] But just how well do those search engines do ? Not very, is the answer. And with good reason. Airline pricing has grown so complex that it is now practically impossible to design an algorithm that will find the cheapest fare. In mathematical terms, the (idealized) problem of finding [..] the lowest fare is NP hard [..] This is the perhaps surprising result obtained recently by mathematician Carl de Marcken.

Although depressing, this piece of research highlights an interesting aspect of FlySpy : it does not try to find the solution to the problem. Instead it follows the patch of scientific visualisation : when confronted with overwhelming amounts of data, the best way to understand it is to draw a nice picture.

Total Annihilation introduced unparalleled innovation and ambition for its time. I consider it the greatest real-time strategy game I ever played – and I played many. Its legacy still endures ten years later through a robust community of gamers and modders : ever since the demise of the company that created it, Total Annihilation has been enhanced, extended and corrected by its users. My favorite modification is Uberhack : in my opinion Uberhack is the most robust, most complete and best balanced incarnation of the game.

Total Annihilation have kept alive the dream of a game deep enough for hardcore gamers while retaining a strong appeal to the casual crowd. It seems that this baby is in the process of being developped by Gas Powered Games and expected for 2007. It is named Supreme Commander.

Gamespy has a few articles, interviews and screenshots about SupCom. It is a good place to get yourself acquainted to the beast. The Supreme Commander official forums have a nice sticky thread gathering the known public facts. Once you have read all that you can come and join us at the Supreme Commander official forums : 20k posts in six months for a strategy game that will not be out for at least eighteen months is very impressive. If the product holds up to its promises it will surely have a massively commited community of gamers and modders behind it.

My bet is that it is going to be even better than all that it is hyped to be. See you in 18 months !

I now rely on Jabber to publish my presence. Not everyone has a Jabber client at hand but Web access is about ubiquitous so it makes sense to me to make my presence available on a Web page. To me this is a personnal itch and an excuse to practice some PHP and learn a little about the exciting world of XMPP.

The source code for my presence indicator is now available for whoever wants it. It is made possible by class.jabber.php, a Jabber library for PHP. I customized one of the examples provided with class.jabber.php. It has many rough edges such as the user’s name sprinkled across the markup code instead of being a proper parameter, or the page’s presence being sent to the polled account instead of being hidden. And it is quite slow because it queries at load time. But I like it better than the example packaged with class.jabber.php.

It basically does the job and I am therefore quite sure that there will be interested users. So this is the “early” part of “Release Early, Release Often”. Take that hack, customize it, tweak it and generally improve it… And most of all please report back somewhere so that everyone can benefit. Here would be a good start.

As many have said before, Jabber is to instant messaging what SMTP is to email:

Today most people understand instant messaging almost entirely in terms provided by AOL, Microsoft and Yahoo. As with on-line services and operating systems, closed IM systems became familiar to the general public long before the open ones. In terms of architecture and control, AOL’s, Microsoft’s and Yahoo’s instant messaging systems are as closed and non-interoperable as Prodigy, Compuserve and AOL were back in the [early] nineties.

In other words, AOL, Microsoft and Yahoo all want to keep private something that should be no less public than mail and web services. All of those companies’ IM systems use the Internet and offer free clients, but they offer no internet service anybody can build on.

This quote dates back from 2001… After all these years I at last consider the Jabber ecosystem to reach both reasonnable usability and critical mass. I believe we can thank Google for tipping the scale. So what’s next ?

I have said it privately since 2002 : the buddy list will become the center of user interaction with the mobile “phone”. Presence management holds every synchronous media together. Through presence management, messaging has moved from the periphery of the desktop toward its core. It changes the environment in which other services interoperate. For example it makes the answering machine obsolete : no one is going to knowingly initiate a call when there is on one on the other end. And the buddy list is where convergence happens : any communication can be initiated from there, not just what the operator deems reasonnable to provide.

As usual the operators are busy fighting rear-guard action, but they cannot keep the customer from craving the ideal of the Stupid Network. GPRS and UMTS enable innovation at the edge and that is the end of the walled gardens. Of course capital expenditure and spectrum scarcity still provide them with a fairly entrenched position, but we are now firmly in Internet land and there is no going back.

So as we all have been thinking all along, the open protocol takes over the closed ones. It takes time, but it is inexorable. The mobile networks are next and they know it… But that does not mean they can avoid it.

Mobile network operators will not like it but instant messaging is going to be priced at bulk data prices, not anywhere near the EUR 1000 a MB that they can get away with for SMS. Earlier attempts to sell services instead of selling raw data have met mixed responses – it will only get worse and I do not believe this to be only my wishful thinking.

It has been a few days since I have begin using Chatopus as a Jabber client on my Treo 650. Overall I am very pleased. Chatopus is very simple and quite polished. It works as advertised with no effort and no undue feature creep.

The only major functionnality that I miss in Chatopus is background operation during an SSL secured connexion. It works fine with a cleartext Jabber connexion, but not with SSL. According to the author, Tony Yat-Tung Cheung it “is an issue with Palm OS’ built-in SSL library. Unfortunately, PalmSource is no longer updating the Palm OS Garnet“.

I also found a few minor annoyances. The first one is that the roster can only display at any one time the contacts from one categories or all catefories together, but not a choice of selected categories. Although that Palm OS UI convention is normally simple and practical this is not the first time I find it annoying… This illustrate the fact that UI conventions are guidelines that sometime can be overriden by specific needs. But by sticking to the convention Chatopus does a good job of remaining simple and easy to use.

Chatopus also lacks a custom status in addition to the standard standard Presence Type (Available, Away etc.). For exemple, when the Exodus Jabber client sets the “Extended Away” Presence Type it transmits the standard “xa” flag and an additional customizable and more informative status details such as “Extended away as a result of idle”. It would be nice if Chatopus transmitted this too in addition to the standard Presence Type.

Last minor gripe : if for some reason the GPRS link goes down Chatopus does not reconnect and restart it. Maybe this should be a user-configurable option.

Overall Chatopus makes the mobile use of Jabber a breeze. On my Treo 650 it now has its own button so that my roster is only a keypress away… That tells something about how useful I find it !

When I type the beginning of a name in Mutt, pressing CTRL-T will give me a list of addresses whose real name matches my entry, and I just have to choose the one I want. As a result I seldom type more than a few letters to enter a complete address. This trick is called an external address query which is the mechanism by which Mutt supports connecting to external directory databases through a wrapper script.

The wrapper script I used is called lbdbq and it is part of lbdb. Lbdb provides an abstraction layer to a wide variety of address sources. The one I use most is the m_inmail database built by lbdb-fetchaddr. lbdb-fetchaddr reads a mail on stdin and extracts adresses from the mail header to append them to $HOME/.lbdb/m_inmail.list. Thanks to lbdb-fetchaddr, if we ever exchanged mail I can completion your name to you address.

There are two ways to feed the m_inmail database. The first one is to do it as you go using a MDA filter that pipes a copy of incoming messages to lbdb-fetchaddr. I copied the maildrop recipe from Mark Weinem’s .mailfilter example :

if ( $SIZE < 10000 )
cc "|lbdb-fetchaddr"

As you may have guessed the filter excludes messages larger than 10 KB because lbdb-fetchaddr is not supposed to work with “big messages”. But Mark Weinem’s .mailfilter example does not provide any further explanation and I found none myself. What I did find is that this rule does from time to time cause deferals after maildrop delivery failures for a reason I have yet to discover. Until I find that reason I will refrain from using it.

So I found another way to do it : feed lbdb-fetchaddr my 4 GB of mail dating back from 2001 (the moment when a disk crash taught me the value of good backups). This method has the added value of taking advantage of historical data, not just future traffic. On mutt-users, Jason Helfman asked “if anyone out there has any success in piping their inbox to lbdb-fetchaddr” and got no answer. Indeed, lbdb-fetchaddr does not work like that : each message must be fed individually. Of course I had to automate the process so that all my maildirs can be processed in one go, so I wrote a couple of small scripts :

If you do not have an existing lbdb database, you should run lbdb-fetchalladdresses-firsttime.sh first : it scours a maildir hierarchy to records all addresses from the headers in the lbdb m_inmail database. Once you have done that you can from time to time update your database with lbdb-fetchalladdresses-daily.sh who does the same thing as lbdb-fetchalladdresses-firsttime.sh but only for messages not older than a certain user-set age.

For performance reasons lbdb-fetchaddr appends new addresses to the database without removing duplicates – duplicates are only removed at query time. For 4 GB of mail, lbdb-fetchaddr produces a 6 MB m_inmail file which piping through ‘uniq‘ reduces to a mere 300 KB. Since the process is normally a one-off, I believe it is well worth the transient load. The update script only deals with a few additional entries, so the ‘uniq‘ load is negligible especially since this is likely to be a nightly cron job.

Don’t forget to install the lbdb package before and to add the following line to your muttrc :

'set query_command = "lbdbq '%s'

So there you go : happy CTRL-T completion !