Serendipitous altruism

Bob Sinclar has for a long time sought inspiration in African music, and his latest hit “What I Want” continues that trend – it features a beat with a clearly identifiable African ascent and a video with one of the dancers performing west-African dance moves. But what I had not realized is that even Fireball’s surprising vocal performance may well find its roots in African music.

On hearing that track, the Bamileke mother of a Camerounese friend told me she definitely recognized a tune she used to sing in her childhood with her friends as a playful vocal trick. I checked on the web and found no one having noticed that. So there, a musical scoop !

We host applications on a couple dozen domain names with more subdomains than I count offhand. We have a policy that anything over which passwords transit should be encrypted, so we have plenty of Apache mod_ssl virtual hosts along with TLS or SSL versions of POP, IMAP, SMTP and XMPP. To provide all that as cheaply as possible, we run our own certificate authority and issued our own root certificate. Certificate authority is a pretty big word for a bunch of Openssl commands, but they do the job fine until we deploy something else to help us. So far, so good.

Since our root certificate is of course not bundled with any browser or operating system, our users are constantly nagged by their browser and mail client until they store it locally. In addition, with no root authority for other servers to refer to, server to server communication is wide open to man in the middle attacks. So at the moment, our cryptography is about as good as snake oil.

The limitations of the current implementation of HTTPS make it difficult to deploy correctly on the cheap. When a client requests a HTTPS connection, it does not tell the server the name of the host it wants to connect to. So the server has no way to choose a certificate, and this is why there can be only one certificate per IP address. IP address being an expensive resource, having one for each virtual host can quickly be prohibitively expensive, at least until IPv6 becomes sufficiently widespread.

With multiple sub-domains, we could use wildcard certificates. They have more risks than benefits and they are not universally supported, but at least they provide a cheap solution. But we host multiple domains, so even that is not the way out for us, nor for the countless wretched sysadmins that share our predicament.

But despair not, wretched sysadmin : you savior has arrived, and its name is Server Name Indication ! SNI is a TLS extension that allows multiple certificates per IP address. Paul Querna has an excellent and easy explanation of what SNI is about – which I reproduce here :

When a client connects to a server using SSL, the server will send the Public Certificate to them. This enables them to actually decrypt the data sent from the server later. Here is a short simplified example:

1. C: (TLS Handshake) Hello, I support XYZ Encryption.
2. S: (TLS Handshake) Hi There, Here is my Public Certificate,
                      and lets use this encryption algorithm.
3. C: (TLS Handshake) Sounds good to me.
4. C: (Encrypted) HTTP Request
5. S: (Encrypted) HTTP Reply

The problem in HTTP is we don’t know which Public Certificate to send, until step 4. This is long after the public certificate has been sent. Protocols such as IMAP and SMTP, which use STARTTLS, have a different pattern:

1. C: (Cleartext) I am using server 'mail.example.com'
2. S: (Cleartext) By The Way, I also support TLS Encryptionn.
3. C: (Cleartext) Lets use Encryption, aka 'STARTTLS'.
4. C: (TLS Handshake) Hello, I support XYZ Encryption.
5. S: (TLS Handshake) Hi There, Here is my Public Certificate,
                      and lets use this encryption algorithm.
6. C: (TLS Handshake) Sounds good to me.
7. C & S: (Encrypted) Exchange Data

Since the client tells the server which host it is connecting to in step 1, the server can pick the correct certificate in step 5. It is possible to do this in HTTP, using TLS Upgrade. This is slightly more complicated, and presents other security issues. The Server Name Indication approach has a much simplier setup:

1. C: (TLS Handshake) Hello, I support XYZ Encryption, and
                      I am trying to connect to 'site.example.com'.
2. S: (TLS Handshake) Hi There, Here is my Public Certificate,
                      and lets use this encryption algorithm.
3. C: (TLS Handshake) Sounds good to me.
4. C: (Encrypted) HTTP Request
5. S: (Encrypted) HTTP Reply

The only difference is a few extra bytes sent in Step 1. The client passes along which hostname it wants, and the server now has a clue which public certificate to send.

The good people at CAcert are following closely how SNI is supported in major pieces of web infrastructure. To summarize, SNI has been supported in mod_gnutls since 2005, but the ominous warning on the mod_gnutls home page does not make mass deployment likely in the short term : “mod_gnutls is a very new module. If you truely care about making your server secure, do not use this module yet. With time and love, this module can be a viable alternative to mod_ssl, but it is not ready“. But fear not : Apache bug 34607 tracks the development of SNI support for mod_ssl, and it only has to wait for the 0.9.9 release of OpenSSL which is said to include support for SNI. So the future is bright ! Support on the client side is more patchy at the moment, but it will likely improve fast as soon as the servers are available.

So when I say the the savior has arrived, I should rather say that it is still underway and it is taking its time. SNI is described in section 3.1 of RFC3546 which dates from June 2003 ! And Paul’s post is from April 2005 – although at that time SNI was already supported in mod_gnutls. I am surprised that the development of such a liberating feature so critical to the providers of collective hosting has been so slow in a an essential pillar of infrastructure such as OpenSSL. I am even more surprised that I have not heard of it before – but now I am quite excited about it !

Since CAcert is tracking SNI support, I guess they will eventually offer name based certificates. Count me in !

Since I began playing with Net_SmartIRC, I found a new way to put that library to work : a Munin plugin script to monitor the number of users in an IRC channel.

Here is an example of the graphical output provided by Munin :

As you can see, the Debian IRC channel is a very crowded place ! You may also notice small gaps in the data : the script sometimes fails on a refused connection, and I have not elucidated the cause. But as the graph shows, I have coded the script so that those failure cases only result in a null output, which Munin handles well by showing a blank record.

Because my lacking skills and crass lazyness prevented me from writing it all in a single language, I hacked that plugin by simply patching together the parts I could produce rapidly :

• irc_channel_users is a Munin plugin script that calls irc_channel_users.php

• irc_channel_users.php performs the actual login and count on the IRC server.

The PHP script is uses Net_SmartIRC which is available in Debian as php-net-smartirc. It must be configured by modifying the hardcoded server and channel – that may not be what is best in production use, but for the moment it works for me. Here is the full extent of the PHP code :

< ?php
include_once('/usr/share/php/Net/SmartIRC.php');
$irc = &new Net_SmartIRC();
//$irc->setDebug(SMARTIRC_DEBUG_ALL);
$irc->setUseSockets(TRUE);
$irc->setBenchmark(TRUE);
$irc->connect('irc.eu.freenode.net', 6667);
$irc->login('usercount', 'Users counting service for Munin monitoring',
'0', 'usercount');
$irc->getList('#test_channel');
$resultar = $irc->objListenFor(SMARTIRC_TYPE_LIST);
$irc->disconnect();
if (is_array($resultar)) {
    echo $resultar[0]->rawmessageex[4];
} else {
}
?>

The irc_channel_users Bash script is also quite simple. Apart from the barely modified boilerplate adapted from other simple Munin bash scripts, the specific meat of the script is as follow :

work_directory=/home/jim/applications/munin/irc_channel_users
php_interpreter=`which php`
user_population=`$php_interpreter $work_directory/irc_channel_users.php
 | awk -F"#" '{print($1)}' | grep -e '^[0-9]+$'`
echo -n "population.value "
echo $user_population

As you can see, the munin bash script is mostly about setting a few Munin variables, calling the php script and formatting the output.

Here are sample outputs :

15:32 munin@kivu /etc/munin/plugins% ./irc_channel_users autoconf
yes

15:32 munin@kivu /etc/munin/plugins% ./irc_channel_users config
graph_title #b^2 IRC channel users
graph_args --base 1000 -l 0
graph_vlabel population
graph_scale no
population.label users

15:32 munin@kivu /etc/munin/plugins% ./irc_channel_users
population.value 6

No demonstration is available on a public site, but the above graph is about all there is to know about the output of this plugin.

The code resides on its own page and updates if they ever appear shall be stored there.

This experience taught me that coding basic Munin plugins is fun and easy. I will certainly come back to it for future automated graphing needs.

And for those who wonder about the new syntax highlighting, it is produced using GeSHi by Ryan McGeary‘s very nice WP-Syntax WordPress plugin.

“When I started skating I pronated terribly, all I did was chase the fastest skaters and tried to keep up at all costs even if this meant pronating for miles. This is the incorrect way to start speed skating”.

When I read that last year, I recognized myself and realized what was wrong with my skating… And luck has it that the author of this statement follows though with a complete step by step method to introduce beginners to speed skating.

One year later I am nowhere near mastering even a quarter of what is explained there, but I can already attest that even that quarter will definitely change your skating and pull you from eternal stagnation.

Just focusing on working the outside edges is a radical departure. It is something I still have to think about and it will still be a long time until it becomes automatic, but the rare times when I am doing this correctly I can feel that I am more efficient and that working on it is the way forward toward better skating.

The method is very well presented with plenty of detailed small video loops and a great progressive approach to make sure that everyone can scale that learning curve. So don’t delay – take a look at it, try the exercises and come back often, it’ll pay !

Looking at server logs in search of clues about a recent filesystem corruption incident, I stumbled upon the following messages :

Aug  5 01:06:01 kivu mdadm: RebuildStarted event detected on md device/dev/md0
Aug  5 01:43:01 kivu mdadm: Rebuild20 event detected on md device /dev/md0
Aug  5 02:15:01 kivu mdadm: Rebuild40 event detected on md device /dev/md0
Aug  5 02:59:02 kivu mdadm: Rebuild60 event detected on md device /dev/md0
Aug  5 04:33:02 kivu mdadm: Rebuild80 event detected on md device /dev/md0
Aug  5 05:24:33 kivu mdadm: RebuildFinished event detected on md device/dev/md0

We never asked for a manual rebuild of that RAID array so I started thinking I was on to something interesting. But ever suspicious of easy leads I went checking for some automated actions. Indeed that was a false alarm : I found that a Debian Cron script packaged with mdadm at /etc/cron.d/mdadm contained the following :

# cron.d/mdadm -- schedules periodic redundancy checks of MD devices
# By default, run at 01:06 on every Sunday, but do nothing unless
# the day of the month is less than or equal to 7. Thus, only run on
# the first Sunday of each month. crontab(5) sucks, unfortunately,
# in this regard; therefore this hack (see #380425).

6 1 * * 0 root [ -x /usr/share/mdadm/checkarray ] && [ $(date +%d) -le
7 ] && /usr/share/mdadm/checkarray --cron --all --quiet

So there, Google fodder for the poor souls who like me will at some point wonder why their RAID array spontaneously rebuilds…

Now why does the periodic redundancy check appear like a rebuild ? Maybe a more explicit log would be nice there.

A few weeks ago I was dealing with strongly backlit black skinned subjects with the occasional shadow and highly contrasted clothing… Needless to say that I had to go wild with exposure compensation and I was not quite self-assured. So I ended up falling back to more chimping than usual.

To tame the bright sunlight sunlight I was wearing a pair of sunglasses. The lenses being corrective for my rather bad myopia I needed to keep them on. So with the combination of unsure exposure and not having my clear corrective lenses I ended up chimping with sunglasses. I suspected this was a daft idea, but now I have seen the results there is no doubt left about it.

LCD monitors on camera backs are quite inaccurate for tonal quality and even more at a less than perfect angle. And looking at them in bright sunlight is as bad as looking at a CRT monitor in daylight. So I guess that chimping has its limits – at some point the instinctive exposure skills of the expert film photographers of old show that their value are still current, even in the digital world.

As some of you may know, I have no fondness whatsoever for proprietary platforms. The mere thought of joining a proprietary instant messaging network sends shivers down my spine : to me the freedom of a decentralized infrastructure is essential and that is why I am a Jabber user. So by now you surely already know what my opinion of Twitter is.

As B. Mann mentions in “Twitter is Jabbber”, XMPP provides Jabber with all the message routing functionnality needed. He adds that “it has a publish and subscribe architecture built in, rather than all these crazy desktop apps that constantly poll the Twitter mothership“.

On top of that, XMPP Extensions enable plenty of functionality to match and surpass Twitter‘s. For example, XEP-0108 “User Activity” defines “an extension mechanism for capturing “extended presence” data about user activities, above and beyond availability“. An XML snippet is worth a thousand words :

<activity xmlns=’http://jabber.org/protocol/activity’>
<relaxing>
<partying/>
</relaxing>
<text xml:lang=’en’>My daughter’s birthday!</text>
</activity>

So let us see what makes Twitter so successful.

First we have mobility. Again B. Mann explains that “my only explanation for the Twitter craze is that North Americans are still enamored of anything that can do the tiniest bit of mobile integration. Yes, Twitter has managed to scale and spend many thousands of dollars paying for SMS gateways“. Sure there are Jabber SMS transports and they are usable from a phone, but you can’t beat free.

But I believe the reason for Twitter‘s success is web integration. Sure, Jabber notifications are provided by many collaborative tools, and there are ressources to make your own such as class.jabber.php, a Jabber library for PHP that I used to build Jabber presence indicator in a web page. But indeed they do not match the level of functionnality that Twitter provides out of the box. We need more web based Jabber clients – that is an interesting area that I’m quite tempted to delve into.

So the proprietary hydra has sprouted one more head, but our swords are far from dull and the jihad shall be eternal !

It has been said from the start but with the availability of a proprietary application platform it became so glaringly obvious that this spring the rumor became insistent – Facebook increasingly looks like the new AOL :

“Fast forward to Facebook 2007 and see similarities: If you want access to their big base of users, develop something in their proprietary language for their people who live in their walled garden. Strangely, many young facebookizens aren’t very net savvy (Facebook *is* their internet) & they have little desire to go beyond the walled garden — just like the old AOL users. There’s even a proprietary Facebook messaging system (kids don’t use much open internet email).”

But it is really Jason Kottke’s “Facebook is the new AOL” followed by “Facebook vs. AOL, redux” that made the rumor grow into a swell in July :

“Facebook is an intranet for you and your friends that just happens to be accessible without a VPN. If you’re not a Facebook user, you can’t do anything with the site. Nearly everything published by their users is private. Google doesn’t index any user-created information on Facebook. All of the significant information and, more importantly, interaction still happens in private. Maybe we shouldn’t be so excited about the web’s future moving onto an intranet.”

Steve Rubel sums that up : “Facebook gives nothing back to the broader web. A lot of stuff goes in, but nothing comes out”.

In a comment to Jeff Atwood’s “Avoiding walled gardens on the Internet”, Alex Chamberlain makes another parrallel with an historical precedent that seems lost to many among the current generation of Internet users :

“I’ve had the same uncomfortable feeling about web-based message boards. Prima facie, the walled-garden model violates the principle that information wants to be free.

Think of how Fidonet helped to open up the insular world of BBSs. Think of how Usenet was designed to be inherently inclusive (just start a news server on a Net-connected machine and all its users instantly join the “conversation”) and eternal (because decentralized). Now, Usenet is irrelevant to all but a tiny online subculture, BBSs are dead, and the traffic that those media would have borne is now happening on Web-based message boards, whose owners can edit content, forget to pay for their server space, or shut down for good at will, and whose content (more important) is essentially invisible to Google unless you know the secret password (the URL of the site’s archives). Balkanized again !”

Just as most of them are using stupid proprietary instant messaging networks instead of Jabber, they are now deliberately walling themselves in again. As Matthew says :

“Facebook is reinventing the wheel a little in an attempt to give anybody and everybody their very own web presence. Except it’s not a web presence, it’s a Facebook presence, bound by Facebook’s rules. The experience feels forced and leaves me wanting more. [..] I want to be able to find you on Google, read your weblog and browse your Flickr photos”.

But it is not just the users who drink the Facebook Kool Aid… As the RSS blog mentions, even developpers are falling for it :

“Everybody is going nutty about the Facebook platform. They are writing custom widgets for Facebook. They are saying that Facebook is the greatest because it support proprietary widgets. WTF ? We already have an API for widgets, it’s called HTML. We’ve been embedding widgets in MySpace for years using HTML. Why does Facebook need a proprietary widgets API ? It’s called lock-in. A walled garden. The work you do on your Facebook widget doesn’t port to other social platforms. In this case, platform means proprietary. When the euforia fades, just how many $billions are going to get spent by 3rd parties to better the Facebook platform ? This is nuts !”

Yes, many developpers are happily coding for a closed platform. And they are definitely locked in as Richard MacManus explains in “How Open Is Facebook, Really ?“ :

“Facebook ultimately is a closed, proprietary system. Primarily this is because Facebook doesn’t use existing Web standards for mark-up or database language. Instead of using HTML and SQL, Facebook uses two “variants” – called FQL and FBML. The official reason for the variants is that they offer more functionality and integration within the Facebook environment – which is no doubt true, however it also of course means your apps can only run in Facebook. As Andreessen noted, the upshot is that “Facebook’s own code and functionality remains closed and proprietary.”

But Facebook now has such an influence on the mass market that it can’t be ignored and even people like me can’t resist taking a look if just to see what all the fuss is about. So here is my Facebook profile… Oops ! That’s right. The walled garden thing. Forgot about that. You have to be a member… Screw that !

So welcome to Faceprison ! Nice clean interface conveniently packaged in a proprietary walled garden from which your data shall never escape.

A little search later I find that the word “Faceprison” has been by used by Neil Dixon who last June posted feelings similar to mine in “Facebook – a very big claustrophobic bubble” :

“There is one element to Facebook that makes the alarm bells ring for me, in contrast to almost every other major social network: you have to be logged-in to do anything, anything at all. Nothing is visible or accessible to the outside. Even notification emails about messages, etc., force you to log in to view them. Everything is designed to get you inside, and keep you there.

Facebook to me feels immediately claustrophobic, a state of interweb virtual bondage where the only safeword is ‘logout’. The sterilised, razorwire-topped walls are (currently) unscalable and the locks are more sturdy than Broadmoor. But even in a physical prison you can have real visitors: in Facebook, your visitors have to join and become part of the exclusive hive themselves, trapped squirming in its peer pressure driven, shallow society.

I fear the worst for those lost souls in particular who will suffer a similar fate to those of us who took up AOL in the early days”

Users never learn and history repeats itself. Have fun poking each other to digital death ! Or go read Ethan Zuckerman’s “Web 2.0 and the web serf” and understand why friends don’t let friends sink their data into proprietary bottomless pits.

Well… I can’t conclude on such a dark note, so I’ll cite the more hopeful outlook of Jason Kottke about Facebook :

“Faced with competition from this open web, AOL lost… Running a closed service with custom content and interfaces was no match for the wild frontier of the web. Maybe if they’d done some things differently, they would have fared better, but they still would have lost. In competitive markets, open and messy trumps closed and controlled in the long run. Everything you can do on Facebook with ease is possible using a loose coalition of blogging software, IM clients, email, Twitter, Flickr, Google Reader, etc. Sure, it’s not as automatic or easy, but anyone can participate and the number of things to see and do on the web outnumbers the number of things you can see and do on Facebook by several orders of magnitude (and always will).

At some point in the future, Facebook may well open up, rendering much of this criticism irrelevant. Their privacy controls are legendarily flexible and precise…it should be easy for them to let people expose parts of the information to anyone if they wanted to. And as Matt Webb pointed out to me in an email, there’s the possibility that Facebook turn itself inside out and be the social network bit for everyone else’s web apps. In the meantime, maybe we shouldn’t be so excited about the web’s future moving onto an intranet”.

As Jonathan Kahn writes, “microformats and OpenID will kill Facebook’s business model“. Information wants to be free !