If you want to skip the making-of story, you can go straight to the laconica2IRC.pl script download. Or in case anyone is interested, here is the why and how…

Some of my best friends are die-hard IRC users that make a point of not touching anything remotely looking like a social networking web site, especially if anyone has ever hinted that it could be tagged as “Web 2.0” (whatever that means). As much as I enjoy hanging out with them in our favorite IRC channel, conversations there are sporadic. Most of the time, that club house increasingly looks like an asynchronous forum for short updates posted infrequently on a synchronous medium… Did I just describe microblogging ? Indeed it is a very similar use case, if not the same. And I don’t want to choose between talking to my close accomplices and opening up to the wider world. So I still want to hang out in IRC for a nice chat from time to time, but while I’m out broadcasting dents I want my paranoid autistic friends to get them too. To satisfy that need, I need to have my IRC voice say my dents on the old boys channel.

The data source could be an OpenMicroblogging endpoint, but being lazy I found a far easier solution : use Laconi.ca‘s Web feeds. Such solution looked easier because there are already heaps of code out there for consuming Web feeds, and it was highly likely that I would find one I could bend into doing my bidding.

To talk on IRC, I had previously had the opportunity to peruse the Net::IRC library with great satisfaction – so it was an obvious choice. In addition, in spite of being quite incompetent with it, I appreciate Perl and I was looking for an excuse to hack something with it.

With knowledge of the input, the output and the technology I wanted to use, I could start implementing. Being lazy and incompetent, I of course turned to Google to provide me with reusable code that would spare me building the script from the ground up. My laziness was of course quick to be rewarded as I found rssbot.pl by Peter Baudis in the public domain. That script fetches a RSS feed and says the new items in an IRC channel. It was very close to what I wanted to do, and it had no exotic dependancies – only Net::IRC library (alias libnet-irc-perl in Debian) and XML::RSS (alias libxml-rss-perl in Debian).

So I set upon hacking this script into the shape I wanted. I added IRC password authentication (courtesy of Net::IRC), I commented out a string sanitation loop which I did not understand and whose presence cause the script to malfunction, I pruned out the Laconi.ca user name and extraneous punctuation to have my IRC user “say” my own Identi.ca entries just as if I was typing them myself, and after a few months of testing I finally added an option for @replies filtering so that my IRC buddies are not annoyed by the noise of remote conversations.

I wanted my own IRC user “say” the output, and that part was very easy because I use the Bip an IRC proxy which supports multiple clients on one IRC server connection. This script was just going to be another client, and that is why I added password authentication. Bip is available in Debian and is very handy : I usually have an IRC client at home, one in the office, occasionally a CGI-IRC, rarely a mobile client and now this script – and to the dwellers of my favorite IRC channel there is no way to tell which one is talking. And whichever client I choose, I never missing anything thanks to logging and replay on login. Screen with a command-line IRC client provides part of this functionality, but the zero maintainance Bip does so much more and is so reliable that one has to wonder if my friends cling to Irssi and Screen out of sheer traditionalism.

All that remained to do was to launch the script in a sane way. To control this sort of simple and permanently executed piece of code and keep it from misbehaving, Daemon is a good way. Available in Debian, Daemon proved its worth when the RSS file went missing during the Identi.ca upgrade and the script crashed everytime it tried to access it for lack of exception catching. Had I simply put it in an infinite loop, it would have hogged significant ressources just by running in circles like a headless chicken. Daemon not only restarted it after each crash, but also killed it after a set number of retries in a set duration – thus preventing any interference with the rest of what runs on our server. Here is the Daemon launch command that I have used :

#!/bin/bash
path=/usr/local/bin/laconica2IRC
daemon -a 16 -L 16 -M 3 -D $path -N -n laconica2IRC_JML -r -O $path/laconica2IRC.log -o $path/laconica2IRC.log $path/laconica2IRC.pl

And that’s it… Less cut and paste from Identi.ca to my favorite IRC channel, and my IRC friends who have not yet adopted microblogging don’t feel left out of my updates anymore. And I can still jump into IRC from time to time for a real time chat. I have the best of both worlds – what more could I ask ?

Sounds good to you ? Grab the laconica2IRC.pl script !

Have you tried one more time to convince you parents to switch to web feeds to get updates from the family ? Do you cringe when you see your colleague clumsily wade through a collections of sites main pages instead of having them aggregated in a single feed ? Or did your technophobe girlfriend miss the latest photo album you posted ? With a wide variety of source acknowledging that web feeds ans web feeds readers being perceived as too technical, many of us have scaled back this particular evangelization effort to focus it on users ripe for transitionning from basic to advanced  tools.

Breaking through that resistance outright is beyond our power, but we can get around it. Electronic mail is a mature tool with well understood use cases with which even the least competent users feels comfortable thanks to how easily it maps with the deeply assimilated physical mail model. This is why Louis Gray has started mailing Google Reader items to promote the use of that web feed reader. But we can do better than that by building a fully automated bridge from web feed to email.

Our hope for plugging the late adopters into the information feeds is named rss2email. As its name suggests, Aaron Swartz’s GPL-licensed rss2email utility converts RSS subscriptions into email messages and sends them to whatever address you specify. Despite the name, it handles Atom feeds as well, so you should be able to use it with just about any feed you like. And of course rss2email is available from Debian.

The nice introduction to rss2email by Joe ‘Zonker’ Brockmeier is all the documentation you need – and rss2email is so simple that you probably don’t even need that. I now have some of my favorite late adopters each plugged into his custom subset of my regular information distribution feeds. The relevant news stories get mailed to them without me having to even think about it. And the best part is that they now read them !

I keep my regular daily reading sources in publicly available aggregator built with the excellent although not updated for a good while Gregarius. I thought it might also be useful to gather all my private feeds into another Gregarius instance for my exclusive perusal.

Kudos to LinkedIn whose “Network Updates” feeds is available through an URL with a path long enough to actually be used as a basic shared secret, which is adequate security for protecting such a low value information. This lends itself perfectly to the sort of private aggregation I want.

But no brownie points to Facebook whose behavior toward open communications never fails to disappoint. Last April, Facebook made its “Friend’s Status Updates” feed available in RSS format through the same sort of pseudo secret URL as LinkedIn. So far so good : a nice gesture of openness which made me happy when I pointed Firefox at my feed’s URL.

As I was compiling the list of feeds I was going to aggregate, I tested each of them from my web server’s Z shell to check their reachability. When I pointer ELinks at my “Friend’s Status Updates” feed URL, here is what I got :

You are using an incompatible web browser.

   Sorry, we're not cool enough to support your browser. Please keep it

   real with one of the following browsers:

     * Firefox

     * Opera

     * Safari

     * Flock

Baaad Facebook ! This is so incredibly lame : not only is it an unnecessary annoyance, but it is also completely ineffective since I’ll just have to insert a wget download in my hourly Gregarius update script and tell wget to pretend being Firefox. Gregarius will then happily download the local copy through my web server. I just tested and wget –user-agent=”Mozilla” works just fine.

Even easier : I’ll modify my local Gregarius copy so that util.php at line 539 reads “$client->agent = Mozilla;” instead of “$client->agent = MAGPIE_USER_AGENT;” so that Magpie (the RSS import library for Gregarius) tells Snoopy (the HTTP client for Magpie) to use whatever Facebook wants to hear to deliver the goods.

So Facebook :

1. Gratuitously annoys its users
2. Does not even do it competently

Now, isn’t it time to really open instead ?

As some of you may know, I have no fondness whatsoever for proprietary platforms. The mere thought of joining a proprietary instant messaging network sends shivers down my spine : to me the freedom of a decentralized infrastructure is essential and that is why I am a Jabber user. So by now you surely already know what my opinion of Twitter is.

As B. Mann mentions in “Twitter is Jabbber”, XMPP provides Jabber with all the message routing functionnality needed. He adds that “it has a publish and subscribe architecture built in, rather than all these crazy desktop apps that constantly poll the Twitter mothership“.

On top of that, XMPP Extensions enable plenty of functionality to match and surpass Twitter‘s. For example, XEP-0108 “User Activity” defines “an extension mechanism for capturing “extended presence” data about user activities, above and beyond availability“. An XML snippet is worth a thousand words :

<activity xmlns=’http://jabber.org/protocol/activity’>
<relaxing>
<partying/>
</relaxing>
<text xml:lang=’en’>My daughter’s birthday!</text>
</activity>

So let us see what makes Twitter so successful.

First we have mobility. Again B. Mann explains that “my only explanation for the Twitter craze is that North Americans are still enamored of anything that can do the tiniest bit of mobile integration. Yes, Twitter has managed to scale and spend many thousands of dollars paying for SMS gateways“. Sure there are Jabber SMS transports and they are usable from a phone, but you can’t beat free.

But I believe the reason for Twitter‘s success is web integration. Sure, Jabber notifications are provided by many collaborative tools, and there are ressources to make your own such as class.jabber.php, a Jabber library for PHP that I used to build Jabber presence indicator in a web page. But indeed they do not match the level of functionnality that Twitter provides out of the box. We need more web based Jabber clients – that is an interesting area that I’m quite tempted to delve into.

So the proprietary hydra has sprouted one more head, but our swords are far from dull and the jihad shall be eternal !

Since my migration to PHP 5 revealed a problem with the ancient Lilina 0.7 my interest in Lilina has been rekindled. As I said, I am quite hopeful because I would like to keep using Lilina for small aggregations and avoid deploying the more complex Gregarius where its better scalability is not needed. What first attracted me toward Lilina still holds true.

I started by checking out the development version from Google Code and I deployed it on a handful of small aggregations such as my personal feed. I then immediately started gently pestering the project lead about a few bugs, first by mail and then using the nice Google Code Lilina issue tracker. Ryan Mc Cue has been very responsive and very nice. I look forward making my small contributions in helping him bring Lilina toward 1.0

A year has passed since Ryan announced his intentions. He has already done a lot but his roadmap for Lilina 1.0 is very ambitious so there is still a lot of work remaining. At the current pace it may take a few more months, but this is a journey I am eager to follow : I like Lilina and since I am a novice PHP coder I am quite happy whenever I find a project I like and to which I can give a little help. I am actually quite proud of being mentioned in the credits for having brought RSS output to Lilina : it was the first time I had some of my code included in a significant project.

After migrating an host to PHP5 I found that Lilina 0.7 no longer works and instead produces the following error :

PHP Fatal error: Cannot redeclare class soapclient in /your-lilina-directory/inc/nusoap.php on line 4096

Happily, Robert Mao at “Inmates are Running the Asylum” had already stumbled on this and found a solution.

Robert found a report of the Nusoap library conflicting with PHP5’s built in SOAP functions. The only use of Nusoap in Lilina is the Google API. So Robert found that by disabling the peripheral functionality dependant on the Google API Lilina no longer produced the error.

It works but it is a quick and dirty fix. Enters Ryan Mc Cue who took over Lilina’s development last year. Ryan soon mentioned that the aforementioned functionality is completely disabled in the current development version of Lilina which therefore works fine with PHP5.

There has not been a release of Lilina in quite a while but indeed Ryan and his friends have not been idle and on top of a Brand new web site there have been many commits to the Lilina Subversion repository on Google Code.

So Lilina 1.0 is in development and I’m going to take look at it. I am quite hopeful because I would like to keep using Lilina for small aggregations and avoid deploying the more complex Gregarius where its better scalability is not needed.

I am currently supporting a French centrist political party, mostly helping local militants to improve their web presence by giving them tailored tools and pushing them toward a sensible communication strategy and the organization that goes with it. WordPress and Dokuwiki were of course among the first tools out of the box. I then soon considered the constellation of militant small-time blogs and decided there was a prime target for aggregation.

I first deployed the Lilina PHP news aggregator – I love it and it worked very well for tiny “me and my friends” feed aggregations. With Lilina under new management there were even prospects for improvement. But when the number of inbound feeds began to soar toward 150 I realized that the whole user experience was sinking into a pit of cold molasses. Unexpectedly the host was not even significantly loaded, it was just that updating from that many feeds sequentially was taking much time.

Enter Gregarius. Of course I knew about Gregarius before. But I had no reason to go through the slight hassle of using the Mysql database that Gregarius needs : deploying Lilina only requires the unpacking of my custom distribution of Lilina patched for provinding RSS output. Lilina crawling gave me the reason. On top of that, the Lilina theme for Gregarius really made migration as painless as possible for Lilina refugees who can feel at home right out of the box.

Installation was dead easy and importing all those incoming feeds was done in the single step of entering the URL for Lilina‘s OPML output and waiting a few seconds for all the feeds to be fetched and parsed. And there you are : 130 blogs (and growing) aggregated effortlessly with reasonnable response time and barely any load on the host. Gregarius is even easier than Lilina to administer, and it has categories and tags that Lilina does not, and also does searching.

So from now on I’ll use Lilina for aggregating up to about a couple dozen feeds. Beyond that the territory belongs to Gregarius !

From Panayotis Vryonis own account, development of the Lilina PHP news aggregator he created has not been very active in the past few months. But fear not : new developments could well appear on the horizon in the coming months – Ryan McCue announced his intentions in a comment on this blog :

“I now a project admin on lilina and plan to make a lot of changes”.

This is good news for Lilina users, especially as Ryan McCue intends to integrate my Lilina RSS output patch to the distribution.

Coldforged mentioned that Lilina “silently failed on several malformed feeds“. Coldforged adds “that wouldn’t be so bad if it survived such indignities but it didn’t. Instead my entire aggregation page would simply not load and the only notice I ever got that something was awry was the PHP error_log in the aggregator directory“. I’ll look out for such behavior, but so far I have had no problems with the few feeds I am using.

Were I looking for something with more features than Lilina I would probably turn to Gregarius with the Lilina theme because I like Lilina’s way so much. And the “RSS View” plugin apparently provides the outgoing something like the aggregated RSS feed that I implemented in Lilina by splicing Feedcreator in.

But for now I’ll stick with Lilina because it is extremely easy to use. The lack of need for a database makes deployment as simple as copying files, setting permissions (the archive of the patched Lilina I provide now features a script to do it for you) and editing a couple of lines in a configuration file. That is hard to beat.

Aggregated RSS feeds presented as HTML and Javascript by Lilina are very sweet. The more we used them, the more we missed having them served as RSS. After much research it seemed to us that there is no nice and easy PHP code capable of mixing RSS as RSS. There are plenty of feed mixers offered as a service but very few offered as a product.

On the fetching and parsing side, Lilina had everything I wanted. All I needed was to make it generate RSS instead of HTML.

I went foraging for RSS creation libraries. The first one I found was XML-RSS-Aggregate . I liked it because the example provided with XML-RSS-Aggregate is an RSS agregator that ouptuts RSS – exactly what I was looking for. But Shlomi Fish mentioned that “this module is unmaintained and no longer works very well. The author (and I) recommend that you use XML::Feed now“. So I took a look at XML-Feed and found it too complex for my meagre skills. And I’m not that hot with Perl anyway. So I went looking somewhere else.

I found my salvation in Feedcreator. Feedcreator creates valid feeds in various formats, features configurable caching, reasonnable documentation and readable code. I found it quite easy to use. All it needs is an array of RSS elements, and that is exactly what Lilina provides.

I took Lilina’s index.php, cleaned up the HTML generation, spliced in the example code from Feedcreator, mapped input to output and lo and behold I had a reasonably valid RSS output by Lilina. Very sweet !

Source code of the modified Lilina with Feedreactor hybridation is available here.

I even added a cute RSS icon to Lilina’s default layout…

Looking for a way to fetch multiple RSS news feeds and present them as a single HTML page I found the wonderful Lilina.

“Lilina is a simple but powerful news aggregator written in PHP. No database is needed, RSS/ATOM parsing is done by the excellent MagpieRSS library”.

That piece of advertisement is all true : Lilina is dead simple to set up, requires no special dependancies and produce very nice aggregated news feeds. This was love at the first sight !

I immediately set up a couple of aggregated news feeds :

• Grabeuh planet shows articles and news items published by my fellow geek friends and myself.
• Jean-Marc’s little corner of the world shows all photos, articles and news items by me.

Next will be a personal feed gathering all my favorite places that publish irregularily. That page will save me quite a lot of clicking around checking for updates.