A few old friends and I often hang together on our usual IRC channel, and while discussing we like to share links. I thought it might be a good idea to use a web tool for that so that we can also share our favorite links with the rest of the world. I thought a private Digg clone could play that role, and I needed an excuse for more PHP development. After some research I found that Pligg was the best candidate. I downloaded Beta 9.7 and I have been quite amazed how fast I got a smooth Pligg site up and running – it seems like much work went into this code.

The first bug that arose was mangled accents in some URLs. I solved it by adding a few cases to a character substitution matrix. This matrix is used to sanitize strings before using them for URLs. The problem is that this method requires each accented character to be treated individually. Maybe a more general solution exists. Anyway I am surprised that no other user noticed that problem with French accents before.

Since I wanted to use Pligg in a way closely associated with our IRC habit, I thought it might be a good idea to make Pligg capable of notifying an IRC channel of an upcoming story. Rumaging through PHP libraries capable of speaking IRC I ended up using Net_SmartIRC which is available in Debian as php-net-smartirc. The examples packaged with the class are quite good at teaching its use so I soon wrote a small proof-of-concept of notifying an IRC channel from PHP.

Some more hacking later I figured how to splice that IRC notification functionality into Pligg. This is a prototype : it works fine for me but you need to overwrite a file from the standard Pligg and edit it with your parameters, and it probably ignores any convention used by those who are familiar with working on Pligg. To reach release quality this functionality should probably be architectured as a proper configurable Pligg plugin. To use it, you need to install Net_SmartIRC which is available in Debian as php-net-smartirc. Make sure that the path of that library as declared in my submit.php is correct.Then you need to edit my submit.php to set connect parameters, login parameters, and the channel. Make a backup of your original submit.php and replace it with mine. That is all !

With IRC notification in place I found that I did not want just anyone spamming our favorite IRC channel. Pligg has three hardcoded user levels : “user”, “admin” and “god”. By default the users can submit posts. I decided to restrict posting to the “admin” and “god” levels. That way we keep control of posting, our friends can vote and comment as “user” and the rest of the anonymous world can read the site.

With the beginning of a familiarity with the Pligg code it did not take me too much grepping to understand the conditional access system and wedge a typical implementation where it should work. And work it did, just as advertised ! I am not sure that the core Pligg developers intended this previously clean switch statement to be hacked like that, but it works for me. A further step would be to make this restriction code conditional and to make the condition a user-configurable parameter in /admin_config.php so that god can choose at which user level submissions are allowed. I am sure that many site operators would appreciate this feature.

So as a summary and conclusion, here is the patch that modifies Pligg’s 9.7 submit.php to provide post submission restriction by user level and IRC notification of posts :

• The original submit.php from Pligg 9.7
• submit.php from Pligg 9.7 patched to provide post restriction by user level and IRC notification of post
• diff between the above files.

Have fun with Pligg ! I’m done with it for now but who knows what other functions I’ll feel like hacking into it in the future…

After migrating an host to PHP5 I found that Lilina 0.7 no longer works and instead produces the following error :

PHP Fatal error: Cannot redeclare class soapclient in /your-lilina-directory/inc/nusoap.php on line 4096

Happily, Robert Mao at “Inmates are Running the Asylum” had already stumbled on this and found a solution.

Robert found a report of the Nusoap library conflicting with PHP5’s built in SOAP functions. The only use of Nusoap in Lilina is the Google API. So Robert found that by disabling the peripheral functionality dependant on the Google API Lilina no longer produced the error.

It works but it is a quick and dirty fix. Enters Ryan Mc Cue who took over Lilina’s development last year. Ryan soon mentioned that the aforementioned functionality is completely disabled in the current development version of Lilina which therefore works fine with PHP5.

There has not been a release of Lilina in quite a while but indeed Ryan and his friends have not been idle and on top of a Brand new web site there have been many commits to the Lilina Subversion repository on Google Code.

So Lilina 1.0 is in development and I’m going to take look at it. I am quite hopeful because I would like to keep using Lilina for small aggregations and avoid deploying the more complex Gregarius where its better scalability is not needed.

My quick and easy tool for renaming files straight from camera in a way that will make sense in most contexts is now even better.

Changelog :

• Field order has been changed in order to fit even more situations
• The script now tolerates upper and lower case file name extensions
• Prefix can be personalized : your initials probaly make more sense than “IMG”
• Extension case set is now set according to user configuration

Example :

“my current directory/img_6051.jpg” taken the 2nd of December 2005 at 9:07:59 AM (according to the embedded EXIF metadata)

becomes :

“my current directory/20051202.090759.JML.6051.my_current_directory.jpg”

Set your personal parameters in five seconds, and call just one simple command with no arguments for the whole directory. How easier can it get ?

See it, download it !

My last rant about Openoffice’s lack of a proper outline mode apparently struck a chord if I judge from the number of pageviews and the reactions I gathered. If, like me, you eagerly await this functionality you will be happy to learn that some recent activity around Openoffice Writer’s longstanding issue 3959 aka “Outline View (aka MS Word)” has provided us with some hope.

Mathias Bauer, project lead of the OpenOffice.org Application Framework and manager of the teams for the application framework, Math and Writer posted this morning a summary of the state of the visions about Writer Views with an encouraging comment :

“I hope it gives you some understanding why such a feature is quite some work to do and what must be done in Writer before we could even start. I agree with everybody here that this is an important feature and so does the whole team. This is one of the bigger features that we will try to implement as soon as some resources will be available”.

As he says : What users call a “View” in Writer is what the developers call a “Layout” – the orientation and positioning of the textual and non-textual content on an output device. The outline mode would be one of those views.

What Mathias summarized about why there should be an Openoffice Writer outline mode :

• “Brainstorming” the structure of a document to create initial hierarchy
• Easy tool for developing and changing document structure
• Prioritize, arrange and rearrange ideas hierarchical; add details later
• Focus on content, no layout should distract from content
• Chose level of details visible in any part of the document

The current state of the proposal about what an Openoffice Writer outline mode should do :

• Present structure of a document (paragraphs, chapters, sections)
• Text indentations created from level of structural element
• Normal text should be displayed below its heading
• No margins
• No page breaks visible
• No preferred way of text wrapping; open for discussions
• No display of page bound elements (header/footer, objects anchored at a page)
• No preferred way of treating any non-textual content; why not display it?
• No preferred way of treating formatting; why not display it?
• Additional control elements that allow to promote/demote paragraphs, fold/unfold structural elements
• Creating, moving and deleting structural elements by keyboard commands or D&D

But implementing this feature will not be a trivial endeavour. Some important preliminary infrastructural work is required :

“There is a particular problem in Writer that needs to be solved before it makes sense to implement more views. A Writer documents always has one layout. If the user switches from “Print Layout” to “Online Layout” the old layout is thrown away and the new layout for the complete document is calculated. On switching back the same happens again. This can become quite annoying when new layouts are used that let switching between layouts happen more often. Perhaps it might also be attractive to have two different layouts visible at a time in two different windows, e.g. Outline Layout and Print Layout. [..] So we should investigate first if we can change the code in a way that it can handle more than one Layout at a time. This will make the implementation of new layouts better and their usage more attractive”.

Multiple simultaneous views ! Not only did the OO team listen, but their ambitions go beyond the requests. Of course, acknowledging the requirements is only a first step, but it is an essential one and I am glad that it has been taken.

Mathias prudently added :

“I want to make clear that my comment wasn’t a promise that we start to work on this immediately – we are just busy with other also important things (bug fixing, ODF support, OOXML filter etc.). But I wanted to let you know that the whole Writer team agrees with you that the Outline View is one of the most important missing features in Writer. Unfortunately it is quite some work to do, especially if you don’t want to just hack the feature but develop an improved Writer view concept. So my plan is to implement the necessary preconditions mentioned in the wiki as soon as time will permit and then start writing the specs. ATM I can’t tell when this will happen, so please be patient with us”.

If you want to be informed as soon as this issue moves you can subscribe to Openoffice Writer’s issue 3959. If you can help in any way, please be sure to leave a note about it !

This is certainly a classic bit of regex wizardry but since it took me a few minutes of searching and can be valuable in a variety of contexts, it might be valuable to you too…

grep -o ‘[[:alnum:]+\.\_\-]*@[[:alnum:]+\.\_\-]*’

I needed it for extracting the adresses returning a 550 from my Postfix logs. But then I found that Sympa, my mailing list management system, handles bounces automatically very well using a scoring algorithm that the list administrator can optionally override.

We shall call this process “serendipitous ignorance“…

While we are trying to make sense of regular expressions, those curious about them and wishing for an introduction geared toward audiences other than the beard and sandals systems administration crew may appreciate the examples provided in “Egrep for Linguists“.

And yes, I do indulge in sandals and facial pilosity in the hope of mastering regexes one day…

I usually shoot a football game using two cameras. When I publish a gallery I want images from both cameras to appear as a single stream. Of course that is easily done by using a naming scheme that puts an alphanumerically sortable timestamp right after the event’s invariant name. But that only works if both cameras have been synchronized before the event starts… Of course I forgot to do it. No problem – Exiv2 comes to the rescue ! A simple ‘exiv2 ad -a 48:08:30 *’ saved my day (actually two days, eight minutes and thirty seconds)… Thank you Exiv2 !

IPTC is the industry standard for photographic metadata so that is what you should use. As usual, a little shell scripting makes life easier so let me introduce you to the comprehensive answer to all your EXIF and IPTC metadata manipulation needs : Exiv2. Using it I whipped up a trivial, quick and dirty way to tag a bunch of files with the generic IPTC metadata of your choice. Edit the script to your taste with your own data and you are set.

Yes it is most trivial, but it is news to me and it is all I need… So there – scratching your own itch and all that…

Last but not the least : be aware that Gimp will rudely overwrite IPTC metadata with no warning and no point. The developers are aware of that and fixes will be forthcoming in an undefined future release. So along your digital photography workflow make sure that you tag your batch of pictures downstream from editing them in Gimp…

OneTouchUbuntuAutomountedCFDumpToJournal.sh is another trivial script I wrote that saves me much manipulation each time I come back to my workstation with removable media full of photos. It copies all images from a removable media to the directory of the day (created on the fly if not existing), autorotates them and sets the permissions right. It is what I use prior to putting the pictures in an appropriately named directory and running dir_date_serial_rename_all.sh to name them according to my standard.

It does about the same thing as OneTouchPhotoDumpToJournal.sh but it does not handles the mounting and unmounting because it assumes a removable medium that Ubuntu mounts automatically. As a bonus there is a very slight addition of polish. I should backport some of the polish to OneTouchPhotoDumpToJournal.sh, but since I no longer use it and received no feedback about it my motivation is quite low.

Go_awstats.sh completely automates the trivial yet tedious task of Awstats batch Apache log reports production, and it even does it somewhat smartly – I’m talking about the “only rebuilding what needs to be rebuilt” part, not the code itself which is plain stupid. This script is in production on the host of Serendipitous Altruism as the sample output testifies.

This release corrects a major year calculation error that appeared when the logs began to span more than two years. I am now confident that the whole thing works, hence the highly symbolic bump up to revision 1.0

I believe that Awstats is still the best HTTPD standard log reporting program, and this is why I am very interested in making it work automatically and painlessly.

Coldforged mentioned that Lilina “silently failed on several malformed feeds“. Coldforged adds “that wouldn’t be so bad if it survived such indignities but it didn’t. Instead my entire aggregation page would simply not load and the only notice I ever got that something was awry was the PHP error_log in the aggregator directory“. I’ll look out for such behavior, but so far I have had no problems with the few feeds I am using.

Were I looking for something with more features than Lilina I would probably turn to Gregarius with the Lilina theme because I like Lilina’s way so much. And the “RSS View” plugin apparently provides the outgoing something like the aggregated RSS feed that I implemented in Lilina by splicing Feedcreator in.

But for now I’ll stick with Lilina because it is extremely easy to use. The lack of need for a database makes deployment as simple as copying files, setting permissions (the archive of the patched Lilina I provide now features a script to do it for you) and editing a couple of lines in a configuration file. That is hard to beat.

Aggregated RSS feeds presented as HTML and Javascript by Lilina are very sweet. The more we used them, the more we missed having them served as RSS. After much research it seemed to us that there is no nice and easy PHP code capable of mixing RSS as RSS. There are plenty of feed mixers offered as a service but very few offered as a product.

On the fetching and parsing side, Lilina had everything I wanted. All I needed was to make it generate RSS instead of HTML.

I went foraging for RSS creation libraries. The first one I found was XML-RSS-Aggregate . I liked it because the example provided with XML-RSS-Aggregate is an RSS agregator that ouptuts RSS – exactly what I was looking for. But Shlomi Fish mentioned that “this module is unmaintained and no longer works very well. The author (and I) recommend that you use XML::Feed now“. So I took a look at XML-Feed and found it too complex for my meagre skills. And I’m not that hot with Perl anyway. So I went looking somewhere else.

I found my salvation in Feedcreator. Feedcreator creates valid feeds in various formats, features configurable caching, reasonnable documentation and readable code. I found it quite easy to use. All it needs is an array of RSS elements, and that is exactly what Lilina provides.

I took Lilina’s index.php, cleaned up the HTML generation, spliced in the example code from Feedcreator, mapped input to output and lo and behold I had a reasonably valid RSS output by Lilina. Very sweet !

Source code of the modified Lilina with Feedreactor hybridation is available here.

I even added a cute RSS icon to Lilina’s default layout…

A while ago I built a Jabber presence indicator on my personal web page by customizing one of the examples provided with class.jabber.php. I have received enthusiastic feedback from a handful of users – that makes me feel warm inside ! Do not neglect the ego of people who provide free services, be they Free software developpers or volunteers for anything : a kind comment or a nice stroke in the right direction go a long way toward feeding their motivation. So next time you enjoy the produce of the gift economy be sure to let people know how much you liked it !

With the kind words came a question which made me aware that a newcomer may not find the required configuration as obvious as I do. One of the least obvious steps is telling the script whose status it is supposed to check. Don’t look for a JID parameter inside the presence indicator script : the presence indicator script itself knows nothing about the user whose status it checks. It is the watcher bot user that has the watched user on his roster. The presence indicator script will return the status of anyone on the watcher bot user’s roster. That makes it quite flexible because it will work for any number of users.

If anything remains unclear to you, please be sure to report it to me.

A month ago I published lbdb-fetchalladdresses-firsttime.sh and lbdb-fetchalladdresses-daily.sh, two scripts that scour a maildir hierarchy to records all addresses from the headers in the lbdb m_inmail database in order to feed whole maildirs to LBDB for Mutt’s address completion.

A month of running these scripts has revealed a few problems wich are now corrected with a speed optimization thrown in for good measure :

• lbdb-fetchalladdresses-firsttime.sh
• lbdb-fetchalladdresses-daily.sh

As I mentioned earlier, if you do not have an existing lbdb database, you should run lbdb-fetchalladdresses-firsttime.sh first. Once you have done that you can from time to time update your database with lbdb-fetchalladdresses-daily.sh who does the same thing as lbdb-fetchalladdresses-firsttime.sh but only for messages not older than a certain user-set age.

One of the problems with my Jabber presence indicator on a web page was that the page’s presence was being sent to the polled account instead of being hidden. I fixed that and the updated code is available. The fix was trivial :

29c29
< $JABBER->SendPresence();
—
> $JABBER->SendPresence(“invisible”);

Although widely implemented (even Edgar does it that way), the “invisible” presence type is not XMPP compliant. JEP 126 explains how to provide selective visibility in a XMPP compliant manner. Of course this is less trivial. But since JEP 126 provides the appropriate XML snippets, I could probably do it the right way using SendPacket($xml) to send the raw XML. Maybe next time… Meanwhile I reported the standard compliance issue on the Edgar bug tracker.

I now rely on Jabber to publish my presence. Not everyone has a Jabber client at hand but Web access is about ubiquitous so it makes sense to me to make my presence available on a Web page. To me this is a personnal itch and an excuse to practice some PHP and learn a little about the exciting world of XMPP.

The source code for my presence indicator is now available for whoever wants it. It is made possible by class.jabber.php, a Jabber library for PHP. I customized one of the examples provided with class.jabber.php. It has many rough edges such as the user’s name sprinkled across the markup code instead of being a proper parameter, or the page’s presence being sent to the polled account instead of being hidden. And it is quite slow because it queries at load time. But I like it better than the example packaged with class.jabber.php.

It basically does the job and I am therefore quite sure that there will be interested users. So this is the “early” part of “Release Early, Release Often”. Take that hack, customize it, tweak it and generally improve it… And most of all please report back somewhere so that everyone can benefit. Here would be a good start.

When I type the beginning of a name in Mutt, pressing CTRL-T will give me a list of addresses whose real name matches my entry, and I just have to choose the one I want. As a result I seldom type more than a few letters to enter a complete address. This trick is called an external address query which is the mechanism by which Mutt supports connecting to external directory databases through a wrapper script.

The wrapper script I used is called lbdbq and it is part of lbdb. Lbdb provides an abstraction layer to a wide variety of address sources. The one I use most is the m_inmail database built by lbdb-fetchaddr. lbdb-fetchaddr reads a mail on stdin and extracts adresses from the mail header to append them to $HOME/.lbdb/m_inmail.list. Thanks to lbdb-fetchaddr, if we ever exchanged mail I can completion your name to you address.

There are two ways to feed the m_inmail database. The first one is to do it as you go using a MDA filter that pipes a copy of incoming messages to lbdb-fetchaddr. I copied the maildrop recipe from Mark Weinem’s .mailfilter example :

if ( $SIZE < 10000 )
cc "|lbdb-fetchaddr"

As you may have guessed the filter excludes messages larger than 10 KB because lbdb-fetchaddr is not supposed to work with “big messages”. But Mark Weinem’s .mailfilter example does not provide any further explanation and I found none myself. What I did find is that this rule does from time to time cause deferals after maildrop delivery failures for a reason I have yet to discover. Until I find that reason I will refrain from using it.

So I found another way to do it : feed lbdb-fetchaddr my 4 GB of mail dating back from 2001 (the moment when a disk crash taught me the value of good backups). This method has the added value of taking advantage of historical data, not just future traffic. On mutt-users, Jason Helfman asked “if anyone out there has any success in piping their inbox to lbdb-fetchaddr” and got no answer. Indeed, lbdb-fetchaddr does not work like that : each message must be fed individually. Of course I had to automate the process so that all my maildirs can be processed in one go, so I wrote a couple of small scripts :

If you do not have an existing lbdb database, you should run lbdb-fetchalladdresses-firsttime.sh first : it scours a maildir hierarchy to records all addresses from the headers in the lbdb m_inmail database. Once you have done that you can from time to time update your database with lbdb-fetchalladdresses-daily.sh who does the same thing as lbdb-fetchalladdresses-firsttime.sh but only for messages not older than a certain user-set age.

For performance reasons lbdb-fetchaddr appends new addresses to the database without removing duplicates – duplicates are only removed at query time. For 4 GB of mail, lbdb-fetchaddr produces a 6 MB m_inmail file which piping through ‘uniq‘ reduces to a mere 300 KB. Since the process is normally a one-off, I believe it is well worth the transient load. The update script only deals with a few additional entries, so the ‘uniq‘ load is negligible especially since this is likely to be a nightly cron job.

Don’t forget to install the lbdb package before and to add the following line to your muttrc :

'set query_command = "lbdbq '%s'

So there you go : happy CTRL-T completion !