Ever since the Linux Advanced Routing & Shaping HOWTO introduced it, I have been a big fan of the Wondershaper, a traffic shaping script that drives Linux’s class based queuing with stochastic fairness queuing (SFQ) in a pretty effective attempt at maintaining low latency for interactive traffic while at the same time maintaining high throughput. There is even a ‘wondershaper’ Debian package that includes some additional polish. This script is key to the joy of perfectly responsive SSH sessions while peer to peer file sharing traffic saturates the uplink.

Some people have even concluded the resulting quality of service is good enough for voice traffic. But even with the Debian Wondershaper ruling my ADSL link I noticed that SIP and IAX still suffer too much packet loss with the saturating traffic occupying the background. I needed better traffic control.

As usual, being a late adopter I am not the only one to have hit that obstacle, and solutions have already been put forth. After rummaging through various mutations, I found Robert Koch’s version of the Wondershaper for the Asus WL-xxx documented on the Wondershaper package page of the WL-500G wiki to be quite promising. Compared to the standard version it prioritizes VOIP traffic by source port for idiot proof configuration, but also by type of service which is much more flexible and can be used thanks to Asterisk being capable of correctly setting TOS fields. As a bonus, using TOS also makes this version of the script capable of distinction between console interactive SSH traffic and bulk SCP traffic using the same protocol and port. And to top it all, it is based on the better hierarchical token bucket (HTB) discipline which is standard since Linux 2.4.20 while the Debian Wondershaper version uses the more based queuing which used to be the more widespread one.

The first shortcoming I found is that it prioritizes SIP and RTP but not IAX and others which I’ll have to add using the SIP stanzas as templates. The other is that taking lists of low priority ports as arguments could make the command line messy and configuration puzzling for the inexperienced user, so I prefer to have this configuration item as a documented variable allocation inside the script. But those are trifles compared to the new VOIP support, enhanced SSH discrimination and overall upgrade.

Hacking on the script I couldn’t resist reorganizing a few things. I originally intended to provide a diff, but that would be pointless since I ended up touching most of the lines. Also be warned that I do not understand why putting ‘prio 1′ everywhere makes the script work whereas other ‘prio’ values at various places made traffic end up in the wrong class and did not make sense at all. In effect, I think that by putting ‘prio 1′ everywhere I just eschewed the use of priority bands inside the classes, which is just fine with me for the intended use. But this show that my tc fluency is still limited and that there are therefore surely ways to enhance this script. I’ll also welcome feedback - whether it works for you or not.

Anyway - it works ! I had a few VOIP conversations across an IAX trunk with lots of background traffic on the uplink and no perceptible effects on voice quality. Life is good. Now that I have removed the last obstacle to taking full advantage of VOIP at home. Soon all my traffic will be routed through Asterisk and there shall be no more RJ11 nor their French T-sockets alter ego in my home.

Here is my modified wondershaper script in all its glory - contrary to Robert Koch’s version it is a drop-in replacement for Debian’s package. Inheriting from the original Wondershaper it is licensed under the GPL so enjoy, modify and share !

Now that in its 2.0 incarnation Ejabberd supports publish-subscribe and therefore personal eventing, it is time to play with it and demonstrate to the wider world the marvellous use-cases that the future holds. A nice first one that should be popular and therefore useful for propaganda purposes is using Psi so that contacts can see in your presence status the music that you are playing. I stumbled upon an Amarok script that notifies Psi’s through the tune file interface and lets Psi publish the currently playing song status via PEP - and it looked good.

PEP is defined “XEP-0163: Personal Eventing via Pubsub“. And Pubsub is defined by “XEP-0060: Publish-Subscribe“. So far so good. But digging around a bit I learned about “XEP-0118: User Tune” and then it dawned on me that there appeared to be room for improvement : the script outputs a composite “tune” element which is a radical simplification of the schema specified in XEP-0118.

So I had a go at modifying the script to get it as close to the specification as possible. You can judge of the resulting output for yourself : not quite XEP-0118 compliant but a good step in that direction.

The source code is available from the usual dump, but if you are an Amarok and Psi user you might actually want to use the Amarok script package that installs and runs in a coupe of clicks - thanks to the previous authors whose work I built upon.

While I was at it I discovered a bug that causes Psi 0.11 to use the element tag “source” to contain the album information, so I promptly provided the psi project team with the trivial patch needed.

It is is 3:30 AM and a few hours ago I did not realize that upgrading Ejabberd would get me that far for today…

As far as I have looked, is no working FQL console application (I just tested the four FQL consoles that are published in the applications directory on Facebook but they either don’t load or crash on query). Although Facebook mentions that one is supposed to exist in the “Tools” page, there is actually none there at the moment. I guess I’ll have to build a small PHP application for playing with FQL.

My immediate practical goal is to be able to select members of two different groups. The query should be something like ‘SELECT uid FROM group_member WHERE gid=my_gid AND uid in (SELECT uid FROM group_member WHERE gid=my_other_gid)’ - for example to cross special interest groups or geographical areas.

There is plenty of potential for useful data mining that is not exposed by Facebook’s default interface. Search with multiple criteria of the same category is an obvious need for finding interesting people. Maybe did Facebook decide that the cost of additional clutter was not worth it for the average user. Or maybe they would prefer that the users don’t realize how much information can emerge from mining their data…

Since I began playing with Net_SmartIRC, I found a new way to put that library to work : a Munin plugin script to monitor the number of users in an IRC channel.

Here is an example of the graphical output provided by Munin :

As you can see, the Debian IRC channel is a very crowded place ! You may also notice small gaps in the data : the script sometimes fails on a refused connection, and I have not elucidated the cause. But as the graph shows, I have coded the script so that those failure cases only result in a null output, which Munin handles well by showing a blank record.

Because my lacking skills and crass lazyness prevented me from writing it all in a single language, I hacked that plugin by simply patching together the parts I could produce rapidly :

• irc_channel_users is a Munin plugin script that calls irc_channel_users.php

• irc_channel_users.php performs the actual login and count on the IRC server.

The PHP script is uses Net_SmartIRC which is available in Debian as php-net-smartirc. It must be configured by modifying the hardcoded server and channel - that may not be what is best in production use, but for the moment it works for me. Here is the full extent of the PHP code :

< ?php
include_once('/usr/share/php/Net/SmartIRC.php');
$irc = &new Net_SmartIRC();
//$irc->setDebug(SMARTIRC_DEBUG_ALL);
$irc->setUseSockets(TRUE);
$irc->setBenchmark(TRUE);
$irc->connect('irc.eu.freenode.net', 6667);
$irc->login('usercount', 'Users counting service for Munin monitoring',
'0', 'usercount');
$irc->getList('#test_channel');
$resultar = $irc->objListenFor(SMARTIRC_TYPE_LIST);
$irc->disconnect();
if (is_array($resultar)) {
    echo $resultar[0]->rawmessageex[4];
} else {
}
?>

The irc_channel_users Bash script is also quite simple. Apart from the barely modified boilerplate adapted from other simple Munin bash scripts, the specific meat of the script is as follow :

work_directory=/home/jim/applications/munin/irc_channel_users
php_interpreter=`which php`
user_population=`$php_interpreter $work_directory/irc_channel_users.php
 | awk -F"#" '{print($1)}' | grep -e '^[0-9]+$'`
echo -n "population.value "
echo $user_population

As you can see, the munin bash script is mostly about setting a few Munin variables, calling the php script and formatting the output.

Here are sample outputs :

15:32 munin@kivu /etc/munin/plugins% ./irc_channel_users autoconf
yes

15:32 munin@kivu /etc/munin/plugins% ./irc_channel_users config
graph_title #b^2 IRC channel users
graph_args --base 1000 -l 0
graph_vlabel population
graph_scale no
population.label users

15:32 munin@kivu /etc/munin/plugins% ./irc_channel_users
population.value 6

No demonstration is available on a public site, but the above graph is about all there is to know about the output of this plugin.

The code resides on its own page and updates if they ever appear shall be stored there.

This experience taught me that coding basic Munin plugins is fun and easy. I will certainly come back to it for future automated graphing needs.

And for those who wonder about the new syntax highlighting, it is produced using GeSHi by Ryan McGeary’s very nice WP-Syntax Wordpress plugin.

Since my migration to PHP 5 revealed a problem with the ancient Lilina 0.7 my interest in Lilina has been rekindled. As I said, I am quite hopeful because I would like to keep using Lilina for small aggregations and avoid deploying the more complex Gregarius where its better scalability is not needed. What first attracted me toward Lilina still holds true.

I started by checking out the development version from Google Code and I deployed it on a handful of small aggregations such as my personal feed. I then immediately started gently pestering the project lead about a few bugs, first by mail and then using the nice Google Code Lilina issue tracker. Ryan Mc Cue has been very responsive and very nice. I look forward making my small contributions in helping him bring Lilina toward 1.0

A year has passed since Ryan announced his intentions. He has already done a lot but his roadmap for Lilina 1.0 is very ambitious so there is still a lot of work remaining. At the current pace it may take a few more months, but this is a journey I am eager to follow : I like Lilina and since I am a novice PHP coder I am quite happy whenever I find a project I like and to which I can give a little help. I am actually quite proud of being mentioned in the credits for having brought RSS output to Lilina : it was the first time I had some of my code included in a significant project.

A few old friends and I often hang together on our usual IRC channel, and while discussing we like to share links. I thought it might be a good idea to use a web tool for that so that we can also share our favorite links with the rest of the world. I thought a private Digg clone could play that role, and I needed an excuse for more PHP development. After some research I found that Pligg was the best candidate. I downloaded Beta 9.7 and I have been quite amazed how fast I got a smooth Pligg site up and running - it seems like much work went into this code.

The first bug that arose was mangled accents in some URLs. I solved it by adding a few cases to a character substitution matrix. This matrix is used to sanitize strings before using them for URLs. The problem is that this method requires each accented character to be treated individually. Maybe a more general solution exists. Anyway I am surprised that no other user noticed that problem with French accents before.

Since I wanted to use Pligg in a way closely associated with our IRC habit, I thought it might be a good idea to make Pligg capable of notifying an IRC channel of an upcoming story. Rumaging through PHP libraries capable of speaking IRC I ended up using Net_SmartIRC which is available in Debian as php-net-smartirc. The examples packaged with the class are quite good at teaching its use so I soon wrote a small proof-of-concept of notifying an IRC channel from PHP.

Some more hacking later I figured how to splice that IRC notification functionality into Pligg. This is a prototype : it works fine for me but you need to overwrite a file from the standard Pligg and edit it with your parameters, and it probably ignores any convention used by those who are familiar with working on Pligg. To reach release quality this functionality should probably be architectured as a proper configurable Pligg plugin. To use it, you need to install Net_SmartIRC which is available in Debian as php-net-smartirc. Make sure that the path of that library as declared in my submit.php is correct.Then you need to edit my submit.php to set connect parameters, login parameters, and the channel. Make a backup of your original submit.php and replace it with mine. That is all !

With IRC notification in place I found that I did not want just anyone spamming our favorite IRC channel. Pligg has three hardcoded user levels : “user”, “admin” and “god”. By default the users can submit posts. I decided to restrict posting to the “admin” and “god” levels. That way we keep control of posting, our friends can vote and comment as “user” and the rest of the anonymous world can read the site.

With the beginning of a familiarity with the Pligg code it did not take me too much grepping to understand the conditional access system and wedge a typical implementation where it should work. And work it did, just as advertised ! I am not sure that the core Pligg developers intended this previously clean switch statement to be hacked like that, but it works for me. A further step would be to make this restriction code conditional and to make the condition a user-configurable parameter in /admin_config.php so that god can choose at which user level submissions are allowed. I am sure that many site operators would appreciate this feature.

So as a summary and conclusion, here is the patch that modifies Pligg’s 9.7 submit.php to provide post submission restriction by user level and IRC notification of posts :

• The original submit.php from Pligg 9.7
• submit.php from Pligg 9.7 patched to provide post restriction by user level and IRC notification of post
• diff between the above files.

Have fun with Pligg ! I’m done with it for now but who knows what other functions I’ll feel like hacking into it in the future…

After migrating an host to PHP5 I found that Lilina 0.7 no longer works and instead produces the following error :

PHP Fatal error: Cannot redeclare class soapclient in /your-lilina-directory/inc/nusoap.php on line 4096

Happily, Robert Mao at “Inmates are Running the Asylum” had already stumbled on this and found a solution.

Robert found a report of the Nusoap library conflicting with PHP5’s built in SOAP functions. The only use of Nusoap in Lilina is the Google API. So Robert found that by disabling the peripheral functionality dependant on the Google API Lilina no longer produced the error.

It works but it is a quick and dirty fix. Enters Ryan Mc Cue who took over Lilina’s development last year. Ryan soon mentioned that the aforementioned functionality is completely disabled in the current development version of Lilina which therefore works fine with PHP5.

There has not been a release of Lilina in quite a while but indeed Ryan and his friends have not been idle and on top of a Brand new web site there have been many commits to the Lilina Subversion repository on Google Code.

So Lilina 1.0 is in development and I’m going to take look at it. I am quite hopeful because I would like to keep using Lilina for small aggregations and avoid deploying the more complex Gregarius where its better scalability is not needed.

My quick and easy tool for renaming files straight from camera in a way that will make sense in most contexts is now even better.

Changelog :

• Field order has been changed in order to fit even more situations
• The script now tolerates upper and lower case file name extensions
• Prefix can be personalized : your initials probaly make more sense than “IMG”
• Extension case set is now set according to user configuration

Example :

“my current directory/img_6051.jpg” taken the 2nd of December 2005 at 9:07:59 AM (according to the embedded EXIF metadata)

becomes :

“my current directory/20051202.090759.JML.6051.my_current_directory.jpg”

Set your personal parameters in five seconds, and call just one simple command with no arguments for the whole directory. How easier can it get ?

See it, download it !

My last rant about Openoffice’s lack of a proper outline mode apparently struck a chord if I judge from the number of pageviews and the reactions I gathered. If, like me, you eagerly await this functionality you will be happy to learn that some recent activity around Openoffice Writer’s longstanding issue 3959 aka “Outline View (aka MS Word)” has provided us with some hope.

Mathias Bauer, project lead of the OpenOffice.org Application Framework and manager of the teams for the application framework, Math and Writer posted this morning a summary of the state of the visions about Writer Views with an encouraging comment :

“I hope it gives you some understanding why such a feature is quite some work to do and what must be done in Writer before we could even start. I agree with everybody here that this is an important feature and so does the whole team. This is one of the bigger features that we will try to implement as soon as some resources will be available”.

As he says : What users call a “View” in Writer is what the developers call a “Layout” - the orientation and positioning of the textual and non-textual content on an output device. The outline mode would be one of those views.

What Mathias summarized about why there should be an Openoffice Writer outline mode :

• “Brainstorming” the structure of a document to create initial hierarchy
• Easy tool for developing and changing document structure
• Prioritize, arrange and rearrange ideas hierarchical; add details later
• Focus on content, no layout should distract from content
• Chose level of details visible in any part of the document

The current state of the proposal about what an Openoffice Writer outline mode should do :

• Present structure of a document (paragraphs, chapters, sections)
• Text indentations created from level of structural element
• Normal text should be displayed below its heading
• No margins
• No page breaks visible
• No preferred way of text wrapping; open for discussions
• No display of page bound elements (header/footer, objects anchored at a page)
• No preferred way of treating any non-textual content; why not display it?
• No preferred way of treating formatting; why not display it?
• Additional control elements that allow to promote/demote paragraphs, fold/unfold structural elements
• Creating, moving and deleting structural elements by keyboard commands or D&D

But implementing this feature will not be a trivial endeavour. Some important preliminary infrastructural work is required :

“There is a particular problem in Writer that needs to be solved before it makes sense to implement more views. A Writer documents always has one layout. If the user switches from “Print Layout” to “Online Layout” the old layout is thrown away and the new layout for the complete document is calculated. On switching back the same happens again. This can become quite annoying when new layouts are used that let switching between layouts happen more often. Perhaps it might also be attractive to have two different layouts visible at a time in two different windows, e.g. Outline Layout and Print Layout. [..] So we should investigate first if we can change the code in a way that it can handle more than one Layout at a time. This will make the implementation of new layouts better and their usage more attractive”.

Multiple simultaneous views ! Not only did the OO team listen, but their ambitions go beyond the requests. Of course, acknowledging the requirements is only a first step, but it is an essential one and I am glad that it has been taken.

Mathias prudently added :

“I want to make clear that my comment wasn’t a promise that we start to work on this immediately - we are just busy with other also important things (bug fixing, ODF support, OOXML filter etc.). But I wanted to let you know that the whole Writer team agrees with you that the Outline View is one of the most important missing features in Writer. Unfortunately it is quite some work to do, especially if you don’t want to just hack the feature but develop an improved Writer view concept. So my plan is to implement the necessary preconditions mentioned in the wiki as soon as time will permit and then start writing the specs. ATM I can’t tell when this will happen, so please be patient with us”.

If you want to be informed as soon as this issue moves you can subscribe to Openoffice Writer’s issue 3959. If you can help in any way, please be sure to leave a note about it !

This is certainly a classic bit of regex wizardry but since it took me a few minutes of searching and can be valuable in a variety of contexts, it might be valuable to you too…

grep -o ‘[[:alnum:]+\.\_\-]*@[[:alnum:]+\.\_\-]*’

I needed it for extracting the adresses returning a 550 from my Postfix logs. But then I found that Sympa, my mailing list management system, handles bounces automatically very well using a scoring algorithm that the list administrator can optionally override.

We shall call this process “serendipitous ignorance“…

While we are trying to make sense of regular expressions, those curious about them and wishing for an introduction geared toward audiences other than the beard and sandals systems administration crew may appreciate the examples provided in “Egrep for Linguists“.

And yes, I do indulge in sandals and facial pilosity in the hope of mastering regexes one day…

I usually shoot a football game using two cameras. When I publish a gallery I want images from both cameras to appear as a single stream. Of course that is easily done by using a naming scheme that puts an alphanumerically sortable timestamp right after the event’s invariant name. But that only works if both cameras have been synchronized before the event starts… Of course I forgot to do it. No problem - Exiv2 comes to the rescue ! A simple ‘exiv2 ad -a 48:08:30 *’ saved my day (actually two days, eight minutes and thirty seconds)… Thank you Exiv2 !

IPTC is the industry standard for photographic metadata so that is what you should use. As usual, a little shell scripting makes life easier so let me introduce you to the comprehensive answer to all your EXIF and IPTC metadata manipulation needs : Exiv2. Using it I whipped up a trivial, quick and dirty way to tag a bunch of files with the generic IPTC metadata of your choice. Edit the script to your taste with your own data and you are set.

Yes it is most trivial, but it is news to me and it is all I need… So there - scratching your own itch and all that…

Last but not the least : be aware that Gimp will rudely overwrite IPTC metadata with no warning and no point. The developers are aware of that and fixes will be forthcoming in an undefined future release. So along your digital photography workflow make sure that you tag your batch of pictures downstream from editing them in Gimp…

OneTouchUbuntuAutomountedCFDumpToJournal.sh is another trivial script I wrote that saves me much manipulation each time I come back to my workstation with removable media full of photos. It copies all images from a removable media to the directory of the day (created on the fly if not existing), autorotates them and sets the permissions right. It is what I use prior to putting the pictures in an appropriately named directory and running dir_date_serial_rename_all.sh to name them according to my standard.

It does about the same thing as OneTouchPhotoDumpToJournal.sh but it does not handles the mounting and unmounting because it assumes a removable medium that Ubuntu mounts automatically. As a bonus there is a very slight addition of polish. I should backport some of the polish to OneTouchPhotoDumpToJournal.sh, but since I no longer use it and received no feedback about it my motivation is quite low.

Go_awstats.sh completely automates the trivial yet tedious task of Awstats batch Apache log reports production, and it even does it somewhat smartly - I’m talking about the “only rebuilding what needs to be rebuilt” part, not the code itself which is plain stupid. This script is in production on the host of Serendipitous Altruism as the sample output testifies.

This release corrects a major year calculation error that appeared when the logs began to span more than two years. I am now confident that the whole thing works, hence the highly symbolic bump up to revision 1.0

I believe that Awstats is still the best HTTPD standard log reporting program, and this is why I am very interested in making it work automatically and painlessly.

Coldforged mentioned that Lilina “silently failed on several malformed feeds“. Coldforged adds “that wouldn’t be so bad if it survived such indignities but it didn’t. Instead my entire aggregation page would simply not load and the only notice I ever got that something was awry was the PHP error_log in the aggregator directory“. I’ll look out for such behavior, but so far I have had no problems with the few feeds I am using.

Were I looking for something with more features than Lilina I would probably turn to Gregarius with the Lilina theme because I like Lilina’s way so much. And the “RSS View” plugin apparently provides the outgoing something like the aggregated RSS feed that I implemented in Lilina by splicing Feedcreator in.

But for now I’ll stick with Lilina because it is extremely easy to use. The lack of need for a database makes deployment as simple as copying files, setting permissions (the archive of the patched Lilina I provide now features a script to do it for you) and editing a couple of lines in a configuration file. That is hard to beat.

Aggregated RSS feeds presented as HTML and Javascript by Lilina are very sweet. The more we used them, the more we missed having them served as RSS. After much research it seemed to us that there is no nice and easy PHP code capable of mixing RSS as RSS. There are plenty of feed mixers offered as a service but very few offered as a product.

On the fetching and parsing side, Lilina had everything I wanted. All I needed was to make it generate RSS instead of HTML.

I went foraging for RSS creation libraries. The first one I found was XML-RSS-Aggregate . I liked it because the example provided with XML-RSS-Aggregate is an RSS agregator that ouptuts RSS - exactly what I was looking for. But Shlomi Fish mentioned that “this module is unmaintained and no longer works very well. The author (and I) recommend that you use XML::Feed now“. So I took a look at XML-Feed and found it too complex for my meagre skills. And I’m not that hot with Perl anyway. So I went looking somewhere else.

I found my salvation in Feedcreator. Feedcreator creates valid feeds in various formats, features configurable caching, reasonnable documentation and readable code. I found it quite easy to use. All it needs is an array of RSS elements, and that is exactly what Lilina provides.

I took Lilina’s index.php, cleaned up the HTML generation, spliced in the example code from Feedcreator, mapped input to output and lo and behold I had a reasonably valid RSS output by Lilina. Very sweet !

Source code of the modified Lilina with Feedreactor hybridation is available here.

I even added a cute RSS icon to Lilina’s default layout…