Unix archived articles

Subscribe to the RSS feed for this category only

Networking & telecommunications and Systems administration and Unix06 Jun 2012 at 11:48 by Jean-Marc Liotier

Today is IPv6 party time so let’s celebrate with a blog post !

Reliable IPV6 connectivity is no longer just nice to have – it is a necessity. If your Internet access provider still does not offer proper native IPv6 connectivity, your next best choice is to use an IPv4 tunnel to an IPv6 point of presence. It works and on the client side it only requires this sort of declaration in /etc/network/interfaces :

auto ipv6-tunnel-he
  iface ipv6-tunnel-he inet6 v4tunnel
  address 2001:170:1f12:425::2
  netmask 64
  endpoint 216.66.84.42
  gateway 2001:170:1f12:425::1

Of course, the same sort of configuration is required at the other endpoint – which means that, among other parameters, you must inform the IPv6 tunnel server of the IPv4 address of the client endpoint. Hurricane Electric, my tunnel broker lets me do that manually through its web interface – which is fine for a static configuration done once, but inadequate if your Internet access provider won’t supply you with a static IPv4 address. By the way, even if, after a few weeks of use, you believe you have a static address, you might just have a dynamic address with a rather long DHCP lease…

But Hurricane Electric also provides a primitive HTTP API that lets you inform the tunnel broker of IPv4 address changes – that is all we need to do it automatically every time our Internet access goes up. Adding this wget command to the uplink configuration stanza in /etc/network/interfaces does the trick :

auto eth3
iface eth3 inet dhcp
  up wget -O /dev/null https://USERNAME:PASSWORD@ipv4.tunnelbroker.net/ipv4_end.php?tid=34764

That’s it – you now can count on IPv6 connectivity, even after a dynamic IPv4 address change.

And after you are done, go test your IPv6 configuration and your IPv6 throughput !

Debian and Networking & telecommunications and Systems administration and Unix17 Oct 2011 at 11:03 by Jean-Marc Liotier

I just wanted to create an Apache virtual host responding to queries only over IPv6. That should have been most trivial considering that I had already been running a dual-stacked server, with all services accessible over both IPv4 and IPv6.

Following the established IPv4 practice, I set upon configuring the virtual host to respond only to queries directed to a specific IPv6 address. That is done by inserting the address in the opening of the VirtualHost stanza : <VirtualHost [2001:470:1f13:a4a::1]:80> – same as an IPv4 configuration, but with brackets around the address. It is simple and after adding an AAAA record for the name of the virtual host, it works as expected.

I should rather say it works even better than expected : all sub-domains of the second-level domain I’m using for this virtual host are now serving the same content that the new IPv6-only virtual host is supposed to serve… Ungood – cue SMS and mail from pissed-off users and a speedy rollback of the changes; the joys of cowboy administration in a tiny community-run host with no testing environment. As usual, I am not the first user to fall into the trap. Why Apache behaves that way with an IPv6-only virtual host is beyond my comprehension for now.

Leaving aside the horrible name-based hack proposed by a participant in the Sixxs thread, the solution is to give each IPv6-only virtual host his own IPv6 address. Since this server has been allocated a /64 subnet yielding him 18,446,744,073,709,551,616 addresses, that’s quite doable, especially since I can trivially get a /48 in case I need 1,208,925,819,614,629,174,706,176 more addresses. Remember when you had to fill triplicate forms and fight a host of mounted trolls to justify the use of just one extra IPv4 address ? Yes – another good reason to love IPv6 !

So let’s add an extra IPv6 address to this host – another trivial task : just create an aliased interface, like :

auto eth0:0
    iface eth0:0 inet6 static
    address 2001:470:1f13:a4a::1
    netmask 64
    gateway 2001:470:1f12:a4a::2

The result :

SIOCSIFFLAGS: Cannot assign requested address
Failed to bring up eth0:0.

This is not what we wanted… You may have done it dozens of time in IPv4, but in IPv6 your luck has ran out.

Stop the hair pulling right now : this unexpected behavior is bug – this one documented in Ubuntu, but I confirm it is also valid on my mongrel Debian system. Thanks to Ronny Roethof for pointing me in the right direction !

The solution : declare the additional address in a post-up command of the main IPv6 interface (and don’t forget to add the post-down command to kee things clean) :

auto he-ipv6
iface he-ipv6 inet6 v4tunnel
    address 2001:470:1f12:a4a::2
    netmask 64
    endpoint 216.66.84.42
    local 212.85.152.17
    gateway 2001:470:1f12:a4a::1
    ttl 64
    post-up ip -f inet6 addr add 2001:470:1f13:a4a::1 dev he-ipv6
    pre-down ip -f inet6 addr del 2001:470:1f13:a4a::1 dev he-ipv6

And now the IPv6-only virtual hosts serves as designed and the other virtual hosts are not disturbed. The world is peaceful and harmonious again – except maybe for that ugly post-up declaration in lieu of declaring an aliased interface the way the Unix gods intended.

All that just for creating an IPv6 virtual host… Systems administration or sleep ? Systems administration is more fun !

Free software and Mobile computing and Systems administration and Technology and Unix09 Aug 2011 at 11:17 by Jean-Marc Liotier

Oh noes – I’m writing about a Google product, again. The omnipresence of the big G in my daily environment is becoming a bit excessive, so I’m stepping up my vigilance about not getting dependent on their services – though I don’t mind them knowing everything about me. In that light, acquiring another Android communicator may not seem logical, but I’m afraid that it is currently the choice of reason : I would have paid pretty much any price for a halfway decent Meego device, but Nokia’s open rejection of its own offspring is just too disgusting to collude with. The Openmoko GTA04 is tempting, but it is not yet available and I need a device right now.

Android does not quite mean I have to remain attached to the Google tit : thanks to CyanogenMod there is now an Android distribution free of Google applications  – and it also offers a variety features and enhancements… Free software is so sweet !

As a bonus, CyanogenMod is also free of the hardware manufacturer’s pseudo-improvements or the carrier’s dubious customizations – those people just can’t keep themselves from mucking with software… Please keep to manufacturing hardware and providing connectivity – it is hard enough to do right that you don’t have to meddle and push software that no one wants !

So when I went shopping for a new Android device after my one year old daughter disappeared my three year-old HTC Magic, I made sure that the one I bought was compatible with CyanogenMod. I chose the Motorola Defy because it is water-resistant, somewhat rugged and quite cheap too. By the way, I bought it free from access provider SIM lock – more expensive upfront, but the era of subsidized devices is drawing to an end and I’m going to enjoy the cheaper subscriptions.

On powering-on the Defy, the first hurdle is to get past the mandatory Motoblur account creation – not only does Motorola insist on foisting its fat supplements on you, but it won’t let you access your device until you give it an email address… In case I was not already convinced that I wanted to get rid of this piece of trash, that was a nice reminder.

This Defy was saddled with some Android 2.2.2 firmware – I don’t remember the exact version. I first attempted to root it using Z4root, but found no success with that method. Then I tried with SuperOneClick and it worked, after some fooling around to find that USB debugging must not be enabled until after the Android device is connected to the PC – RTFM !  There are many Android rooting methods – try them until you find the one that works for you : there is much variety in the Android ecosystem, so your mileage may vary.

Now that I have gained control over a piece of hardware that I bought and whose usage  should therefore never have been restricted by its manufacturer in the first place, the next step is to put CyanogenMod on it. Long story short : I fumbled with transfers and Android boot loader functionalities that I don’t yet fully understand, so I failed and bricked my device. In the next installment of this adventure, I’m sure I’ll have a nice tale of success to tell you about – meanwhile this one will be a tale of recovery.

This brick situation is a Motorola Defy with blank screen and a lit white diode on its front. The normal combination of the power and volume keys won’t bring up the boot loader’s menu on start. But thanks to Motorola’s hardware restrictions designed to keep the user from modifying the software, the user is also kept from shooting himself in the foot and the Defy is only semi-bricked and therefore recoverable. Saved by Motorola’s hardware restrictions… Every cloud has a silver lining. But had the device been completely open and friendly to alien software, I would not have had to hack at it in the first place, I would not had bricked it and there would have been no need for saving the day – so down with user-hostile hardware anyway !

With the Motorola Defy USB drivers installed since the SuperOneClick rooting, I launched RSD lite 4.9 which is the Motorola utility for flashing Motorola Android devices. Here is the method for using RSD lite correctly. RSD lite immediately recognized the device connected across the USB cord. The trick was finding a suitable firmware in .sbf format. After a few unsuccessful attempts with French Android versions, I found that JRDNEM_U3_3.4.2_117-002_BLUR_SIGN_SIGNED
_USAJRDNEMARAB1B8RTGB035.0R_USAJRDNFRYORTGB_P003_A002_HWp3_Service1FF
worked fine and promptly booted me back to some factory default – seeing the dreaded Motoblur signup screen was actually a relief, who would have thought ?

After re-flashing with RSD Lite, I found that there is a Linux utility for flashing Motorola Android devices :  sbf_flash – that would have saved me from borrowing my girlfriend’s Windows laptop… But I would have needed it for SuperOneClick though – isn’t it strange that support tools for Android are Windows-dependent ?

With CyanogenMod in place, my goal will be to make my personal information management system as autonomous as possible – for example I’ll replace Google Contacts synchronization with Funambol. CyanogenMod is just the starting point of trying to make the Android system somewhat bearable – it is still the strange and very un-Unixy world of Android, but is a pragmatic candidate for mobile software freedom with opportunities wide open.

But first I have to successfully transfer it to my Android device’s flash memory… And that will be for another day.

If you need further information about hacking Android devices, great places are Droid Forums and the XDA-Developpers forum – if you don’t go directly, the results of your searches will send you there anyway.

Code and Free software and Networking & telecommunications and Systems administration and Unix01 Mar 2011 at 20:06 by Jean-Marc Liotier

I loathe Facebook and its repressive user-hostile policy that provides no value to the rest of the Web. But like that old IRC channel known by some of you, I keep an account there because some people I like & love are only there. I seldom go to Facebook unless some event, such as a comment on one of the posts that I post there through Pixelpipe, triggers a notification by mail. I would like to treat IRC that way: keeping an IRC application open and connected is difficult when mobile or when using the stupid locked-down mandatory corporate Windows workstation, and I’m keen to eliminate that attention-hogging stream from my environment – especially when an average of two people post a dozen lines a day, most of which are greetings and mealtimes notifications. But when a discussion flares up there, it is excellent discussion… And you never know when that will happen – so you need to keep an eye on the channel. Let’s delegate the watching to some automation !

So let me introduce to you to my latest short script : bipIRCnickmailnotify.sh – it sends IRC log lines by mail when a specific string is mentioned by other users. Of course in the present use case I set it up to watch for occurrences of my nickname, but I could have set it to watch any other string. The IRC logging is done by the bip IRC proxy that among other things keeps me permanently present on my IRC channels of choice and provides me with the full backlog whenever I join with a regular IRC client.

This Unix shell script also uses ‘since’ – a Unix utility similar to ‘tail’ that unlike ‘tail’ only shows the lines appended since the last execution. I’m sure that ‘since’ will come handy in the future !

So there… I no longer have to monitor IRC – bipIRCnickmailnotify.sh does it for me.

With trivial modification and the right library it could soon do XMPP notifications too – send me an instant message if my presence is ‘available’ and mail otherwise. See you next version !

Brain dump and Debian and Free software and Systems administration and Unix17 Nov 2010 at 19:54 by Jean-Marc Liotier

On Identi.ca I stumbled upon this dent by @fabsh quoting @nybill : “Linux was always by us, for us. Ubuntu is turning it into by THEM, for us“.

It definitely relates to my current feelings.

When I set up an Ubuntu host, I can’t help feeling like I’m installing some piece of proprietary software. Or course that is not the case : Ubuntu is (mostly) free software and as controversial as Canonical‘s ambitions, inclusion of non-free software or commercial services may be, no one can deny its significant contributions to the advancement of free software – making it palatable to the desktop mass market not being the least… I’m thankful for all the free software converts that saw the light thanks to Ubuntu. But nevertheless, in spite of all the Ubuntu community outreach propaganda and the involvement of many volunteers, I’m not feeling the love.

It may just be that I have not myself taken the steps to contribute to Ubuntu – my own fault in a way. But as I have not contributed anything to Debian either, aside from supporting my fellow users, religiously reporting bugs and spreading the gospel, I still feel like I’m part of it. When I install Debian, I have a sense of using a system that I really own and control. It is not a matter of tools – Ubuntu is still essentially Debian and it features most of the tools I’m familiar with… So what is it ? Is it an entirely subjective feeling with no basis in consensual reality ?

It may have something to do with the democratic culture that infuses Debian whereas in spite of Mark Shuttleworth‘s denials and actual collaborative moves, he sometimes echoes the Steve Jobs ukase style – the “this is not a democracy” comment certainly split the audience. But maybe it is an unavoidable feature of his organization: as Linus Torvalds unapologetically declares, being a mean bastard is an important part of the benevolent dictator job description.

Again, I’m pretty sure that Mark Shuttleworth means well and there is no denying his personal commitment, but the way the whole Canonical/Ubuntu apparatus communicates is arguably top-down enough to make some of us feel uneasy and prefer going elsewhere. This may be a side effect of trying hard to show the polished face of a heavily marketed product – and thus alienating a market segment from whose point of view the feel of a reassuringly corporate packaging is a turn-off rather than a selling point.

Surely there is is more about it than the few feelings I’m attempting to express… But anyway – when I use Debian I feel like I’m going home.

And before you mention I’m overly critical of Ubuntu, just wait until you hear my feelings about Android… Community – what community ?

Free software and Technology and Unix05 Oct 2010 at 10:58 by Jean-Marc Liotier

I stumbled upon Peter Hutterer’s “thoughts on Linux multitouch” which gives a good overview of the challenges facing X.org & al. in developing multitouch over Linux. Among other things he explains why, in spite of end-user expectations to the contrary shaped by competitive offerings, Linux multitouch is not yet available:

“Why is it taking us so long when there’s plenty of multitouch offerings out there already ? The simple answer is: we are not working on the same problem.

If we look at commercial products that provide multitouch, Apple’s iPhones and iPads are often the first ones that come to mind. These provide multitouch but in a very restrictive setting: one multi-touch aware application running in full-screen. Doing this is suprisingly easy from a technical point of view, all you need is a new API that you write all new applications against. It is of course still hard to make it a good API and design good user interfaces for the new applications, but that is not a purely technical problem anymore. Apple’s products also provide multitouch in a new setting, an evironment that’s closer to an appliance than a traditional desktop. They have a defined set of features, different form factors, and many of the user expectations we have on the traditional desktop do not exist. For example, hardly anyone expects Word or OpenOffice to run as-is on an iPhone.

The main problems we face with integrating multitouch support into the X server is the need for the traditional desktop. Multitouch must work across multiple windowed application windows, with some pointer emulation to be able to use legacy applications on a screen. I have yet to see a commercial solution that provides this, even the Microsoft Surface applications I’ve played with so far only emulate this within very restrictive settings”.

In summary, the reason why Linux multitouch lags behind some of its competitors is that it is a significantly more ambitious project with bigger challenges to overcome.

Among the links from that document, I particularly appreciated ‘s Bill Buxton’s “Multi-touch systems that I have known and loved” that provides a great deal of material to frame the debate over multitouch functionality – I feel less clueless about multitouch now…

Systems administration and Unix09 Mar 2010 at 17:18 by Jean-Marc Liotier

Solid state drives provide incredible IOPS compared to hard disks. But the consideration of cost rules them out as primary mass storage. But for most applications you would not consider storing everything in RAM either – yet RAM cache is part of any storage system. Why wouldn’t we take advantage of Solid state drives as an intermediary tier between RAM and hard disks ? This reasoning is what hierarchical storage management is about, but Sun took it one step further by integrating it into the file system as ZFS‘s Hybrid Storage Pools.

You can read a quick overview of Hybrid Storage Pools in marketing terms, but you will surely find Sun’s Adam Leventhal’s presentation more substantial as a technical introduction. And most impressive are Sun’s Brendan Gregg’s benchmarks showing 5x to 40x IOPS improvement !

Adding SSD to a ZFS storage pool is done at to locations : the ZFS intent-log (ZIL) device and the Second Level Adaptive Replacement Cache (L2ARC). Usually they are set on two separate devices, but Arnaud from Sun showed that they can share a single device just fine.

The ZIL, also known as Logzilla accelerate small, synchronous writes. It does not require a large capacity. The L2ARC, also known as Readzilla accelerates reads. For the gory details of how Logzilla and Readzilla work, Sun’s Claudia Hildebrandt’s presentation is a great source.

Creating a ZFS storage pool with one or more separate ZIL devices is dead easy, but you then may want to tune your system for performance. It costs some DRAM to reference the L2ARC, at a rate proportional to record size – between 1/40th and 1/80th of the L2ARC depending on the tuning (I have seen several different estimates) – so don’t set a L2ARC larger than your DRAM affords you.

I hope that this sort of goodness will some day come to Linux through Btrfs, but ZFS provides it right now – and it is Free software too… So I guess that in spite of my religious fervor toward the GPL, my storage server’s next operating system will be a BSD licensed one… Who would have thought ?

Consumption and Free software and Knowledge management and Mobile computing and Networking & telecommunications and Systems and Technology and Unix19 Oct 2009 at 1:18 by Jean-Marc Liotier

Five months have elapsed since that first week-end when my encounter with Android was a severe case of culture shock. With significant daily experience of the device, I can now form a more mature judgement of its capabilities and its potential – of course from my own highly subjective point of view.

I still hate having to use Google Calendar and Google Contacts for synchronization.  I hope that SyncML synchronization will appear in the future, make Android a better desktop citizen and provide more choice of end points. Meanwhile I use Google. With that out of the way, let’s move on to my impressions of Android itself.

I am grateful for features such as a decent web browser on a mobile device, for a working albeit half baked packaging and distribution system, and for Google Maps which I consider both a superlative application in its own right and the current killer albeit proprietary infrastructure for location enabled applications. But the rigidly simple interface that forces behaviours upon its user feels like a straitjacket : the overbearing feeling when using Android is that its designers have decided that simplicity is to be preserved at all costs regardless of what the user prefers.

Why can’t I select a smaller font for my list items ? Would a parameter somewhere in a customization menu add too much complication ? Why won’t you show me the raw configuration data ? Is it absolutely necessary to arbitrarily limit the number of virtual desktops to three ? From the point of a user who is just getting acquainted with such a powerful platform, those are puzzling questions.

I still don’t like the Android ‘s logic, and moreover I still don’t quite understand it. Of course I manage to use that system, but after five month of daily use it still does not feel natural. Maybe it is just a skin-deep issue or maybe I am just not the target audience – but some features are definitely backwards – package management for example. For starters, the “My Downloads” list is not ordered alphabetically nor in any apparently meaningful order. Then for each upgradeable package, one must first browse to the package, then manually trigger the upgrade package, then acknowledge system privileges the upgraded package and finally clear the download notification and the update notification. Is this a joke ? This almost matches the tediousness of upgrading Windows software – an impressive feat considering that the foundations of Android package management seem serious enough. Where is my APT ?

Like any new user on a prosperous enough system, I am lost in choices – but that is an embarrassment of riches. Nevertheless, I wonder why basics such as a task manager are not installed by default. In classic Unix spirit, even the most basic system utilities are independent applications. But what is bearable and even satisfying on a system with a decent shell and package management with dependencies becomes torture when installing a package is so clumsy and upgrading it so tedious.

Tediousness in package management in particular and user interaction in general makes taming the beast an experience in frustration. When installing a bunch of competing applications and testing them takes time and effort. Experimenting is not the pleasure it normally is on a Linux system. The lack of decent text entry compounds the feeling. Clumsy text selection makes cut and paste a significant effort – something Palm did make quick, easy and painless more than ten years ago. Not implementing pointer-driven selection – what were the developers thinking ?

PIM integration has not progressed much. For a given contact, there is no way to look at a communications log that spans mail, SMS and telephony: each of them is its own separate universe. There is no way to have a list of meetings with a given contact or at given location.

But there basic functionality has been omitted too. For example when adding a phone number to an existing contact, search is disabled – you have to scroll all the way to the contact. There is no way to search the SMS archive and SMS to multiple recipients is an exercise left to applications.

Palm OS may have been unstable, incapable of contemporary operating system features, offering only basic functionality and generally way past its shelf date. But in the mind of users, it remains the benchmark against which all PIM systems are judged. And to this day I still don’t see anything beating Palm OS on its home turf of  PIM core features and basic usability.

Palm OS was a poster child for responsiveness, but on the Android everything takes time – even after I have identified and killed the various errant applications that make it even slower. Actually, the system is very fast and capable of feats such as full-motion video that were far beyond the reach of the Palm OS. But the interaction is spoilt by gratuitous use of animations for everything. Animations are useful for graphically hinting the novice user about what is going on – but then hey are only a drag. But please let me disable animations as I do on every desktop I use !

The choice of a virtual keyboard is my own mistake and I am now aware that I need a physical keyboard. After five months, I can now use the virtual keyboard with enough speed and precision for comfortable entry of a couple of sentences. But beyond that it is tiring and feels too clumsy for any meaningful work. This is a major problem for me – text entry is my daily bread and butter. I long for the Treo‘s keyboard or even the one on the Nokia E71 – they offered a great compromise of typing speed and compacity. And no multitouch on the soft keyboard means no keyboard shortcuts which renders many console applications unusable – sorry Emacs users.

The applications offering is still young and I cannot blame it for needing time to expand and mature. I also still need to familiarize myself with Android culture an develop the right habits to find my way instinctively and be more productive. After five months, we are getting there – one handed navigation has been done right. But I still believe that a large part of the user interface conventions used on the Android does not match the expectations for general computing.

It seems like everything has been meticulously designed to bury under a thick layer of Dalvik and Google plaster anything that could remind anyone of Unix. It is very frustrating to know that there is a Linux kernel under all that, and yet to suffer wading knee-deep in the marshes of toyland. The more I use Android an study it, the more I feel that Linux is a mere hardware abstraction layer and the POSIX world a distant memory. This is not the droid I’m looking for.

Brain dump and Consumption and Mobile computing and Unix17 Jun 2009 at 22:23 by Jean-Marc Liotier

I acquired an HTC “G2” Magic less than two days ago. It runs the Android operating system. The feeling of being confronted with something very alien pushed me to record my first impressions with it to give an account of how a foreigner perceives the Android world with his naïve eyes, in his own words. For other systems where I’m a power user, I find the experience of newcomers interesting when they candidly point out problems we ignore because we have simply grown used to them.

This entry relates my feelings from friday night to sunday night. It may seem ridiculous in the future, but it is an instantaneous snapshot – for what it is worth.

First contact with the Android is a severe case of culture shock. More than ten years of Palm OS have shaped my expectations, and the disappointing past year with S60 on the Nokia E71 has not changed them much. But plunging into Android is unlike anything I have used so far – many of the UI conventions feel utterly strange. The home screen has a familiar status bar – but beyond that, Android is a class of its own. For example, instead of grabbing the scroll bar and sliding it, one has to slide the list itself – not illogical, but the contrary of any familiar widget kit I have come accross anywhere else. And many other things are just as alien.

After poking around a little, my first reaction is disorientation from the lack of keyboard. I could write tolerably fast with Palm’s Graffiti, but I was in love with the Treo-style keyboards – on the the Treos as well as on the E71 they let me write considerable volumes fluidly and without excessive strain. But my first attempts at text entry on the G2 are stumbling hit and miss torture with each word containing at least two typing errors. And why does the automatic correction insist on changing my “jm@liotier.org” address to “kn@liotier.org” every single time I enter it in a web form ? Text entry on the G2 is tedious enough without this sort of annoyance…

To be fair, I knew I had to expect text entry woes – I had anticipated that risk when choosing the G2 over the G1. Learning a new tool takes times, especially when low level reflexes are involved, so I have budgeted a few weeks for climbing up the learning curve. Then I’ll decide if I like onscreen keyboards or not. But whatever the learning, it seems that text entry on a virtual keyboard requires to keep one’s eyes on the keyboard – whereas with a physical keyboard, after a while muscle memory sets in and you can forget the keyboard to concentrate on what you are writing. So I’m not optimistic so far – but I’ll keep my mind open. Meanwhile, tactile feedback screens are on the way – I’ll keep an eye on them.

I miss the four-ways arrows button, but the obscene pointing device works rather well although like the rest it will take time to get used to. A “page down” button would be even better than having to swipe the whole screen every time I want to scroll down one step – one page at the time would be more precise than scrolling a random number of lines according to how much inertia the widget takes from the swipe. Screen swipe and inertia are sexy gimmicks, but I don’t understand how heavy users tolerate them for more than five minutes. I’m the sort of user who disables smooth scrolling and any on-screen animation that introduces the slightest lag in my interaction with the system – and I know I’m not the only one who wants responsiveness above everything else.

A combination of importing in Evolution a CSV file generated with Outlook, synchronization from the Nokia E71 to Google and copying native Evolution contacts to Google did not manage to capture at once all of the information I wanted transferred – so I had to munge some of the data and re-enter quite a few of the notes and adresses manually… Hopefully that is the last time I do that. Those contacts had often been through various synchronizations between Palm OS devices, Outlook and Evolution – but getting them to Google seemed lossier than usual. I have read about many other contorted data migration paths, and this one looked straightforward enough – but if I have to do it again I’ll spend time setting up better automation. I’ll concede that it does not have much to do with Android – anywhere you look the synchronization ecosystem seems quite wet behind the ears.

I won’t complain too much about how tightly tied the system is with Google’s applications – after all that is a major feature of Google’s Android business model. With the defaults applications, Android is a seamless extension of the Google universe. Synchronization of calendar and contacts is excellent – although it only happens eventually and you have no way to know when or to trigger it (this is the first time I ever see this implemented with no control by the user). In addition, I am very uncomfortable with the idea of using a third party as my synchronization hub and I’ll look for another way.

But every functionnality seems available through an API – so with the user in control and free to act there is no reason to complain. I’ll ignore the Gmail and Google Talk clients, and replace them with a decent XMPP client and an IMAP client better than the default one – and maybe I’ll even find a decent contacts manager. Meanwhile the native Google Maps client is such a pleasure to use that I could forget everything else (though I wonder why the relief layer has been omitted – I find it very useful for planning human powered movement).

The scarcity of exposed configuration options, output logs and exposed information in general leaves me wanting. For example the Jabiru Jabber client tells me “connexion error” but won’t explain anything, resulting in frustration. Of course, Jabiru is not part of the basic system, but this rarefied atmosphere seems to be the norm in the Android world. And why is there no option to sort the contacts by “family name, first name” instead of the default “first name, family name” ? Would that clutter the interface too much ? Even the simplicity-worshipping Palm OS gave that choice…

I guess that a compromise has been struck in favor of simplicity by default over configurability, and that developper tools are available to provide advanced access to deeply buried parameters – but for example not being able to set the language to english with the french keyboard upsets me a lot. I’m used the english as a device language, and I’m used to the french “AZERTY” keyboard – reading french or typing on a QWERTY keyboard with no accents feels awkward. On any other system I know, keyboard and language are two separate options – but not on Android. I hope I find an application that provides finer grained options. I was also frutrated not to find any configuration option that would solve my above-mentioned problem with the scrolling style.

The Android Market feels sluggish. I have been spoilt by APT caching all packages descriptions locally – and now I have to suffer Android Market loading package descriptions and icons slower than I scroll accross the list that only shows six items per page with no way of getting the device to display smaller characters in order to cram more lines per page. I can understand that the icons must be stored online for storage space’s sake, and maybe the user comments in order to keep them current – but why not load the whole package list at once ? And why does every list on this device use a standard widget that seems sized large enough for legibility by half blind users ? Where is the configuration option ?

Many of my gripes are probably related with the default applications, and after exploring the Android Market for a while I’m sure I’ll feel better. I’m commenting an operating system in its default form, and this is obviously not how I’m going to use it – In a few weeks, after the normal process of appropriation, my Android will hopefully not look and feel like its current state at all.

So see you in a couple of months for a look back at these first impressions – we’ll see which were real problems and which were merely artefacts of the clash of cultures ! For now I have the eery feeling of having stumbled in a sort of Apple-esque Disneyland with my hands tied…

Code and Debian and Free software and Knowledge management and RSS and Social networking and Systems and Unix18 May 2009 at 12:15 by Jean-Marc Liotier

If you want to skip the making-of story, you can go straight to the laconica2IRC.pl script download. Or in case anyone is interested, here is the why and how…

Some of my best friends are die-hard IRC users that make a point of not touching anything remotely looking like a social networking web site, especially if anyone has ever hinted that it could be tagged as “Web 2.0” (whatever that means). As much as I enjoy hanging out with them in our favorite IRC channel, conversations there are sporadic. Most of the time, that club house increasingly looks like an asynchronous forum for short updates posted infrequently on a synchronous medium… Did I just describe microblogging ? Indeed it is a very similar use case, if not the same. And I don’t want to choose between talking to my close accomplices and opening up to the wider world. So I still want to hang out in IRC for a nice chat from time to time, but while I’m out broadcasting dents I want my paranoid autistic friends to get them too. To satisfy that need, I need to have my IRC voice say my dents on the old boys channel.

The data source could be an OpenMicroblogging endpoint, but being lazy I found a far easier solution : use Laconi.ca‘s Web feeds. Such solution looked easier because there are already heaps of code out there for consuming Web feeds, and it was highly likely that I would find one I could bend into doing my bidding.

To talk on IRC, I had previously had the opportunity to peruse the Net::IRC library with great satisfaction – so it was an obvious choice. In addition, in spite of being quite incompetent with it, I appreciate Perl and I was looking for an excuse to hack something with it.

With knowledge of the input, the output and the technology I wanted to use, I could start implementing. Being lazy and incompetent, I of course turned to Google to provide me with reusable code that would spare me building the script from the ground up. My laziness was of course quick to be rewarded as I found rssbot.pl by Peter Baudis in the public domain. That script fetches a RSS feed and says the new items in an IRC channel. It was very close to what I wanted to do, and it had no exotic dependancies – only Net::IRC library (alias libnet-irc-perl in Debian) and XML::RSS (alias libxml-rss-perl in Debian).

So I set upon hacking this script into the shape I wanted. I added IRC password authentication (courtesy of Net::IRC), I commented out a string sanitation loop which I did not understand and whose presence cause the script to malfunction, I pruned out the Laconi.ca user name and extraneous punctuation to have my IRC user “say” my own Identi.ca entries just as if I was typing them myself, and after a few months of testing I finally added an option for @replies filtering so that my IRC buddies are not annoyed by the noise of remote conversations.

I wanted my own IRC user “say” the output, and that part was very easy because I use the Bip an IRC proxy which supports multiple clients on one IRC server connection. This script was just going to be another client, and that is why I added password authentication. Bip is available in Debian and is very handy : I usually have an IRC client at home, one in the office, occasionally a CGI-IRC, rarely a mobile client and now this script – and to the dwellers of my favorite IRC channel there is no way to tell which one is talking. And whichever client I choose, I never missing anything thanks to logging and replay on login. Screen with a command-line IRC client provides part of this functionality, but the zero maintainance Bip does so much more and is so reliable that one has to wonder if my friends cling to Irssi and Screen out of sheer traditionalism.

All that remained to do was to launch the script in a sane way. To control this sort of simple and permanently executed piece of code and keep it from misbehaving, Daemon is a good way. Available in Debian, Daemon proved its worth when the RSS file went missing during the Identi.ca upgrade and the script crashed everytime it tried to access it for lack of exception catching. Had I simply put it in an infinite loop, it would have hogged significant ressources just by running in circles like a headless chicken. Daemon not only restarted it after each crash, but also killed it after a set number of retries in a set duration – thus preventing any interference with the rest of what runs on our server. Here is the Daemon launch command that I have used :

#!/bin/bash
path=/usr/local/bin/laconica2IRC
daemon -a 16 -L 16 -M 3 -D $path -N -n laconica2IRC_JML -r -O $path/laconica2IRC.log -o $path/laconica2IRC.log $path/laconica2IRC.pl

And that’s it… Less cut and paste from Identi.ca to my favorite IRC channel, and my IRC friends who have not yet adopted microblogging don’t feel left out of my updates anymore. And I can still jump into IRC from time to time for a real time chat. I have the best of both worlds – what more could I ask ?

Sounds good to you ? Grab the laconica2IRC.pl script !

Email and Knowledge management and RSS and Social networking and Systems administration and The Web and Unix27 Nov 2008 at 13:17 by Jean-Marc Liotier

Have you tried one more time to convince you parents to switch to web feeds to get updates from the family ? Do you cringe when you see your colleague clumsily wade through a collections of sites main pages instead of having them aggregated in a single feed ? Or did your technophobe girlfriend miss the latest photo album you posted ? With a wide variety of source acknowledging that web feeds ans web feeds readers being perceived as too technical, many of us have scaled back this particular evangelization effort to focus it on users ripe for transitionning from basic to advanced  tools.

Breaking through that resistance outright is beyond our power, but we can get around it. Electronic mail is a mature tool with well understood use cases with which even the least competent users feels comfortable thanks to how easily it maps with the deeply assimilated physical mail model. This is why Louis Gray has started mailing Google Reader items to promote the use of that web feed reader. But we can do better than that by building a fully automated bridge from web feed to email.

Our hope for plugging the late adopters into the information feeds is named rss2email. As its name suggests, Aaron Swartz’s GPL-licensed rss2email utility converts RSS subscriptions into email messages and sends them to whatever address you specify. Despite the name, it handles Atom feeds as well, so you should be able to use it with just about any feed you like. And of course rss2email is available from Debian.

The nice introduction to rss2email by Joe ‘Zonker’ Brockmeier is all the documentation you need – and rss2email is so simple that you probably don’t even need that. I now have some of my favorite late adopters each plugged into his custom subset of my regular information distribution feeds. The relevant news stories get mailed to them without me having to even think about it. And the best part is that they now read them !

Free software and Technology and Unix16 Oct 2008 at 0:09 by Jean-Marc Liotier

Stumbing on an article by about why Free software and open formats are increasingly important as our lives are turning digital, I was suprised to read the following :

If I haven’t convinced you yet that Linux is going to take over the appliance world, I strongly suggest you look at Sony’s web site. There you can find a page full of television models going back to 2003, all of which run on Linux (for those essential moments when you must have the source code to your television, naturally).

Sony is such a big group that the right hand does not know what the left hand is doing – so I am not surprised that Free software is being used, although Sony‘s attitude toward it has had both highs and lows in the past.

Free software on a television is quite intriguing. Some of those televisions not only run the kernel, but also well known user environments such Bash, Busybox and Freetype. Would they be powerful enough to run a thin client with RDP or X11 ? Could they even run a media player ? I would not be suprised : as far back as 2003, Sony President and Chief Operating Officer Kunitake Ando said that “the TV will replace the PC as the center of online entertainment, running on a Linux-based platform for connected home electronics devices. If Sony implements that vision with adequate hardware running Free software, Sony televisions may one day be a tinkerer’s device of choice for the home entertainment projects that currently most often use a dedicated computer to display on the television screen.

Code and Systems administration and Unix and VOIP01 Jun 2008 at 18:55 by Jean-Marc Liotier

Ever since the Linux Advanced Routing & Shaping HOWTO introduced it, I have been a big fan of the Wondershaper, a traffic shaping script that drives Linux‘s class based queuing with stochastic fairness queuing (SFQ) in a pretty effective attempt at maintaining low latency for interactive traffic while at the same time maintaining high throughput. There is even a ‘wondershaper’ Debian package that includes some additional polish. This script is key to the joy of perfectly responsive SSH sessions while peer to peer file sharing traffic saturates the uplink.

Some people have even concluded the resulting quality of service is good enough for voice traffic. But even with the Debian Wondershaper ruling my ADSL link I noticed that SIP and IAX still suffer too much packet loss with the saturating traffic occupying the background. I needed better traffic control.

As usual, being a late adopter I am not the only one to have hit that obstacle, and solutions have already been put forth. After rummaging through various mutations, I found Robert Koch’s version of the Wondershaper for the Asus WL-xxx documented on the Wondershaper package page of the WL-500G wiki to be quite promising. Compared to the standard version it prioritizes VOIP traffic by source port for idiot proof configuration, but also by type of service which is much more flexible and can be used thanks to Asterisk being capable of correctly setting TOS fields. As a bonus, using TOS also makes this version of the script capable of distinction between console interactive SSH traffic and bulk SCP traffic using the same protocol and port. And to top it all, it is based on the better hierarchical token bucket (HTB) discipline which is standard since Linux 2.4.20 while the Debian Wondershaper version uses the more based queuing which used to be the more widespread one.

The first shortcoming I found is that it prioritizes SIP and RTP but not IAX and others which I’ll have to add using the SIP stanzas as templates. The other is that taking lists of low priority ports as arguments could make the command line messy and configuration puzzling for the inexperienced user, so I prefer to have this configuration item as a documented variable allocation inside the script. But those are trifles compared to the new VOIP support, enhanced SSH discrimination and overall upgrade.

Hacking on the script I couldn’t resist reorganizing a few things. I originally intended to provide a diff, but that would be pointless since I ended up touching most of the lines. Also be warned that I do not understand why putting ‘prio 1’ everywhere makes the script work whereas other ‘prio’ values at various places made traffic end up in the wrong class and did not make sense at all. In effect, I think that by putting ‘prio 1’ everywhere I just eschewed the use of priority bands inside the classes, which is just fine with me for the intended use. But this show that my tc fluency is still limited and that there are therefore surely ways to enhance this script. I’ll also welcome feedback – whether it works for you or not.

Anyway – it works ! I had a few VOIP conversations across an IAX trunk with lots of background traffic on the uplink and no perceptible effects on voice quality. Life is good. Now that I have removed the last obstacle to taking full advantage of VOIP at home. Soon all my traffic will be routed through Asterisk and there shall be no more RJ11 nor their French T-sockets alter ego in my home.

Here is my modified wondershaper script in all its glory – contrary to Robert Koch’s version it is a drop-in replacement for Debian’s package. Inheriting from the original Wondershaper it is licensed under the GPL so enjoy, modify and share !

Debian and Systems administration and Unix08 May 2008 at 20:08 by Jean-Marc Liotier

After reading the USP and elevator pitch for Ack, someone took the time to review the hype and make his own opinion about ack. It appears to him that although Ack is not a drop-in replacement for Grep, the DWIM factor makes it a very interesting alternative for many situations where interactive use by a human is involved.

So I decided to give it a spin. ‘apt-cache search ack’ matched 15971 packages related to crack, backup, pack, slack and whatever else you can imagine, but an ‘apt-cache search ack | grep -E ^ack’ solved that problem at once.

In Debian, there was already a kanji code converter named “ack” – but since we don’t intend as far as I know to write Kanji anytime soon I tought it was reasonable to alias ack=’ack-grep’ so that our shell takes advantage of the short name.

After toying with it a little, it appears that Ack is indeed a handy tool which I’ll be definitely using in the future. There is a performance tradeoff when operating with the C locale, but since we strive to be an all UTF-8 shop I don’t care much about that. And anyway, in most interactive situations, brain usage and fingers motricity are far more precious ressources than run time or CPU time…