Serendipitous altruism

I just bought a handful of cheap metal screw mount hot shoe adapters. I have not yet had a chance to use them. Actually I might say I have been lucky not to have used them yet… While learning about umbrella adapters I just stumbled on a piece piece of practical advice whose timing could not have been better :

“Oh, and before I forget, as soon as you get it, if the shoe mount is made of metal put a piece of electrical tape on the top of it where the flash sits. It could screw up the electronics in the month’s-rent-worth-of-strobe you have sitting up there if you don’t. I’m not kidding about that”.

One more reason to have a large stock of duct tape at home. And one more reason to read Strobist’s blog – if I needed any…

My photographic output is generally well received by models and friends alike, but they are not a very critical bunch so being content with the status quo would be utterly presumptuous from my part. But there is no danger of that, as kaizen is very dear to my heart.

Although I now have a good grasp of basic tools such as various selection implements, levels, curves, noise reduction and sharpening, my retouching skills are still very far from anything serious. Training helps me creep slowly forward, but it has now been a few months since it dawned on me that those skills probably won’t improve much more until I get myself to learn dealing with layers and transparency. Spray painting with the clone brush on the background layer is better than nothing, but a good layered makeup with gaussian blur and assorted masking trickery is what I am really aiming for.

The tool I am missing is called “adjustment layers” : “an adjustment layer typically applies a common effect like brightness or saturation to other layers. However, as the effect is stored in a separate layer, it is easy to try it out and switch between different alternatives, without destroying the original layer. In addition, an adjustment layer can easily be edited, just like a layer mask, so an effect can be applied to just part of the image“.

So I went looking for explanation on how to use adjustment layers. I found Tea Leaves explaining how to use adjustment layers with an example in the context of Photoshop – according to him “it is like having a darkroom notebook that also remembers exactly how to edit your pictures for you. It really is magic“. Matt Greer explains in more detail the benefits of adjustment layers :

“The benefit to using adjustment layers is that no edit is permanent until you flatten the image. You can even save the image with all of its adjustment layers as a Photoshop Document (.psd), and when you reopen it, all the changes you made to the adjustment layers will still be there for you to change back, remove, or alter.

If you were to, for example, edit curves without layers, then go on to change saturation, crop the image, then add vignetting, the only way to go back and change what you did to the curves would be to go back in the history, to when you changed the curve (thus losing all work done since), or start the image editing from scratch.

With adjustment layers, however, so long as that adjustment layer is still there, you can go back and alter the adjustment at any point in the editing process”.

Phong explains how easy it is to add an adjustment layer, and Martin from t-tutorials.com explains their power at further length. So far so good, I’m sold on adjustment layers ! But what about my photo retouching tool of choice, Gimp ?

Gimp lacks adjustment layers and users have been complaining about it for quite a while. Actually I should say that the lack of adjustment layers is an essential part of the classic list of rants against Gimp. Raphael Quinet, Gimp.org webmaster from 2001 to 2004 says : “Adjustment layers will not be in 2.4. This will probably have to wait until the GEGL-ification of GIMP is complete (i.e., GIMP 3.0)“. Raphael even mentions a feature wishlist item about about adjustment layers that has been open as a bug since 2002.

GEGL-ification of GIMP ? That could take a while. There have recently been a few encouraging noises about a revival of GEGL’s development, but by all estimates GEGL still appears on the same horizon where it has stayed there for the last seven years. GEGL is supposed to cure GIMP‘s woes by letting it scale freely in image sizes, number of layers, bit depth, functionnality and anything else you might imagine. GEGL looks very promising and the rumor says that it is rooted in very sound foundations. But in the context of GIMP, GEGL smells like a severe case of second system effect. I am not betting my money on a GEGL-ified GIMP appearing anytime soon. I do believe that GEGL will deliver an awesome next-generation graphical framework, but for GIMP it will be too late.

So where do GIMP refugees run to ? Photoshop is quite expensive, and CS2, its latest incarnation is does not run well in Linux with Wine. But hope is not lost : Krita has had adjustment layers for almost one year. Krita looks like the light at the end of the tunnel, the potential savior also bringing forth the high bit depth colorspaces that will one day enable the full 16 bit workflow that begins with the RAW image files. Krita even already mention adjustment layers in its documentation – that is a very good sign. So for now I am going to take a very close look at Krita and experiment with it with the goal of replacing GIMP in the short term. Adjustment layers alone pushed me to consider migration but with the prospects of a 16 bit workflow on the horizon my motivation is now even stronger.

But with all this talk about tools, let’s not forget to sharpen those retouching skills too. And while retouching let’s not forget about those photographic skills either. Let’s start with this example of bad lighting :

Under bounce flash, the shadows under the cheeks of this particular model are not nice. Next time I’ll try using my brand new pocket studio lighting to get rid of them… And that illustrates that as usual there is more than one way to improve an image. After a whole article of ranting about image manipulation tools I want to conclude by putting them back into perspective : digital hackery will always come second to getting a good exposure.

After agonizing for a few months over a lighting equipment purchase decision I finally took the plunge and bought a couple of additional Canon Speedlite 580EX, in accordance to the teachings of the guru of small shoe-mount flashes.

Even cheap AC powered lights provide more power than the Canon portable strobes will ever put out, but I can carry the portable strobes anywhere in my backpack and set the up on a whim – and that fits my lifestyle much better. As Strobist says : “larger strobes have their place, but they tend to spend a lot of time in trunks and stuffed under beds. But the small, everyday strobe is always in the waistpack ready to go“.

I began playing with my new toys on this Christmas week-end as I happened to have a willing model at hand. My first setup was definitely random and the results are rather haphazard but I have at least the above picture to be quite happy with.

After that first experience I can already say that I do not regret my decision. A bunch of 580EX are really a pocket studio all by themselves – and with a €4 screw mount adapter I can even make good use of those flimsy toy tripods given away with breakfast cereal packs and spotting scopes. I now have ample room to grow my lighting skills and have loads of fun on the way. Expect more multiple flash experiments !

I am using mod_proxy to hide my host-wide Geneweb setup behind a bunch of Apache vhosts. I was surprised to find that after migrating to Apache 2.2 my mod_proxy setup had ceased working. The vhost’s access.log was showing a 403 and the error.log was churning messages containing “proxy: No protocol handler was valid for the URL”. I fed that message to Google and after looking at a few random threads I began to understand that the mod_proxy configuration had most probably changed between Apache 2.0 and Apache 2.2.

In addition to mod_proxy.so additional modules now have to be loaded in order to support a few configuration directives. The mod_proxy configuration for my Geneweb setup is as follow :

RewriteEngine On
ProxyPass /robots.txt http://www.bensaude.org/robots.txt
ProxyPass / http://kivu.grabeuh.com:2317/
ProxyPassReverse / http://kivu.grabeuh.com:2317/

A quick look at the available modules in /etc/apache2/mods-available showed me that in addition to mod_proxy.so I also had mod_proxy_ajp.so, mod_proxy_balancer.so, mod_proxy_connect.so, mod_proxy_ftp.so, mod_proxy_html.so and mod_proxy_http.so. On a hunch I decided that mod_proxy_http.so was the best candidate so I tried that first.

ln -s /etc/apache2/mods-available/proxy_http.load \
/etc/apache2/mods-enabled/proxy_http.load
apache2ctl configtest
apache2ctl graceful

Lo and behold – it now works…
Merry whatever to all of you !

I automatically generate daily statistical reports for my web sites traffic using Awstats. Awed by Awstats extensive reporting capabilities I enabled everything with full details and let it run like that. Erik, one of my favorite contradictors, found that I may have gone a bit too far on that one. Of course I first dismissed that as one of his usual privacy rants – we both have very different ideas of how much personal information we should let the public know about us. But a quick costs/benefits analysis showed that for once we actually had some common ground.

First he mentioned that my reports were indexed by search engines. I was aware of that but I saw no wrong about it and did not even bother adding a robots exclusion pattern. But having the statistical reports indexed brought no one any significant value : all users had other ways to access them through links. So the benefit was zero. In addition, the indexation of pages containing referer links promotes referer spam – and everyone know how much I love to hate spammers. The costs/benefits analysis provided a clear conclusion and the corresponding robots.txt was therefore swiftly added.

Then Erik mentioned the presence of IP addresses in the Awstats reports. I had never given any thought about those, but the privacy breach was obvious : ill intentioned organizations could easily track the users who indulge in a visit to my hall of deviant ramblings. My first reaction was to consider that whoever wants to hide can use an anonymous proxy or a Tor onion routing gateway. But Erik made me realize that we are dealing with the clueless masses. And as plentypotent semi-divinities with root access we have a duty to protect them from their own lack of clues.

Moreover it occurred to me that this report is not very useful. I need the IP addresses as raw material to generate about every piece of statistical data, but that can very well be done done anonymously. The only redeeming value of the section of the report containing IP addresses is letting me know if a handful of hosts are actually generating all the traffic. The value of this information strongly decreases as traffic reaches statistically significant numbers. So once again the costs/benefits analysis provides an easy conclusion : letting the hosts report go would not be too painful either. Ideally I would keep it in an anonymous form. But that would require modifying Awstats and I am not going to allocate resources to that today. So for now I am just going to tell Awstats to skip it.

So here we go :

cd /etc/awstats

perl -i -pe 's/ShowHostsStats=PHBL/ShowHostsStats=0/g' *.conf

That’s all folks ! I now just have to force regeneration of all my web traffic reports. Good thing that all that is now completely automated !

To those who doubt that I can change my mind : I can readily change my mind with ease, but I require to be convinced either by myself alone or with the assistance of a third party. Let this be an example for those who lost all hope of convincing me.

I thought I had spam pretty much under control, with only about one getting though every few days. And then came image spam. No suspicious words, just a load of bayes poison and an image to carry the actual message. Half of my antispam arsenal was suddenly rendered useless. I was back to suffering one or two spam messages every day.

“The level of image spam has increased dramatically this year,” says Carole Theriault, a senior consultant at Sophos cited by New Scientist. According to New Scientist, Sophos estimates that, at the beginning of the year, image spam accounted for only 18% of unsolicited mail but that this has since risen to 40%.

Less impressive but much more useful than statistical FUD from a biased source, were the few articles about using optical character recognition to fight image spam, from Debian Administration and Linux Weekly News among others.

But with a production server on my hands and precious little time to maintain it I wished to stick to packages distributed by Debian so I waited a little longer for packaging while my users suffered. To my great relief, Christmas came a few days earlier this year – FuzzyOcr hit Debian unstable yesterday ! Mail server administrators rejoice ! Somebody must have been even more pissed off than me about image spam and decided to make the Debian packaging work…

So here is the FuzzyOcr Debian package blurb, straight from the horse’s mouth :

This Spamassassin plugin checks for specific keywords in image/gif, image/jpeg or image/png attachments, using gocr (an optical character recognition program). This plugin can be used to detect spam that puts all the real spam content in an attached image, while the mail itself is only random text and random html, without any URL’s or identifiable information. Additionally to the normal OcrPlugin, it can do approximate matches on words, so errors in recognition or attempts to obfuscate the text inside the image will not cause the detection to fail.

But a debug log is worth a thousand words, so here is a choice output :

[2006-12-19 12:03:39] Debug mode: Starting FuzzyOcr...
[2006-12-19 12:03:39] Debug mode: Attempting to load personal wordlist...
[2006-12-19 12:03:39] Debug mode: No personal wordlist found, skipping...
[2006-12-19 12:03:39] Debug mode: Analyzing file with content-type "image/gif"
[2006-12-19 12:03:39] Debug mode: Image is single non-interlaced...
[2006-12-19 12:03:39] Debug mode: Recognized file type: 1
[2006-12-19 12:03:39] Debug mode: Image hashing disabled in configuration, skipping...
[2006-12-19 12:03:40] Debug mode: Found word "price" in line
"lmatpriceguaranteeftdenrey"
with fuzz of 0 scanned with scanset /usr/bin/gocr -i -
[2006-12-19 12:03:40] Debug mode: Found word "price" in line
"lmatpriceguaranteeftdenrey"
with fuzz of 0 scanned with scanset /usr/bin/gocr -l 180 -d 2 -i -
[2006-12-19 12:03:40] Debug mode: Found word "viagra" in line
"viaraloomgaooomaoo"
with fuzz of 0.166666666666667 scanned with scanset /usr/bin/gocr -i -
[2006-12-19 12:03:40] Debug mode: Found word "viagra" in line
"viaqrastloomaaomaa"
with fuzz of 0.166666666666667 scanned with scanset /usr/bin/gocr -i -
[2006-12-19 12:03:40] Debug mode: Found word "viagra" in line
"viaqrastloomaarialisnomaa"
with fuzz of 0.166666666666667 scanned with scanset /usr/bin/gocr -l 180 -d 2 -i -
[2006-12-19 12:03:40] Debug mode: Found word "cialis" in line
"viaqrastloomaarialisnomaa"
with fuzz of 0.166666666666667 scanned with scanset /usr/bin/gocr -l 180 -d 2 -i -
[2006-12-19 12:03:40] Debug mode: Found word "valium" in line
"valiumlomgaooantivanmgalo"
with fuzz of 0 scanned with scanset /usr/bin/gocr -l 180 -d 2 -i -
[2006-12-19 12:03:40] Debug mode: Found word "legal" in line
"vlaraloomgaoorlalisomaoo"
with fuzz of 0.2 scanned with scanset /usr/bin/gocr -l 180 -d 2 -i -
[2006-12-19 12:03:40] Debug mode: Starting FuzzyOcr...
[2006-12-19 12:03:40] Debug mode: Attempting to load personal wordlist...
[2006-12-19 12:03:40] Debug mode: No personal wordlist found, skipping...
[2006-12-19 12:03:40] Debug mode: FuzzyOcr ending successfully...
[2006-12-19 12:03:40] Debug mode: Message is spam (score 10)...
[2006-12-19 12:03:40] Debug mode: Words found:
"price" in 1 lines
"viagra" in 2 lines
"cialis" in 1 lines
"valium" in 1 lines
"legal" in 1 lines
(6 word occurrences found)
[2006-12-19 12:03:40] Debug mode: FuzzyOcr ending successfully...

Sweet isn’t it ? And that antispam OCR goodness is just an ‘apt-get install fuzzyocr’ away !

The only parameters I changed in /etc/FuzzyOcr.cf are the following :

focr_verbose 2.0
focr_logfile /var/log/fuzzyocr.log
focr_timeout 16

The two first are self-explanatory : I just want to know what is going on. The original timeout was 12 seconds and I found that it was often too short for my puny server – apparently 16 seconds are more than enough. I restarted Amavisd-new who handles calling SpamAssassin and I was done !

I was afraid that FuzzyOcr would load my host too much but I found my fears unfounded : FuzzyOcr only scan messages which where not recognized yet as ham or spam by other SpamAssassin rules or plugins. So the additional load was not noticeable among all the heavy antispam and antiviral machinery that already operated. FuzzyOcr is full of nice surprises !

As a conclusion I must say that, on our mail server, FuzzyOcr is a complete success. I recommend that you install it as soon as possible !

Versamail, the default mail application on my Treo 650 is good for casually checking a handful of new messages in the inbox but it comes short for about anything else. For searching accross my 5 GB of mail in a hundred folders and accessing them accross a wet string, Mutt is by far the best remote solution.

Although twenty-four years younger than Vi, Mutt was apparently also designed to thrive in an environment made of appaling latency, tiny bitrates and screens with an even lower definition than the Treo‘s screen. GPRS really feels like my first 2400 baud modem, so to me a command line with Lbdb, Mairix, Mutt and Vim definitely seems to be the perfect match for GPRS on the Treo.

A few weeks ago I established a SSH session from my Treo with Pssh through Orange France‘s GPRS network ! SSH on Orange France‘s GPRS network used to be stupidly forbidden for no apparent reason, neither technical nor commercial, but someone apparently wised up.

Mobile SSH is just grand ! A few years ago with a laptop I used 9600 bps CSD from an Erisson SH888 accross an IrCOMM infrared link. But the burden of a laptop is gone and SSH is now truly available on the go.

As far as I have searched, Pssh is the only free and functionnal SSH client on the Palm Platform. For command line work the Treo‘s keyboard comes very handy. And Pssh has a few workarounds to deal with the differences between the Treo‘s keyboard and the standard PC keyboard that most command line software normally expects.

Pssh‘s inability to go to the background without losing its connection is annoying. But apart from that it does the job just fine by giving me mobile access to the wonderful world of the Unix command line – and that means that I can now potentially do almost anything anywhere. Isn’t that sweet ?

Those among you who pay attention may have wondered about the meaning of “serendipity“. You shall soon wonder no more – here is an extract from the Journal of Investigative Dermatology explaining salient elements of the etymology of the term “Serendipity” :

The “Tale of the Three Princes of Serendip” is [a] literary framework of ancient Indo-Persian origin. [..] Renaissance brought this collection of Oriental legends of travels, riddles, sagacity in solving them, to the European stage. The Tramezzini of Venice, brother-editors, used a fictional author to offer it to the public in 1557. Horace Walpole (1717–1797), son of the famous British prime minister, sitting in his estate in Strawberry Hill in Twickenham, Middlesex, translated the sagacity of the three princes into a concept which he labeled “serendipity” in a letter to Horace Mann, British envoy at the Florentine Court, dated January 28, 1754. Literally, he wrote of the three princes: “they were always making discoveries, by accidents and sagacity, of things they were not in quest of”. [..] Semantically, the referral points to the island of Sri Lanka (Ceylon of yesteryear). (Serendip) is the historical Arabo-Persian- form of Sri Lanka, a word with Sanskrit (Pali) origin, naming the island of Singhalese (Sinhala dvipa) and kept alive in folk tales and legends of this area.

How that relates with Investigative Dermatology is anybody’s guess but I would bet that the author has made a lame attempt at making it topical by putting it on account of serendipity. Anyway…

“The Three Princes of Serendip” by Richard Boyle explains the story in more details and the dissertation continues in “Serendipity: How the Vogue word became Vague“.

Now that I have delivered those extensive explanations, my problem now is that the term is no longer obscure enough to be fit for a trendy title… But maybe I’m old enough to stop caring about that…