Serendipitous altruism

« Aliens from the communist planet of Rooskee are invading peaceful, democratic planets and turning their inhabitants into “Communist Mutants”. The communist mutant armies are controlled by the Mother Creature, a strange alien who has gone mad due to irradiated vodka. »

Is this real ? Is this really the synopsis for a 1982 computer game ? Wikipedia and various other sources agree that Communist Mutants from Space really did exist. I did not have the privilege of playing it on my Atari 2600 at the time – and somehow I’m glad that the Cold War propaganda we were exposed to did not go to such baroque lengths…

Stéphane Bortzmeyer has a very long track record of interesting commentary about the Internet – his blog goes back to 1996. Its a pity that my compatriot doesn’t write in English more often: I believe he would find a big audience for his excellent articles. But as he told me : “Many people write in English already, English readers do not need one more writer”. I object – there is always room for good information to be brought to a greater audience. And since his writings are licensed under the GFDL, I’ll do the translation myself when I feel like it.

Maybe this will be the only of his articles I translate – or maybe there will be others in the future… Meanwhile here is this one. I chose it because DNS hijacking is a subject I am sensitive about – and maybe because of the exoticism of Chinese shenanigans…

---

Before reading this interesting article, please heed this forewarning : as soon as we talk about China, we should admit our ignorance. Most people who pontificate about the state of the Internet in China do not speak Chinese – their knowledge of the country stops at the doorstep of international hotels in Beijing and Shanghai. The prize for the most ludicrous pro-Chinese utterance goes to the Jacques Myard, representative at the National Assembly and member of the UMP party, for his support for the Chinese dictatorship [translator’s note : he went on the record saying that “the Internet is utterly rotten” and went on to say that it “should be nationalized to give us better control – the Chinese did it”]. When it comes to DNS, one of the least understood Internet services, the bullshit production rate goes up considerably and sentences where both « DNS » and « China » occur are most likely to be false.

I am therefore going to try not emulating Myard, and only talk about what I know, which will make this article quite short and full of conditional. Unlike criminal investigations in US movies, this article will name no culprit and you won’t even know if there was really a crime.

DNS root servers hijacking for the purpose of implementing the policy (notably censorship) of the Chinese dictatorship has been discussed several times – for example at the 2005 IETF meeting in Paris. It is very difficult to know exactly what happens in China because Chinese users, for cultural reasons, but mostly for fear of repression, don’t provide much information. Of course, plenty of people travel to China, but few of them are DNS experts and it is difficult to get them to provide data from mtr or dig correctly executed with the right options. Reports on censorship in China are often poor in technical detail.

However, from time to time, DNS hijacking in China has visible consequences outside of Chinese territory. On the 24th March, the technical manager for the .cl domain noted that root server I, anycast and managed by Netnod, answered bizarrely when queried from Chile :

$ dig @i.root-servers.net www.facebook.com A

; <<>> DiG 9.6.1-P3 <<>> @i.root-servers.net www.facebook.com A
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7448
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.facebook.com.              IN      A

;; ANSWER SECTION:
www.facebook.com.       86400   IN      A       8.7.198.45

;; Query time: 444 msec
;; SERVER: 192.36.148.17#53(192.36.148.17)
;; WHEN: Wed Mar 24 14:21:54 2010
;; MSG SIZE  rcvd: 66

[translator’s note : sign of the times, the Chilean administrator chose to query facebook.com – google.com and, before that, microsoft.com used to be classic example material Mauricio used facebook.com (or twitter.com) because it is hijacked by the chinese govt, unlike microsoft.com (or even google.com)]

The root servers are not authoritative for facebook.com. The queried server should therefore have answered with a pointer to the .com domain. Instead, we find an unknown IP address. Someone is screwing with the server’s data :

• The I root server’s administrators as well as its hosts deny any modifications of the data obtained from VeriSign (who manages the DNS root master server).
• Other root servers (except, oddly, D) are also affected.
• Only UDP traffic is hijacked – TCP is unaffected. Traceroute sometimes ends up at reliable instances of the I server (for example, in Japan) which seem to suggest that the manipulation only affects port 53 – the one used by the DNS.
• Affected names are those of services censored in China, such as Facebook or Twitter. They are censored not just for political reasons, but also because they compete with Chinese interests.

If you want to check it yourself, 123.123.123.123 is hosted by China Unicom and will let you resolve a name :

% dig A www.facebook.com @123.123.123.123 

; <<>> DiG 9.5.1-P3 <<>> A www.facebook.com @123.123.123.123
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44684
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.facebook.com.              IN      A

;; ANSWER SECTION:
www.facebook.com.       86400   IN      A       37.61.54.158

;; Query time: 359 msec
;; SERVER: 123.123.123.123#53(123.123.123.123)
;; WHEN: Fri Mar 26 10:46:52 2010
;; MSG SIZE  rcvd: 66

37.61.54.158 is a currently unassigned address and it does not belong to Facebook. [translator’s note : I get 243.185.187.39 which is also abnormal]

It is therefore very likely that rogue root servers exist in China and that Chinese ISP have hacked their IGP (OSPF for example) to hijack traffic bound toward the root servers. This does not quite explain everything – for example why the known good instances installed in China still see significant traffic. But it won’t be possible to know more without in-depth testing from various locations in China. A leak from this routing hack (similar to what affected YouTube in 2008) certainly explains how the announcement from the rogue server reached Chile.

« The Great DNS Wall of China » and « Report about national DNS spoofing in China » are among the reliable sources of information about manipulated DNS in China.

For more information about the problem described in this article, you may also read « China censorship leaks outside Great Firewall via root server » (a good technical  article), « China’s Great Firewall spreads overseas » or « Web traffic redirected to China in mystery mix-up ».

This article is distributed under the terms of the GFDL. The original article was published on Stéphane Bortzmeyer’s blog on the 26 March 2010 and translated by Jean-Marc Liotier the same day.

In my inbox right now :

“I agree with you that is not logic to have some 0=OFF and 0=ON but this is the way is coded in this version; HQ will try to improve in next version”.

Can you imagine how someone thought it would be a jolly good idea to have “0” mean “OFF” or “ON” for different variables in the same context… That will make parameters management so much more fun !

Names withheld to protect the somewhat innocents.

Gnutella was the first decentralized file sharing network. It celebrated a decade of existence on March 14, 2010. Once Audiogalaxy went down in 2002, it became my favorite service for clandestine file sharing. In late 2007, it was the most popular file sharing network on the Internet with an estimated market share of more than 40%. But nowadays, BitTorrent steals the limelight. How did that happen ?

Gnutella has structural scalability limitations that even its creator acknowledged from the very start. Over the years, major improvements were introduced, but search horizon and network size remain intrinsic limitations due to search traffic. On the other hand, BitTorrent outsourced much of the search and indexing of files to torrent web sites, only handling the actual distribution of data within the client.

Providing search across the indexes requires other parties to provide them, but that architectural constraint has paradoxically become a key driver of BitTorrent’s popularity by providing a simple business model. Ernesto at TorrentFreak explains that easy monetization explains the ubiquity of indexes : “BitTorrent sites can generate some serious revenue, enough to sustain the site and make a decent living. In general, ad rates per impression are very low, but thanks to the huge amounts of traffic it quickly adds up. This money aspect has made it possible for sites to thrive, and has also lured many gold diggers into starting a torrent site over the years“.

With commercial interests comes spam and legal vulnerabilities – so I feel much more comfortable knowing that decentralized protocols exist to provide resilience towards the censorship that lurks over us in the dark, waiting for us to become complacently reliant on centralized resources. Happy birthday Gnutella !

In a comment to a nostalgic utterance by Louis Gray, I found that Joelle Nebbe (iphigenie) expressed best what happens when participants in online communities move on :

“That’s the problem of online communities – they cannot move. It doesn’t matter how good the community is now, people just wont agree to move to the same place as one… They fragment, people move, new ones form, but large groups just never manage to move as one to a new platform. You get fond memories, and happy surprises when names reappear on another community later”.

The proverbial cat herding is well known to anyone who has had to deal with human change management, but in online communities not bound by any organizational structure the problem is even worse.

Online communities will continue to rise and fall, and with that there will always be fond memories and happy surprises !

Solid state drives provide incredible IOPS compared to hard disks. But the consideration of cost rules them out as primary mass storage. But for most applications you would not consider storing everything in RAM either – yet RAM cache is part of any storage system. Why wouldn’t we take advantage of Solid state drives as an intermediary tier between RAM and hard disks ? This reasoning is what hierarchical storage management is about, but Sun took it one step further by integrating it into the file system as ZFS‘s Hybrid Storage Pools.

You can read a quick overview of Hybrid Storage Pools in marketing terms, but you will surely find Sun’s Adam Leventhal’s presentation more substantial as a technical introduction. And most impressive are Sun’s Brendan Gregg’s benchmarks showing 5x to 40x IOPS improvement !

Adding SSD to a ZFS storage pool is done at to locations : the ZFS intent-log (ZIL) device and the Second Level Adaptive Replacement Cache (L2ARC). Usually they are set on two separate devices, but Arnaud from Sun showed that they can share a single device just fine.

The ZIL, also known as Logzilla accelerate small, synchronous writes. It does not require a large capacity. The L2ARC, also known as Readzilla accelerates reads. For the gory details of how Logzilla and Readzilla work, Sun’s Claudia Hildebrandt’s presentation is a great source.

Creating a ZFS storage pool with one or more separate ZIL devices is dead easy, but you then may want to tune your system for performance. It costs some DRAM to reference the L2ARC, at a rate proportional to record size – between 1/40th and 1/80th of the L2ARC depending on the tuning (I have seen several different estimates) – so don’t set a L2ARC larger than your DRAM affords you.

I hope that this sort of goodness will some day come to Linux through Btrfs, but ZFS provides it right now – and it is Free software too… So I guess that in spite of my religious fervor toward the GPL, my storage server’s next operating system will be a BSD licensed one… Who would have thought ?