A critical opinion of Nokia’s “Experimental Study of Home Gateway Characteristics”
If you can read French and if you are interested in networking technologies, then you must read Stephane Bortzmeyer’s blog – interesting stuff in every single article. Needless to say I’m a fan.
Stéphane commented an article by Nokia people : « An Experimental Study of Home Gateway Characteristics » – it exposes the results of networking performance tests on 34 residential Internet access CPE. For a condensed and more clearly illustrated version, you’ll appreciate the slides of « An Experimental Study of Home Gateway Characteristics » presented at the IETF 78’th meeting.
The study shows bad performance and murky non-compliance issues on every device tested. The whole thing was not really surprising, but it still sounded rather depressing to me.
But my knowledge of those devices is mostly from the point of few of an user and from the point of view of an information systems project manager within various ISP. I don’t have the depth of knowledge required for a critical look at this Nokia study. So I turned to a friendly industry expert who shall remain anonymous – here is his opinion :
[The study] isn’t really scientific enough testing IMHO. Surely most routers aren’t high performance due to cost reasons and most DSL users (Telco environments don’t have more than 8 Mbit/s (24 Mbit/s is max).
[Nokia] should check with real highend/flagships routers such as Linksys E3000. Other issues are common NAT issues or related settings or use of the box DNS Proxy’s. Also no real testing method explained here so useless IMHO. Our test plan has more than 500 pages with full description and failure judgment… :)
So take « An Experimental Study of Home Gateway Characteristics » with a big grain of salt. Nevertheless, in spite of its faults I’m glad that such studies are conducted – anything that can prod the consumer market into raising its game is a good thing !
Experimental study on 34 residential CPE by Nokia: http://j.mp/abqdf6 – Bad performance and murky non-compliance all ove
Experimental study on 34 residential CPE by Nokia: http://j.mp/abqdf6 – Bad performance and murky non-compliance all over
r
2 responses to “A critical opinion of Nokia’s “Experimental Study of Home Gateway Characteristics””
Leave a Reply
You must be logged in to post a comment.
First off, thanks for the interest in our study. Stephane pointed me at your blog post. I’d like to respond to your anonymous “industry expert”.
There seems to be a serious misunderstanding here about the purpose of our study. The intent is NOT to do any sort of in-depth standards conformance testing, which would require the 100s of pages of test planning your expert is referring to.
I also STRONGLY object to his claims that our methodology wasn’t scientifically accurate; especially, because he doesn’t substantiate that claim with any further facts. Did this person READ Section 3?
The intent of our study to demonstrate the diverse behaviors that applications can expect to encounter when running across boxes that people have deployed in their homes right now.
This is also the reason we did explicitly not limit our study to high-end routers, but tried to gather as diverse a set of boxes, box versions and firmware versions as possible. (We now have over 100 boxes, and results will be published in a follow-on study.)
Finally, since your expert was mostly commenting on throughput numbers: Yes, the throughput measurements show that many of these boxes can support the slower speeds of many current ADSL deployments. But more and more folks have FTTH or cable connections, and even those that don’t may be limited by their very broken home router, if case they chose the wrong model.
As to your conclusion: It is important to understand the limiting factors of any scientific study. Ours is no exception. With that understanding, however, I do believe our results are obtained in a scientifically sound manner. No salt needed.
Thank you for your detailed rebuttal and clarification. I’m afraid this thread will end here – my anonymous friend doesn’t wish to join the conversation. But I look forward seeing more of your efforts to shine light on those networking devices !