Re: Internet Backbone Index

[Sean Donelan <SEAN () SDG DRA COM>]

Ok, so I said I wasn't going to comment on the methodology, I lied.
I wouldn't say the Keynote study is the worst ever.  There are some
really rotten studies in the fields of psychology and sociology.

Since the Boardwatch/Keynote study didn't 'test' DRA Net, I guess I'm
one of the few independent, disinterested parties to comment on the
study's methods.

A problem with the Keynote study is it seems very dependent on the
location, type and connections of the testing platforms.  Keynote
mentions that connections from Dallas and Phoenix were slow to 'every'
backbone site.  This would indicate some systematic problem with
the testing sites.  Perhaps the results are even more dependent on
the testing systems than the systems under test.  There are also
problems with outlier data points.  For example, elsewhere on the
Keynote site, the MCI web site had very fast access from 28 test
sites (< 4secs), and very slow access from one test site in
philly (> 14secs).  Mixing and matching data points, if you left
out the one outlier data point, MCI would have been faster than
Savvis.  So I don't know if the rankings are very meaningful if a
single test site can have such a pronounced effect.

Unlike a scientific study, there doesn't seem to be enough information
to independently reproduce the results, so I'm just going from the bits
and pieces I can glean from the Keynote pages, press release, and article.

> Concerning Internet performance, there have always been a variety of ways
> of measuring it. It all depends on what you are really trying to measure.
> The Keynote study is attempting to measure something to which the average 
> Internet user (not engineers) can relate.  However, There are also clearly 
> the possibility of artifacts in the data because of the testing machine's 
> TCP stack or other issues (Vern Paxson has covered these issues at NANOG 
> and IETF meetings over the last few years). Checking their web site, their 
> software appears to run on top of the TCP stacks of many systems, so the 
> known artifacts of some of these platforms could be an issue.
-- 
Sean Donelan, Data Research Associates, Inc, St. Louis, MO
  Affiliation given for identification not representation