Keynote/Boardwatch Results

["Jack Rickard" <jack.rickard () boardwatch com>]

I hate to say I told you so.  Actually, I don't, but I can be such a pain
in the ass anyway what difference does it make.

It would appear that everyone is pretty smugly satisfied by concensus that
the performance series we ran actually measures server performance and that

since all ISPs run weeny home servers, this was not "really" a test, flawed
methodology, etc.  I corresponded with Doug the Hump at Digex about this. 
I've liked this guy since I first met him largely because he's funny and
doesn't take himself too seriously.  He's got a yen for black helicopters
that still has me in stitches.

In any event, he didn't appear to be emotionally involved, but noted that
he did think their web server was the problem in their case and that it was
a real issue with the other backbones as well.  He said that if we had
measured one of their honking customer web servers that it would have all
been better.

Now I was clear with Doug, as I have been in this mailing list, that
servers DO have impact.  When you are attempting to measure end to end, you
certainly hope that the results cummulatively represent everything that has
an effect.  But I have been very clear that I didn't buy the "measuring
server" theory.  It will have an effect, but not near the effect you all
have apparently dreamed up amongst you with no data at all.

Anyway, Doug coughed the names of a couple of "honkin" customer sites.  One
on a UNIX machine running Apache.  One on an NT server.  As it so happens
they are the NIKE site and the FORBES site.  I agreed to run them all for a
few days and publish the results rather openly.  Here they are:

Www.digex.com

Metro Area      Abbr    Population      Mean    Std Dev Data Pts.       Rating  
Omaha   OMA     640000  -       -       -       -       
Norfolk ORF     1443000 0.917   0.787   477     -       
Milwaukee       MKE     1607000 2.641   5.545   427     -       
Cleveland       CLE     2860000 2.893   4.155   432     -       
Washington D.C  WAS     6727000 3.326   10.274  436     -       
Kansas City     MKC     1583000 3.334   3.881   440     -       
Detroit DTT     5187000 3.424   11.593  470     -       
Atlanta ATL     2960000 3.434   5.774   447     -       
Denver  DEN     1980000 3.497   2.725   433     -       
Tampa   TPA     2068000 3.541   9.753   440     -       
Minneapolis-St. Paul    MSP     2539000 3.697   7.263   420     -       
Pittsburgh      PIT     2395000 4.039   9.484   441     -       
Miami   MIA     3193000 4.336   11.227  409     -       
Chicago CHI     8240000 4.433   7.977   432     -       
Philadelphia    PHL     5893000 5.687   36.621  446     -       
Columbus        CMH     1345000 6.582   10.275  352     -       
San Diego       SAN     2498000 7.124   6.563   111     -       
Houston HOU     3731000 7.156   19.197  426     -       
Boston  BOS     5455000 8.147   68.522  342     -       
New York        NYC     19550000        8.911   48.549  436     -       
San Francisco   SFO     6253000 9.462   70.834  367     -       
Phoenix PHX     2238000 12.226  26.802  447     -       
Seattle SEA     2970000 14.989  74.818  201     -       
Dallas-Ft. Worth        DFW     4037000 16.24   36.844  421     -       
Los Angeles     LAX     14532000        33.879  78.169  572     -       
Salt Lake City  SLC     1072000 48.719  80.656  383     -       
Portland        PDX     1793000 65.23   88.641  446     -

                        11.287  42.959  10654
www.nike.com

Metro Area      Abbr    Population      Mean    Std Dev Data Pts.       Rating  
Omaha   OMA     640000  -       -       -       -       
Norfolk ORF     1443000 1.359   6.704   385     -       
Washington D.C  WAS     6727000 2.838   6.486   426     -       
Cleveland       CLE     2860000 2.892   5.958   424     -       
Detroit DTT     5187000 3.032   8.818   380     -       
Milwaukee       MKE     1607000 3.262   10.087  420     -       
Tampa   TPA     2068000 3.263   5.876   427     -       
Philadelphia    PHL     5893000 3.697   7.305   433     -       
Kansas City     MKC     1583000 3.82    7.477   429     -       
Los Angeles     LAX     14532000        3.979   7.142   441     -       
Denver  DEN     1980000 4.22    8.725   421     -       
Miami   MIA     3193000 4.338   13.774  398     -       
Pittsburgh      PIT     2395000 4.392   10.68   431     -       
Minneapolis-St. Paul    MSP     2539000 4.545   8.688   409     -       
Atlanta ATL     2960000 4.577   16.279  438     -       
New York        NYC     19550000        5.163   13.025  427     -       
Boston  BOS     5455000 5.725   16.02   347     -       
Chicago CHI     8240000 5.95    11.926  425     -       
San Francisco   SFO     6253000 7.254   21.74   353     -       
Houston HOU     3731000 7.557   14.675  421     -       
Seattle SEA     2970000 9.359   22.033  184     -       
Columbus        CMH     1345000 9.991   21.586  358     -       
Phoenix PHX     2238000 14.089  25.896  432     -       
Dallas-Ft. Worth        DFW     4037000 17.677  44.361  407     -       
San Diego       SAN     2498000 19.029  10.833  16      -       
Salt Lake City  SLC     1072000 37.156  92.765  302     -       
Portland        PDX     1793000 79.404  166.148 437     -       
        
                        9.868   44.374  9971
www.forbes.com

Metro Area      Abbr    Population      Mean    Std Dev Data Pts.       Rating  
Omaha   OMA     640000  -       -       -       -       
Kansas City     MKC     1583000 0.0050  0.0     14      -       
Norfolk ORF     1443000 2.165   11.616  380     -       
Miami   MIA     3193000 2.715   10.382  396     -       
Washington D.C  WAS     6727000 3.151   22.964  419     -       
Philadelphia    PHL     5893000 3.177   14.547  423     -       
Milwaukee       MKE     1607000 3.204   19.791  416     -       
Atlanta ATL     2960000 3.364   17.068  434     -       
Minneapolis-St. Paul    MSP     2539000 3.847   10.974  405     -       
Cleveland       CLE     2860000 3.879   20.768  415     -       
Denver  DEN     1980000 3.974   22.303  417     -       
Tampa   TPA     2068000 4.001   19.877  421     -       
Pittsburgh      PIT     2395000 4.333   25.179  426     -       
Detroit DTT     5187000 4.428   20.891  376     -       
New York        NYC     19550000        4.912   21.919  423     -       
Boston  BOS     5455000 5.347   30.106  340     -       
Seattle SEA     2970000 6.324   12.105  179     -       
Chicago CHI     8240000 6.378   26.431  421     -       
San Francisco   SFO     6253000 6.575   35.339  351     -       
Houston HOU     3731000 7.49    20.885  417     -       
Columbus        CMH     1345000 7.726   19.924  351     -       
Dallas-Ft. Worth        DFW     4037000 13.068  22.705  404     -       
Phoenix PHX     2238000 15.445  50.526  427     -       
Los Angeles     LAX     14532000        23.117  150.429 460     -       
Salt Lake City  SLC     1072000 47.801  227.832 297     -       
San Diego       SAN     2498000 85.378  153.109 14      -       
Portland        PDX     1793000 89.043  241.734 425     -
        
                        11.53   79.068  9451


The bottom line is that there is some slight variation, but as I predicted,
not much.  And as it so happens, it was generally in the wrong direction. 
Nike was a little better on the mean and a little worse on the standard
deviation.  Forbes was a little worse (in fractions) from Digex on the
mean, and more so on the deviation.  Digex's original figures for the April
20- May 20 period were 9.162 seconds on the mean and 31.752 on the standard
deviation - slightly better than average.  Note that 30 days and five days
are apples and oranges if you comprendo fruit.  The means for the five day
period were roughly 9.9, 11.3, and 11.5 with the "weeny" Digex server in
the middle.

So yeah, servers do have an impact, but not nearly what you had hoped and
believed. I would say miniscule in a universe where our results ran from
1.5 to 26.8 seconds.  And while I'm not shy about "I told you so's" my real
reason for putting this out is that I have heard from several backbones
that are scrambling to upgrade and move their home page servers etc.  I
personally would get them up to what you think they ought to be anyway, but
if you go to extraordinary measures, you're probably going to be
disappointed in how little the numbers move - as I predicted.  It just
won't move the numbers much.  A little perhaps, and if you're not careful -
potentially the wrong direction.  Doug was pretty emphatic that these
customer servers were the "good" ones on the good part of the net, and the
home server was the weeny one.  Maybe Doug Mohney can jump in and remind me
which was which as far as NT and UNIX goes if anyone is interested.  I
would guess off hand that Forbes is taking a little more load than Nike,
but I may be reading messages from God in standard deviation cloud
formations.  There just isn't that much difference - certainly not in the
mean.

The bottom line is that if we are actually measuring server performance, we
should be able to measure three different servers, two avowed muscle boxes
and one avowed weeny one, all on the same network (connected differently
I'm told) and get at least as wide a variation as we saw between networks. 
We didn't by about a mile and a fortnight.  At LEAST it ought to be in the
predicted direction.  This clearly was not.  Good theory - but not so -
even in the lab.

Again, I think you guys should take a look at this stuff a little more open
mindedly and professionally.  It's certainly NOT to scientific laboratory
standards, but it is certainly interesting and I would claim very VALID
information.  Better information than you have previously had at your
disposal.  It's an attempt to look at the FOREST, not tree limb diameters,
leaf patterns and nutrient flows - all very interesting though those may be
I do grant you.  A great deal of this network has operated on theories that
once scaled up, nobody really knows if they work that way or not.  I can
tell you from personal experience that most of what I know is wrong, and I
find that out over, and over, and over again.  I might also mention that a
lot of what I'm told turns out to be wrong as well.  I'm only going to
SUGGEST that I may not be alone.

We'll continue to work on it.  I discussed the universal test page
suggestion with Gene Shklar this afternoon and we will make it so.  Again,
I don't think it will move any numbers around much, but certainly, as
Forest Gump says, ONE LESS THING....  And if you can make a case for a
different server ON YOUR OWN NETWORK, we will certainly entertain requests
to shoot at another machine.  Nominally July 15-August 15th though I'm not
signing up to those precise dates at this time.  

Regards


===================================================================
Jack Rickard                                    Boardwatch Magazine
Editor/Publisher                    8500 West Bowles Ave., Ste. 210
jack.rickard () boardwatch com                     Littleton, CO 80123
www.boardwatch.com                             Voice: (303)973-6038
===================================================================