Re: help needed - state of california needs a benchmark

[Jeff Richmond <jeff.richmond () gmail com>]

Mike, nothing is perfect, so let's just start with that. What the FCC has done to measure this is to partner with Sam 
Knows and then have friendly DSL subs for the participating telcos to run modified CPE firmware to test against their 
servers. We have been collecting data for this for the past couple of months, actually. More can be found here:

http://www.samknows.com/broadband/fcc_and_samknows

While even that I have issues with, it certainly is better than hitting that speedtest site where anything at all 
problematic on the customer LAN side of the CPE can cause erroneous results.

Good luck,
-Jeff


On Jan 29, 2011, at 10:00 AM, Mike wrote:

> Hello,
>
>       My company is small clec / broadband provider serving rural communities in northern California, and we are the 
> recipient of a small grant from the state thru our public utilities commission. We went out to 'middle of nowhere' 
> and deployed adsl2+ in fact (chalk one up for the good guys!), and now that we're done, our state puc wants to gather 
> performance data to evaluate the result of our project and ensure we delivered what we said we were going to. Bigger 
> picture, our state is actively attempting to map broadband availability and service levels available and this data 
> will factor into this overall picture, to be used for future grant/loan programs and other support mechanisms, so 
> this really is going to touch every provider who serves end users in the state.
>
>       The rub is, that they want to legislate that web based 'speedtest.com' is the ONLY and MOST AUTHORITATIVE 
> metric that trumps all other considerations and that the provider is %100 at fault and responsible for making 
> fraudulent claims if speedtest.com doesn't agree. No discussion is allowed or permitted about sync rates, packet 
> loss, internet congestion, provider route diversity, end user computer performance problems, far end congestion 
> issues, far end server issues or cpu loading, latency/rtt, or the like. They are going to decide that the quality of 
> any provider service, is solely and exclusively resting on the numbers returned from 'speedtest.com' alone, period.
>
>       All of you in this audience, I think, probably immediately understand the various problems with such an 
> assertion. Its one of these situations where - to the uninitiated - it SEEMS LIKE this is the right way to do this, 
> and it SEEMS LIKE there's some validity to whats going on - but in practice, we engineering types know it's a far 
> different animal and should not be used for real live benchmarking of any kind where there is a demand for 
> statistical validity.
>
>       My feeling is that - if there is a need for the state to do benchmarking, then it outta be using statistically 
> significant methodologies for same along the same lines as any other benchmark or test done by other government 
> agencies and national standards bodies that are reproducible and dependable. The question is, as a hotbutton issue, 
> how do we go about getting 'the message' across, how do we go about engineering something that could be considered 
> statistically relevant, and most importantly, how do we get this to be accepted by non-technical legislators and 
> regulators?
>
> Mike-