Re: Low BW between Mountain View and OR -- why?

[Eygene Ryabinkin <rea+nanog () grid kiae ru>]

Tue, Feb 17, 2015 at 11:47:04AM +0530, Glen Kent wrote:
> I have a server in Mountain View and i am doing a speedtest with a
> server in Oregon. I see that the upload/download BW that i am
> getting is low -- around 10.0Mbps and 5.0Mbps.
>
> gkent@ubuntu:~/ics$ speedtest-cli --server 4082
> Retrieving speedtest.net configuration...
> Retrieving speedtest.net server list...
> Testing from Comcast Cable (50.250.251.210)...
> Hosted by Eastern Oregon Net, Inc. (La Grande, OR) [913.33 km]: 120.959 ms
> Testing download speed........................................
> Download: 5.08 Mbits/s
> Testing upload speed..................................................
> Upload: 10.89 Mbits/s
>
> When i check my connectivity with a server in NYC, its much better, though
> the server is much further away.
>
> gkent@ubuntu:~/ics$ speedtest-cli --server 2947
> Retrieving speedtest.net configuration...
> Retrieving speedtest.net server list...
> Testing from Comcast Cable (50.250.251.210)...
> Hosted by Atlantic Metro (New York City, NY) [4129.02 km]: 307.568 ms
> Testing download speed........................................
> Download: 38.52 Mbits/s
> Testing upload speed..................................................
> Upload: 10.62 Mbits/s
>
> I am trying to understand why this is so? I would wager that NYC being
> further away would give me a worse throughput than OR, but the speedtest
> tells me otherwise.

Packet loss or just congestion on the path (resulting in packet loss
again) that makes your TCP windows to shrink and not letting you to
get close to either your allocated BW on the channel to OR or your
maximal throughput per TCP stream that is governed by the maximal size
of the TCP buffer?  There could be some shaping as well, but this
might be not your case, since it fluctuates.  However, many interesting
things could happen.

iperf in UDP mode should give you fairly good overview of what't
happening with bandwidth and packet loss.  And iperf in TCP mode with
varying number of streams (and obtained scaling of throughput or lack
of it) should give you some hints on what's possibly going on.

I also assume that you're talking about TCP and your bandwidth-delay
product is used to tune TCP buffer sizes.  If not, I'd recommend
checking
  http://www.psc.edu/index.php/networking/641-tcp-tune

> The 2nd and more puzzling observation is that while OR is giving a
> download of around 5.08Mbps, it will improve and become much better
> later in the day. There are times when i see it going up as high as
> 48Mbps.
>
> Sometimes while a transfer is in progress i see that my download
> suddenly goes down from 48Mbps to 2Mbps.

Varying routing during the day, fluctuating congestion and many other
things could happen.  Probably here something like smokeping will give
you an overview of the RTT (if it varies) and loss on the ICMP.  ICMP
loss isn't neccessarily coupled to UDP/TCP due to the QoS and other
stuff that can happen on WAN, so it will provide just additional hints,
not the complete answer.

> Can somebody here tell me why such a drastic fluctuation is seen?

No answers here, sorry, just some hints and possibilities.
-- 
Eygene Ryabinkin, National Research Centre "Kurchatov Institute"

Always code as if the guy who ends up maintaining your code will be
a violent psychopath who knows where you live.