Re: Best utilizing fat long pipes and large file transfer

["Kevin Oberman" <oberman () es net>]

> Date: Thu, 12 Jun 2008 19:26:56 -0400
> From: Robert Boyle <robert () tellurian com>
>
> At 06:37 PM 6/12/2008, you wrote:
> > I'm looking for input on the best practices for sending large files 
> > over a long fat pipe between facilities (gigabit private circuit, ~20ms RTT).
> > I'd like to avoid modifying TCP windows and options on end hosts 
> > where possible (I have a lot of them). I've seen products that work 
> > as "transfer stations" using "reliable UDP" to get around the 
> > windowing problem.
> >
> > I'm thinking of setting up servers with optimized TCP settings to 
> > push big files around data centers but I'm curious to know how 
> > others deal with LFN+large transfers.
>
> In our experience, you can't get to line speed with over 20-30ms of 
> latency using TCP regardless of how much you tweak it. We transfer 
> files across the US with 60-70ms at line speeds with UDP based file 
> transfer programs. There are a number of open source projects out 
> there designed for this purpose.

Clearly you have failed to try very hard or to check into what others
have done. We routinely move data at MUCH higher rates over TCP at
latencies over 50 ms. one way (>100 ms. RTT). We find it fairly easy to
move data at over 4 Gbps continuously.

If you can't fill a GE to 80% (800 Mbps) at 30 ms, you really are not
tying very hard. Note: I am talking about a single TCP stream running
for over 5 minutes at a time on tuned systems. Tuning for most modern
network stacks is pretty trivial. Some older stacks (e.g. FreeBSD V6)
are hopeless. I can't speak to how Windows does as I make no use of it
for high-speed bulk transfers.

It's also fairly easy to run multiple parallel TCP streams to completely
fill a 10 Gbps pipe at any distance. This is the preferred method for
moving large volumes of data around the world in the R&E community as
it requires little or no system tuning and is available in several
open-source tools.
-- 
R. Kevin Oberman, Network Engineer
Energy Sciences Network (ESnet)
Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab)
E-mail: oberman () es net                       Phone: +1 510 486-8634
Key fingerprint:059B 2DDF 031C 9BA3 14A4  EADA 927D EBB3 987B 3751

Attachment: _bin
Description: