nanog mailing list archives
RE: Best utilizing fat long pipes and large file transfer
From: "Buhrmaster, Gary" <gtb () slac stanford edu>
Date: Thu, 12 Jun 2008 17:28:52 -0700
Hi, I'm looking for input on the best practices for sending large files
There are both commercial products (fastcopy) and various "free"(*) products (bbcp, bbftp, gridftp) that will send large files. While they can take advantage of larger windows they also have the capability of using multiple streams (dealing with the inability to tune the tcp stack). There are, of course, competitors to these products which you should look into. As always, YMWV. Some references: http://www.softlink.com/fastcopy_techie.html (Some parts of NASA seem to like fastcopy) http://nccs.gov/user-support/general-support/data-transfer/bbcp/ (Full disclosure, bbcp was written by someone who sits about 3 meters from where I am sitting, but I cannot find a nice web reference from him about the product, so I am showing a different sites documentation) http://doc.in2p3.fr/bbftp/ (One of the first to use multistream for BaBar) http://www.globus.org/grid_software/data/gridftp.php (Part of the globus toolkit. Somewhat heavier weight if all you want is file transfer.) http://fasterdata.es.net/tools.html (A reference I am surprised Kevin did not point to :-) http://www.slac.stanford.edu/grp/scs/net/talk/ggf5_jul2002/NMWG_GGF5.pdf (A few year old performance evaluation) www.triumf.ca/canarie/amsterdam-test/References/010402-ftp.ppt (Another older performance evaluation) Gary (*) Some are GPL, and some (modified) BSD licenses. Which one is "free enough" depends on some strongly held beliefs of the validator.
Current thread:
- Re: Best utilizing fat long pipes and large file transfer, (continued)
- Re: Best utilizing fat long pipes and large file transfer Kevin Oberman (Jun 13)
- Re: Best utilizing fat long pipes and large file transfer Robert Boyle (Jun 13)
- Re: Best utilizing fat long pipes and large file transfer Deepak Jain (Jun 13)
- Re: Best utilizing fat long pipes and large file transfer goemon (Jun 13)
- Re: Best utilizing fat long pipes and large file transfer Glen Turner (Jun 15)
- Re: Best utilizing fat long pipes and large file transfer goemon (Jun 16)
- Re: Best utilizing fat long pipes and large file transfer Bob Bradlee (Jun 16)
- Re: Best utilizing fat long pipes and large file transfer Jay R. Ashworth (Jun 22)
- Re: Best utilizing fat long pipes and large file transfer Kevin Oberman (Jun 13)
- Re: Best utilizing fat long pipes and large file transfer Kevin Oberman (Jun 13)
- comcast Thompson, Taeko (Jun 12)
- Re: comcast Matt Palmer (Jun 12)
- Re: comcast Randy Bush (Jun 12)
- Re: comcast Steve Pirk (Jun 12)
- RE: comcast ekagan (Jun 12)
- Re: comcast Steven M. Bellovin (Jun 12)