Re: [pfSense Support] Strange TCP connection behavior 2.0 RC2 (+3)

[Eugen Leitl <eugen () leitl org>]

----- Forwarded message from William Salt <williamejsalt () googlemail com> -----

From: William Salt <williamejsalt () googlemail com>
Date: Wed, 29 Jun 2011 13:16:32 +0100
To: support () pfsense com
Subject: Re: [pfSense Support] Strange TCP connection behavior 2.0 RC2 (+3)
Reply-To: support () pfsense com

Hi all, thanks for the input.
We have now swapped the cards to em card at both ends, instead of igb at one
end, and em at the other. We are now seeing near gig speeds in both
directions. Before, we saw very different speeds in each direction.

We have now managed to reach around 860-900mbps each way with the following
values in our sysctl.conf:

kern.ipc.maxsockbuf=20971520
net.inet.tcp.recvbuf_max=20971520
net.inet.tcp.sendbuf_max=20971520
net.inet.tcp.recvbuf_inc=524288
net.inet.tcp.sendbuf_inc=524288

However, even though we can reach around the upper threshold of the
connection, we are seeing the boxes crash, or tcp performance hit the
miximum 860-900mbps then drop, and stick at around 8mbps, until a reboot.


I might add that we are running 32bit (i386) RC3 at both ends, with 6gb of
ram.(probably alot less in the OS, need to upgrade to x64)

When i replicated these settings on two fresh boxes beyong the routers at
either end, i saw no performance increase...

Regards
Will


On Tue, Jun 28, 2011 at 3:34 PM, Eugen Leitl <eugen () leitl org> wrote:
> ----- Forwarded message from Rhys Rhaven <rhys () rhavenindustrys com> -----
>
> From: Rhys Rhaven <rhys () rhavenindustrys com>
> Date: Tue, 28 Jun 2011 09:30:06 -0500
> To: nanog () nanog org
> Subject: Re: [pfSense Support] Strange TCP connection behavior 2.0 RC2
(+3)
> Organization: Rhaven Industrys
> User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
>        rv:1.9.2.17) Gecko/20110516 Lightning/1.0b2 Thunderbird/3.1.10
>
> Obviously not helping if you are trying to tune standard TCP, but I
> lament that protocols like Tsunami are not in wider use.
> http://tsunami-udp.sourceforge.net/ Short of it, a TCP control channel
> takes care of error checking and resends while the data channel is a UDP
> stream, specifically built to max out LFNs.
>
> On 06/28/2011 03:52 AM, Eugen Leitl wrote:
> > ----- Forwarded message from William Salt <williamejsalt () googlemail com>
-----
> > From: William Salt <williamejsalt () googlemail com>
> > Date: Tue, 28 Jun 2011 08:03:25 +0100
> > To: support () pfsense com
> > Subject: [pfSense Support] Strange TCP connection behavior 2.0 RC2 (+3)
> > Reply-To: support () pfsense com
> >
> > Hi All,
> >          For the last couple of months i have been pulling my hair out
> > trying to solve this problem.
> > We have a 1Gbps transatlantic link from the UK to the US, which has
> > successfully passed the RFC2544 test.
> >
> > At either end, we have a media converter, and a supermicro server with
an
> > intel quad port NIC running pfsense 2 (RC2 at one end RC3 at the other)
and
> > the IGB driver on the quad port.
> >
> > We can pass 1gbps either way with UDP. However we are experiencing very
> > strange issues with tcp connections.
> >
> > With window scaling enabled, and a max socket buffer set to 16MB, we see
no
> > difference.
> > Even disabling window scaling and setting the window to 16MB makes no
> > difference.
> >
> > Each TCP connection starts very slowly, and will max out at around
190mbps,
> > taking nearly 2 minutes to climb to this speed before *plateauing*.
> >
> > We have to initiate many (5+) connections to saturate the link with tcp
> > connections with iperf.
> >
> > Real world tests transferring files, max out at 100mbps, using multiple
> > connections.
> >
> > I have followed guides like this:
> > http://www.psc.edu/networking/projects/tcptune/#FreeBSD
> >
> > With no luck, and have tweaked, disabled, and enabled nearly every
relevant
> > sysctl parameter with no luck.
> >
> > Can anyone shed some light on this?
> >
> > I am now doubting the IGB driver, and am looking to swap out the cards
as a
> > last ditch effort.
> > However, we have tried different hardware (L3 switches, media convertes
+
> > laptops etc), and the symptoms still persist...
> > The only constant is freebsd 8.1 - pfsense (or 8.2 for our production
> > systems).
> > I have tried the freebsd net mailinglist, but im hoping you lot can help
me!
> > Cheers in advance
> > Will
> >
> > ----- End forwarded message -----
>
>
> ----- End forwarded message -----
> --
> Eugen* Leitl <a href="http://leitl.org";>leitl</a> http://leitl.org
> ______________________________________________________________
> ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org
> 8B29F6BE: 099D 78BA 2FD3 B014 B08A  7779 75B0 2443 8B29 F6BE
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: support-unsubscribe () pfsense com
> For additional commands, e-mail: support-help () pfsense com
>
> Commercial support available - https://portal.pfsense.org

----- End forwarded message -----
-- 
Eugen* Leitl <a href="http://leitl.org";>leitl</a> http://leitl.org
______________________________________________________________
ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org
8B29F6BE: 099D 78BA 2FD3 B014 B08A  7779 75B0 2443 8B29 F6BE