nanog mailing list archives

Re: TCP time_wait and port exhaustion for servers


From: "Fred Baker (fred)" <fred () cisco com>
Date: Wed, 5 Dec 2012 22:06:29 +0000

If you want to get into software rewriting, the simplest thing I might come up with would be to put TCBs in some form 
of LRU list and, at a point where you need a port back, close the TCB that least recently did anything. My 
understanding is that this was implemented 15 years ago to manage SYN attacks, and could be built on to manage this 
form of "attack".

Or, change the period of time a TCB is willing to stay in time-wait. Instead of 60 seconds, make it 10.

On Dec 5, 2012, at 1:11 PM, Jon Lewis wrote:

On Wed, 5 Dec 2012, Ray Soucy wrote:

So if I rebuild the kernel to use a 20 second timeout, then that 30000
port pool can sustain 1500, and a 60000 port pool can sustain 3000
connections per second.

The software could be re-written to round-robin though IP addresses
for outgoing requests, but trying to avoid that.

It's kind of a hack, but you don't have to rewrite the software to get different source IPs for different 
connections.  On linux, you could do the following:

*) Keep your normal default route
*) Configure extra IPs as aliases (eth0:0, eth0:1,...) on the proxy
*) Split up the internet into however many subnets you have proxy host IPs *) route each part of the internet to your 
default gateway tacking on "dev eth0:n".

This will make the default IP for reaching each subnet of the internet the IP from eth0:n.

Of course you probably won't get very good load balancing of connections over your IPs that way, but it's better than 
nothing and a really quick fix that would give you immediate additional capacity.

I was going to also suggest, that to get better balancing, you could periodically (for some relatively short period) 
rotate the internet subnet routes such that you'd change which parts of the internet were pointed at which dev eth0:n 
every so many seconds or minutes, but that's kind of annoying to people like me (similar to the problem I recently 
posted about with AT&T 3G data web proxy).  Having your software round robin the source IPs would probably introduce 
the same problem/effect.

----------------------------------------------------------------------
Jon Lewis, MCP :)           |  I route
Senior Network Engineer     |  therefore you are
Atlantic Net                |
_________ http://www.lewis.org/~jlewis/pgp for PGP public key_________




Current thread: