Re: TCP time_wait and port exhaustion for servers

[Jon Lewis <jlewis () lewis org>]

On Wed, 5 Dec 2012, Ray Soucy wrote:

> So if I rebuild the kernel to use a 20 second timeout, then that 30000
> port pool can sustain 1500, and a 60000 port pool can sustain 3000
> connections per second.
>
> The software could be re-written to round-robin though IP addresses
> for outgoing requests, but trying to avoid that.

It's kind of a hack, but you don't have to rewrite the software to get 
different source IPs for different connections.  On linux, you could do 
the following:

*) Keep your normal default route
*) Configure extra IPs as aliases (eth0:0, eth0:1,...) on the proxy
*) Split up the internet into however many subnets you have proxy host IPs 
*) route each part of the internet to your default gateway tacking on "dev 
eth0:n".

This will make the default IP for reaching each subnet of the internet the 
IP from eth0:n.

Of course you probably won't get very good load balancing of connections 
over your IPs that way, but it's better than nothing and a really quick 
fix that would give you immediate additional capacity.

I was going to also suggest, that to get better balancing, you could 
periodically (for some relatively short period) rotate the internet subnet 
routes such that you'd change which parts of the internet were pointed at 
which dev eth0:n every so many seconds or minutes, but that's kind of 
annoying to people like me (similar to the problem I recently posted about 
with AT&T 3G data web proxy).  Having your software round robin the source 
IPs would probably introduce the same problem/effect.

----------------------------------------------------------------------
 Jon Lewis, MCP :)           |  I route
 Senior Network Engineer     |  therefore you are
 Atlantic Net                |
_________ http://www.lewis.org/~jlewis/pgp for PGP public key_________