nanog mailing list archives
Re: TCP time_wait and port exhaustion for servers
From: Mark Andrews <marka () isc org>
Date: Thu, 06 Dec 2012 09:01:27 +1100
In message <CAP-guGW6oXo=UfTfg+SDiFjB4=qxPShO+YfK6vxnLkCC58PvgQ () mail gmail com>, William Herrin writes:
On Wed, Dec 5, 2012 at 2:55 PM, Ray Soucy <rps () maine edu> wrote:For each second that goes by you remove X addresses from the available pool of ports for new connections for whatever the TCP_TIMEWAIT_LEN is set to (60 seconds by default in Linux). In this case it's making quick connections for HTTP requests (most of which finish in less than a second). Say you have a pool of 30,000 ports and 500 new connections per second (typical): 1 second goes by you now have 29500 10 seconds go by you now have 25000 30 seconds go by you now have 15000 at 59 seconds you get to 29500, at 60 you get back 500 and stay at 29500 and that keeps rolling at 29500. Everyone is happy.The thing is, Linux doesn't behave quite that way. If you do an anonymous connect(), that is you socket() and then connect() without a bind() in the middle, then the limit applies *per destination IP:port pair*. So, you should be able to do 30,000 connections to 192.168.1.1 port 80, another 30,000 connections to 192.168.1.2 port 80, and so on.
The socket api is missing a bind + connect call which restricts the source address when making the connect. This is needed when you are required to use a fixed source address.
You should only fail if you A) bump against the top of NR_OPEN or B) try to do a massive number of TCP connections to the same remote IP address. Try it: set up a listener on discard that just closes the connection and repeat connect() to 127.0.0.5 until you get an error. Then confirm that you're out of ports: telnet 127.0.0.5 9 Trying 127.0.0.5... telnet: Unable to connect to remote host: Cannot assign requested address And confirm that you can still make outbound connections to a different IP address: telnet 127.0.0.4 9 Trying 127.0.0.4... Connected to 127.0.0.4. Escape character is '^]'. Connection closed by foreign host. Regards, Bill Herrin -- William D. Herrin ................ herrin () dirtside com bill () herrin us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka () isc org
Current thread:
- TCP time_wait and port exhaustion for servers Ray Soucy (Dec 05)
- Re: TCP time_wait and port exhaustion for servers JÁKÓ András (Dec 05)
- Re: TCP time_wait and port exhaustion for servers Ray Soucy (Dec 05)
- Re: TCP time_wait and port exhaustion for servers joel jaeggli (Dec 05)
- Re: TCP time_wait and port exhaustion for servers William Herrin (Dec 05)
- Re: TCP time_wait and port exhaustion for servers Owen DeLong (Dec 05)
- Re: TCP time_wait and port exhaustion for servers Ray Soucy (Dec 05)
- Re: TCP time_wait and port exhaustion for servers William Herrin (Dec 05)
- Re: TCP time_wait and port exhaustion for servers Mark Andrews (Dec 05)
- Re: TCP time_wait and port exhaustion for servers William Herrin (Dec 05)
- Re: TCP time_wait and port exhaustion for servers Mark Andrews (Dec 05)
- Re: TCP time_wait and port exhaustion for servers Ray Soucy (Dec 06)
- Re: TCP time_wait and port exhaustion for servers Miquel van Smoorenburg (Dec 05)
- Re: TCP time_wait and port exhaustion for servers Mark Andrews (Dec 05)
- Re: TCP time_wait and port exhaustion for servers Mark Andrews (Dec 05)
- Re: TCP time_wait and port exhaustion for servers William Herrin (Dec 05)
- Re: TCP time_wait and port exhaustion for servers Ray Soucy (Dec 05)
- Re: TCP time_wait and port exhaustion for servers JÁKÓ András (Dec 05)
- Re: TCP time_wait and port exhaustion for servers Jon Lewis (Dec 05)
- Re: TCP time_wait and port exhaustion for servers Fred Baker (fred) (Dec 05)
- Re: TCP time_wait and port exhaustion for servers David Conrad (Dec 05)