Time wait · IT notes

Reduce TIME_WAIT connections

How many TIME_WAITs you have hanging out (run this like root):

netstat -na -p tcp | awk '{print $6}' | sort | uniq -c | sort -n

On Linux:

netstat -nat | awk '{print $6}' | sort | uniq -c | sort -n

Output will be something like:

# netstat -na -p tcp | awk '{print $6}' | sort | uniq -c | sort -n
   1
   1 Foreign
   1 LAST_ACK
   1 been
   3 LISTEN
  25 SYN_RCVD
1455 CLOSE_WAIT
17518 ESTABLISHED

The formula to calculate the value (net.inet.tcp.msl) to time of seconds is 2 times of the net.inet.tcp.msl value. which means the value of net.inet.tcp.msl with 30000 means 60000ms (because 2x30000), thus 60 seconds. In order set net.inet.tcp.msl to 15 seconds, change the value of net.inet.tcp.msl to 7500. for example:

sysctl net.inet.tcp.msl=7500

This will cause the TIME_WAIT sockets to terminate after waiting for 15 seconds, if no reply from the destination that this connection can be terminated.

This can be added to the /etc/sysctl.conf:

net.inet.tcp.fast_finwait2_recycle=1 # recycle FIN/WAIT states quickly
net.inet.tcp.finwait2_timeout=15000  # TCP FIN_WAIT_2 timeout waiting for client FIN packet before state close (default 60000, 60 sec)
net.inet.tcp.msl=2500                # Maximum Segment Lifetime, time the connection spends in TIME_WAIT state (default 30000, 2*MSL = 60 sec)

notice the 2500 it will be set to 5 seconds

On Linux edit /etc/sysctl.conf:

# Decrease TIME_WAIT seconds
net.ipv4.tcp_fin_timeout = 15

net.ipv4.tcp_tw_reuse = 1

net.ipv4.tcp_tw_recycle has been removed from Linux 4.12. https://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux

tcp state diagram

Transmission Control Protocol (TCP) is designed to be a bidirectional, ordered, and reliable data transmission protocol between two end points (programs). In this context, the term reliable means that it will retransmit the packets if it gets lost in the middle. TCP guarantees the reliability by sending back Acknowledgment (ACK) packets back for a single or a range of packets received from the peer.

This goes same for the control signals such as termination request/response. [RFC 793][1] defines the TIME-WAIT state to be as follows:

TIME-WAIT - represents waiting for enough time to pass to be sure the remote TCP received the acknowledgment of its connection termination request.

TCP is a bidirectional communication protocol, so when the connection is established, there is not a difference between the client and the server. Also, either one can call quits, and both peers needs to agree on closing to fully close an established TCP connection.

Let's call the first one to call the quits as the active closer, and the other peer the passive closer. When the active closer sends FIN, the state goes to FIN-WAIT-1. Then it receives an ACK for the sent FIN and the state goes to FIN-WAIT-2. Once it receives FIN also from the passive closer, the active closer sends the ACK to the FIN and the state goes to TIME-WAIT. In case the passive closer did not received the ACK to the second FIN, it will retransmit the FIN packet.

RFC 793 sets the TIME-OUT to be twice the Maximum Segment Lifetime, or 2MSL. Since MSL, the maximum time a packet can wander around Internet, is set to 2 minutes, 2MSL is 4 minutes. Since there is no ACK to an ACK, the active closer can't do anything but to wait 4 minutes if it adheres to the TCP/IP protocol correctly, just in case the passive sender has not received the ACK to its FIN (theoretically).

In reality, missing packets are probably rare, and very rare if it's all happening within the LAN or within a single machine.