Windows Server – Ephemeral Ports and Stale Sockets

We’ve been managing a mail server for a customer of ours for about 5 years now. We started them out, as their needs were very modest, on a Windows 2000 Server running ArgoSoft Mail Server and then early 2005, we migrated them to a newer system running Windows 2003 Server Web Edition running MDaemon from AltN.

We chose Web Edition as it was the least expensive of the Server product line and since we were adding third-party software, this system didn’t require any specific built-in Microsoft server components that are available with Standard, Enterprise or Small Business Server.

This system ran very nicely for quite some time as usage continued to grow.

Late 2006, the system had over 1500 active email addresses and over 500 active POP3 users. Over one particular weekend, many users started to see intermittent connection failures. After investigating the issue we came up with two particular issues that needed tweaking.

First, a bit background material. When an application based on the TCP/IP protocol wants to communicate with another system two pieces of information are required: the destination IP address and the port. Systems that provide a service listen on the Well Known Ports since a convention is needed for the particular protocol. For example, Simple Mail Transfer Protocol (SMTP), the language that mail servers use to transfer mail, uses the well known port 25.

A system that is initiating a connection to another that is listening on a well known port (like an email program on a client machine to a mail server) must also use a particular port at the source so that packets that are sent from the receiving end can properly reach back to the initiating program. These ports are usually picked at random and are only used for a very short duration and as such called the Ephemeral Ports. On a mail server, such ports will come into use when the mail server software wants to process certain tests for doing such things as spam processing, dns blacklist checking, anti-virus updates, etc… so a system can quickly eat a large number of these ports as necessary for mail processing. Also in delivery, ephemeral ports will be required for the outgoing SMTP connection.

Second, sockets that are closed (communication between the two ends is closed gracefully) do not immediately go back into the pool of available. These are in what is called the TIME_WAIT state This is so that reopening the connection to the client and server costs less than establishing a new connection. IBM: Windows Tuning

In the end, we decreased the TcpTimedWaitDelay value, so sockets could be recycled faster and we increased the MaxUserPort value, so we could have a larger pool of available sockets.

What was curious was that these values are essentially the same when the base OS is a Windows client version such as Windows XP. The base OS on this server was the “Web Edition”, so having to find and then tune the values for a server OS was a bit strange.

As they say, if you can’t measure it, you can’t manage it, so we setup two data sources and graphs for this system to monitor the active and stale socket connections that feed into cacti:

Since making these tweaks, the server has been humming along smoothly.