Max number of socket on Linux
It seems that the server is limited at ~32720 sockets. I have tried every known variable change to raise up this limit. But the server stay limited at 32720 opened socket, even if there is still 4Go of free memory and 80% of idle cpu. Here’s the configuration
~# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 63931 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 798621 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 2048 cpu time (seconds, -t) unlimited max user processes (-u) 63931 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited net.netfilter.nf_conntrack_max = 999999 net.ipv4.netfilter.ip_conntrack_max = 999999 net.nf_conntrack_max = 999999 Just so it’s said: If you need more than 32000 sockets at once, you have bigger problems than just that number being too low. A normal server doesn’t ever have more than a few hundred sockets (maybe even a couple thousand, for a busy server) open at once.
@TheSquad: do you have some security framework loaded, that limits the number of fd’s and/or connections?
Experience. Even extremely busy web sites rarely serve more than a couple thousand simultaneous clients — once they get to that point, they’re clustered or otherwise distributed to reduce load. And the QuakeNet IRC network, the best example i could think of for mass long-lived TCP client/server stuff, has maybe 80k simultaneous users spread over 40+ servers. That’s about 2k per.
@mvds: The limit is most likely not due to security stuff — security would kick in WAY before 32k sockets.
8 Answers 8
If you’re dealing with openssl and threads, go check your /proc/sys/vm/max_map_count and try to raise it.
In IPV4, the TCP layer has 16 bits for the destination port, and 16 bits for the source port.
Seeing that your limit is 32K I would expect that you are actually seeing the limit of outbound TCP connections you can make. You should be able to get a max of 65K sockets (this would be the protocol limit). This is the limit for total number of named connections. Fortunately, binding a port for incoming connections only uses 1. But if you are trying to test the number of connections from the same machine, you can only have 65K total outgoing connections (for TCP). To test the amount of incoming connections, you will need multiple computers.
Note: you can call socket(AF_INET. ) up to the number of file descriptors available, but you cannot bind them without increasing the number of ports available. To increase the range, do this:
echo «1024 65535» > /proc/sys/net/ipv4/ip_local_port_range (cat it to see what you currently have—the default is 32768 to 61000)
Perhaps it is time for a new TCP like protocol that will allow 32 bits for the source and dest ports? But how many applications really need more than 65 thousand outbound connections?
The following will allow 100,000 incoming connections on linux mint 16 (64 bit) (you must run it as root to set the limits)
#include #include #include #include #include #include void ShowLimit() < rlimit lim; int err=getrlimit(RLIMIT_NOFILE,&lim); printf("%1d limit: %1ld,%1ld\n",err,lim.rlim_cur,lim.rlim_max); >main() < ShowLimit(); rlimit lim; lim.rlim_cur=100000; lim.rlim_max=100000; int err=setrlimit(RLIMIT_NOFILE,&lim); printf("set returned %1d\n",err); ShowLimit(); int sock=socket(AF_INET,SOCK_STREAM,IPPROTO_TCP); sockaddr_in maddr; maddr.sin_family=AF_INET; maddr.sin_port=htons(80); maddr.sin_addr.s_addr=INADDR_ANY; err=bind(sock,(sockaddr *) &maddr, sizeof(maddr)); err=listen(sock,1024); int sockets=0; while(true) < sockaddr_in raddr; socklen_t rlen=sizeof(raddr); err=accept(sock,(sockaddr *) &raddr,&rlen); if(err>=0) < ++sockets; printf("%1d sockets accepted\n",sockets); >> > Increasing the maximum number of TCP/IP connections in Linux
I am programming a server and it seems like my number of connections is being limited since my bandwidth isn’t being saturated even when I’ve set the number of connections to «unlimited». How can I increase or eliminate a maximum number of connections that my Ubuntu Linux box can open at a time? Does the OS limit this, or is it the router or the ISP? Or is it something else?
@Software Monkey: I answered this anyway because I hope this might be useful to someone who actually is writing a server in the future.
@derobert: I saw that +1. Actually, I had the same thought after my previous comment, but thought I would let the comment stand.
5 Answers 5
Maximum number of connections are impacted by certain limits on both client & server sides, albeit a little differently.
On the client side: Increase the ephermal port range, and decrease the tcp_fin_timeout
To find out the default values:
sysctl net.ipv4.ip_local_port_range sysctl net.ipv4.tcp_fin_timeout The ephermal port range defines the maximum number of outbound sockets a host can create from a particular I.P. address. The fin_timeout defines the minimum time these sockets will stay in TIME_WAIT state (unusable after being used once). Usual system defaults are:
This basically means your system cannot consistently guarantee more than (61000 — 32768) / 60 = 470 sockets per second. If you are not happy with that, you could begin with increasing the port_range . Setting the range to 15000 61000 is pretty common these days. You could further increase the availability by decreasing the fin_timeout . Suppose you do both, you should see over 1500 outbound connections per second, more readily.
To change the values:
sysctl net.ipv4.ip_local_port_range="15000 61000" sysctl net.ipv4.tcp_fin_timeout=30 The above should not be interpreted as the factors impacting system capability for making outbound connections per second. But rather these factors affect system’s ability to handle concurrent connections in a sustainable manner for large periods of «activity.»
Default Sysctl values on a typical Linux box for tcp_tw_recycle & tcp_tw_reuse would be
net.ipv4.tcp_tw_recycle=0 net.ipv4.tcp_tw_reuse=0 These do not allow a connection from a «used» socket (in wait state) and force the sockets to last the complete time_wait cycle. I recommend setting:
sysctl net.ipv4.tcp_tw_recycle=1 sysctl net.ipv4.tcp_tw_reuse=1 This allows fast cycling of sockets in time_wait state and re-using them. But before you do this change make sure that this does not conflict with the protocols that you would use for the application that needs these sockets. Make sure to read post «Coping with the TCP TIME-WAIT» from Vincent Bernat to understand the implications. The net.ipv4.tcp_tw_recycle option is quite problematic for public-facing servers as it won’t handle connections from two different computers behind the same NAT device, which is a problem hard to detect and waiting to bite you. Note that net.ipv4.tcp_tw_recycle has been removed from Linux 4.12.
On the Server Side: The net.core.somaxconn value has an important role. It limits the maximum number of requests queued to a listen socket. If you are sure of your server application’s capability, bump it up from default 128 to something like 128 to 1024. Now you can take advantage of this increase by modifying the listen backlog variable in your application’s listen call, to an equal or higher integer.
sysctl net.core.somaxconn=1024 txqueuelen parameter of your ethernet cards also have a role to play. Default values are 1000, so bump them up to 5000 or even more if your system can handle it.
ifconfig eth0 txqueuelen 5000 echo "/sbin/ifconfig eth0 txqueuelen 5000" >> /etc/rc.local Similarly bump up the values for net.core.netdev_max_backlog and net.ipv4.tcp_max_syn_backlog . Their default values are 1000 and 1024 respectively.
sysctl net.core.netdev_max_backlog=2000 sysctl net.ipv4.tcp_max_syn_backlog=2048 Now remember to start both your client and server side applications by increasing the FD ulimts, in the shell.
Besides the above one more popular technique used by programmers is to reduce the number of tcp write calls. My own preference is to use a buffer wherein I push the data I wish to send to the client, and then at appropriate points I write out the buffered data into the actual socket. This technique allows me to use large data packets, reduce fragmentation, reduces my CPU utilization both in the user land and at kernel-level.
Max number of socket on Linux
It seems that the server is limited at ~32720 sockets. I have tried every known variable change to raise up this limit. But the server stay limited at 32720 opened socket, even if there is still 4Go of free memory and 80% of idle cpu. Here’s the configuration
~# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 63931 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 798621 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 2048 cpu time (seconds, -t) unlimited max user processes (-u) 63931 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited net.netfilter.nf_conntrack_max = 999999 net.ipv4.netfilter.ip_conntrack_max = 999999 net.nf_conntrack_max = 999999 4 Answers 4
I have found whats was limiting everything :
Thanks to everyone who answered !
You are looking in the wrong place for this one; you’re not running into a user limit, you’re running into a system limit which is generally the 15th power of 2 on a 32-bit system, which is what I’m guessing your system to be. Check:
% cat /proc/sys/kernel/pid_max 32768 % But you can change that too; on a 32-bit machine that’d be 2**22 as an absolute upper limit, so:
% sudo bash -c 'echo 4194303 > /proc/sys/kernel/pid_max' % I’d be interested to know how you go.