Is there a limit on number of tcp/ip connections between machines on linux?
I have a very simple program written in 5 min that opens a sever socket and loops through the request and prints to the screen the bytes sent to it. I then tried to benchmark how many connections I can hammer it with to try to find out how many concurrent users I can support with this program. On another machine (where the network between them is not saturated) I created a simple program that goes into a loop and connects to the server machine and send the bytes «hello world». When the loop is 1000-3000 the client finishes with all requests sent. When the loop goes beyond 5000 it starts to have time outs after finish the first X number of requests. Why is this? I have made sure to close my socket in the loop. Can you only create so many connections within a certain period of time? Is this limit only applicable between the same machines and I need not worry about this in production where 5000+ requests are all coming from different machines?
you can monitor your sockets using ss -s command. And follow the steps to increase socket limit if needed
you can reuse TIMED_WAIT sockets like: s = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0) s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
6 Answers 6
There is a limit, yes. See ulimit .
In addition, you need to consider the TIME_WAIT state. Once a TCP socket is closed (by default) the port remains occupied in TIME_WAIT status for 2 minutes. This value is tunable. This will also «run you out of sockets» even though they are closed.
Run netstat to see the TIME_WAIT stuff in action.
P.S. The reason for TIME_WAIT is to handle the case of packets arriving after the socket is closed. This can happen because packets are delayed or the other side just doesn’t know that the socket has been closed yet. This allows the OS to silently drop those packets without a chance of «infecting» a different, unrelated socket connection.
I just checked with netstat on both machines, and there are indeed a ton of TIMED_WAIT on the client side but no TIMED_WAIT on the server side. Is this the bahaviour you are describing? Assuming yes: 1) Does this mean this won’t be an issue in production because the limit seems to be coming from the client side (running out of socket) and not he server side (where no sockets are created) 2) Is there a way to get around this so I can test my server with load similar to production?
The behavior of TIMED_WAIT is OS-specific. Yes you can get around it — it’s possible to change the TIMED_WAIT timeout e.g. from 120 seconds to 30 or even less.
When looking for the max performance you run into a lot of issue and potential bottlenecks. Running a simple hello world test is not necessarily going to find them all.
Possible limitations include:
- Kernel socket limitations: look in /proc/sys/net for lots of kernel tuning..
- process limits: check out ulimit as others have stated here
- as your application grows in complexity, it may not have enough CPU power to keep up with the number of connections coming in. Use top to see if your CPU is maxed
- number of threads? I’m not experienced with threading, but this may come into play in conjunction with the previous items.
Is your server single-threaded? If so, what polling / multiplexing function are you using?
Using select() does not work beyond the hard-coded maximum file descriptor limit set at compile-time, which is hopeless (normally 256, or a few more).
poll() is better but you will end up with the scalability problem with a large number of FDs repopulating the set each time around the loop.
epoll() should work well up to some other limit which you hit.
10k connections should be easy enough to achieve. Use a recent(ish) 2.6 kernel.
How many client machines did you use? Are you sure you didn’t hit a client-side limit?
On my system hardcoded select limit was 1024 and it is indeed impossible to go above this (the limit is imposed by a data type that holds the map of file descriptors to watch).
The quick answer is 2^16 TCP ports, 64K.
The issues with system imposed limits is a configuration issue, already touched upon in previous comments.
The internal implications to TCP is not so clear (to me). Each port requires memory for it’s instantiation, goes onto a list and needs network buffers for data in transit.
Given 64K TCP sessions the overhead for instances of the ports might be an issue on a 32-bit kernel, but not a 64-bit kernel (correction here gladly accepted). The lookup process with 64K sessions can slow things a bit and every packet hits the timer queues, which can also be problematic. Storage for in transit data can theoretically swell to the window size times ports (maybe 8 GByte).
The issue with connection speed (mentioned above) is probably what you are seeing. TCP generally takes time to do things. However, it is not required. A TCP connect, transact and disconnect can be done very efficiently (check to see how the TCP sessions are created and closed).
There are systems that pass tens of gigabits per second, so the packet level scaling should be OK.
There are machines with plenty of physical memory, so that looks OK.
The performance of the system, if carefully configured should be OK.
The server side of things should scale in a similar fashion.
I would be concerned about things like memory bandwidth.
Consider an experiment where you login to the local host 10,000 times. Then type a character. The entire stack through user space would be engaged on each character. The active footprint would likely exceed the data cache size. Running through lots of memory can stress the VM system. The cost of context switches could approach a second!
What is the theoretical maximum number of open TCP connections that a modern Linux box can have
Assuming infinite performance from hardware, can a Linux box support >65536 open TCP connections? I understand that the number of ephemeral ports (<65536) limits the number of connections from one local IP to one port on one remote IP. The tuple (local ip, local port, remote ip, remote port) is what uniquely defines a TCP connection; does this imply that more than 65K connections can be supported if more than one of these parameters are free. e.g. connections to a single port number on multiple remote hosts from multiple local IPs. Is there another 16 bit limit in the system? Number of file descriptors perhaps?
3 Answers 3
A single listening port can accept more than one connection simultaneously.
There is a ’64K’ limit that is often cited, but that is per client per server port, and needs clarifying.
Each TCP/IP packet has basically four fields for addressing. These are:
source_ip source_port destination_ip destination_port
Inside the TCP stack, these four fields are used as a compound key to match up packets to connections (e.g. file descriptors).
If a client has many connections to the same port on the same destination, then three of those fields will be the same — only source_port varies to differentiate the different connections. Ports are 16-bit numbers, therefore the maximum number of connections any given client can have to any given host port is 64K.
However, multiple clients can each have up to 64K connections to some server’s port, and if the server has multiple ports or either is multi-homed then you can multiply that further.
So the real limit is file descriptors. Each individual socket connection is given a file descriptor, so the limit is really the number of file descriptors that the system has been configured to allow and resources to handle. The maximum limit is typically up over 300K, but is configurable e.g. with sysctl.
The realistic limits being boasted about for normal boxes are around 80K for example single threaded Jabber messaging servers.
How many open udp or tcp/ip connections can a linux machine have?
There are limits imposed by available memory, bandwidth, CPU, and of course, the network connectivity. But those can often be scaled vertically. Are there any other limiting factors on linux? Can they be overcome without kernel modifications? I suspect that, if nothing else, the limiting factor would become the gigabit ethernet. But for efficient protocols it could take 50K concurrent connections to swamp that. Would something else break before I could get that high? I’m thinking that I want a software udp and/or tcp/ip load balancer. Unfortunately nothing like that in the open-source community seems to exist, except for the http protocol. But it is not beyond my abilities to write one using epoll. I expect it would go through a lot of tweaking to get it to scale, but that’s work that can be done incrementally, and I would be a better programmer for it.
4 Answers 4
The one parameter you will probably have some difficulty with is jitter. Has you scale the number of connections per box, you will undoubtedly put strain on all the resources of the said system. As a result, the jitter characteristics of the forwarding function will likely suffer.
Depending on your target requirements, that might or not be an issue: if you plan to support mainly elastic traffic (traffic which does not suffer much from jitter and latency) then it’s ok. If the proportion of inelastic traffic is high (e.g. interactive voice/video), then this might be more of an issue.
Of course you can always over engineer in this case 😉
You raise a good point about jitter and latency and the effect on inelastic traffic (i.e. specifically the kind of traffic you would use UDP for.)
would the person that down-voted my post care to explain? drive-by down-voting without comment is just plain rude.
For TCP, the other concern is the amount of incoming data. Incoming data occupies kernel buffers until it is processed by a user process. If your application doesn’t process the memory «fast enough», then the kernel can run out of buffers and panic. This can be ameliorated by setting a small Rx buffer size on each socket.
. and a small Rx buffer size in turn might run yourself in «underrun» conditions. but hey, nothing is free 🙂
If you intend to have a server which holds one socket open per client, then it needs to be designed carefully so that it can efficiently check for incoming data from 10k+ clients. This is known as the 10k problem.
Modern Linux kernels can handle a lot more than 10k connections, generally at least 100k. You may need some tuning, particularly the many TCP timeouts (if using TCP) to avoid closing / stale sockets using up lots of resource if a lot of clients connect and disconnect frequently.
If you are using netfilter’s conntrack module, that may also need tuning to track that many connections (this is independent of tcp/udp sockets).
There are lots of technologies for load balancing, the most well-known is LVS (Linux Virtual Server) which can act as the front end to a cluster of a real servers. I don’t know how many connections it can handle, but I think we use it with at least 50k in production.