Linux socket buffer sizes

How Does a Linux Socket Buffer Overflow

If you want see your buffer size in terminal, you can take a look at:

They contain three numbers, which are minimum, default and maximum memory size values (in byte), respectively.

How the buffering work in socket on linux

For UDP socket client will never know — the server side will just start dropping packets after the receive buffer is filled.

TCP, on the other hand, implements flow control. The server’s kernel will gradually reduce the window, so the client will be able to send less and less data. At some point the window will go down to zero. At this point the client fills up its send buffer and receives an error from the send(2) .

What will happen if the socket buffer size is not enough

If there is not enough space in the buffer data will be discarded. In the case of TCP this will result in the peer sending the data again since they were not ack’ed. In case of UDP the data are simply lost. TCP will also advertise its receive window so that the peer will hopefully not send the data faster than they can be processed (i.e. taken out of the buffer).

Are there any side effects to increasing the socket buffer size in Linux?

The only side-effect is memory usage. Increase them gradually and monitor the system. As long as you leave enough memory for existing processes you should be golden.

What happens on buffer overflow?

Short answer is this. «send» calls on a TCP socket will just block until the TCP sliding window (or internal queue buffers) opens up as a result of the remote endpoint receiving and consuming data. It’s not much different than trying to write bytes to a file faster than the disk can save it.

If your socket is configured for non-blocking mode, send will return EWOULDBLOCK or EAGAIN, until data can be sent. Standard poll, select, and epoll calls will work as expected so you know when to «send» again.

What are the differences between Kernel Buffer, TCP Socket Buffer and Sliding Window

Linux does not handle TCP’s sliding window as a separate buffer, rather as several indices indicating how much has already been received / read. The Linux kernel packet handling process can be described in many ways and can be divided to small parts as yo go deeper, but the general flow is as follows:

1. The kernel prepares to receive data over a network interface, it prepares SKB (Socket Buffer) data structures and map them to the interface Rx DMA buffer ring.
2. When packets arrive, they fill these preconfigured buffers and notify the kernel in an interrupt context of the packets arrival. In this context, the buffers are moved to a recv queue for the network stack to handle them out of an interrupt context.
3. The network stack retrieves these packets and handles them accordingly, eventually arriving to the TCP layer (if they are indeed TCP packets) which in turn handles the window.
4. See struct tcp_sock member u32 rcv_wnd which is then used in tp->rcvq_space.space as the per-connection space left in window.
5. The buffer is added to socket receive queue and is read accordingly as stream data in tcp_recvmsg()

The important thing to remember here is that copies is the worst thing regarding performance. Therefore, the kernel will always (unless absolutely necessary) will avoid copies and use pointers instead.

What happens in TCP when the internal buffer fills up

Can the internal network buffer fill up?

There are two internal buffers: the send buffer and the receive buffer. Both can fill up.

If so, then does the TCP protocol specify what will happen in this scenario?

There are no discarded packets. TCP does not discard packets in this circumstance. What happens when the send buffer fills depends on whether you are in blocking or non-blocking mode, or whether you are using an asynchronous API:

• blocking mode: the sender blocks
• non-blocking mode: the sender gets an error EAGAIN/EWOULDBLOCK
• asynchronous: the operation continues to be deferred.

treated as if they were lost in transit and just re-transmitted like any other lost packages?

Or are these packets truly lost

and this something I have to consider when desiging my own communication protocol on top of the TCP?

No. See above. But the various conditions are certainly something you have to consider in your implementation of the protocol.

Does the answer vary between operating systems?

For a record, the system above should have some kind of congestion control mechanism

TCP already has a congestion control mechanism.

which server could use to tell the client that it’s under heavy load

How? if the client isn’t reading, how can the server tell it anything?

or it’s free again. However, I’m curious to know how the TCP by the default would behave in this scenario.

Источник

What is the max size of AF_UNIX datagram message in Linux?

Currently I’m hitting a hard limit of 130688 bytes. If I try and send anything larger in one message I get a ENOBUFS error. I have checked the net.core.rmem_default , net.core.wmem_default , net.core.rmem_max , net.core.wmem_max , and net.unix.max_dgram_qlen sysctl options and increased them all but they have no effect because these deal with the total buffer size not the message size. I have also set the SO_SNDBUF and SO_RCVBUF socket options, but this has the same issue as above. The default socket buffer size are set based on the default socket options anyways. I’ve looked at the kernel source where ENOBUFS is returned in the socket stack, but it wasn’t clear to me where it was coming from. The only places that seem to return this error have to do with not being able to allocate memory. Is the max size actually 130688? If not can this be changed without recompiling the kernel?

That is a huge datagram. In my opinion, by the time that you have a datagram that large, you may as well have used TCP.

Yea, that doesn’t help. As I stated in the post, it won’t let you send a message over 130688 regardless of your wmem settings. I have them over 32MB and have tried many combinations below that.

Just to add to that. Its a misconception that the send buffers and receive buffers are for single messages. The buffer is the total kernel buffer for all messages. The wmem and qlen sysctl options actually will effect how and when send blocks. As the send buffer fills up (assuming no one receives), when the total bytes in the buffer would go beyond the buffer size or the total count will go beyond the qlen, send will block.

I get your point (and the question) better. Redacted confusing comment and upvoted; will explore around as time permits, as I’m interested in the answer as well.

I agree that its possible this is the hard limit. Just trying to find some proof and maybe some reasoning behind it.

Источник

What values may Linux use for the default unix socket buffer size?

Linux documents the default buffer size for tcp, but not for AF_UNIX («local») sockets. The value can be read (or written) at runtime.

cat /proc/sys/net/core/[rw]mem_default 

Is this value always set the same across different Linux kernels, or is there a range of possible values it could be?

1 Answer 1

The default is not configurable, but it is different between 32-bit and 64-bit Linux. The value appears to written so as to allow 256 packets of 256 bytes each, accounting for the different per-packet overhead (structs with 32-bit v.s. 64-bit pointers or integers).

On 64-bit Linux 4.14.18: 212992 bytes

On 32-bit Linux 4.4.92: 163840 bytes

The default buffer sizes are the same for both the read and write buffers. The per-packet overhead is a combination of struct sk_buff and struct skb_shared_info , so it depends on the exact size of these structures (rounded up slightly for alignment). E.g. in the 64-bit kernel above, the overhead is 576 bytes per packet.

/* Take into consideration the size of the struct sk_buff overhead in the * determination of these values, since that is non-constant across * platforms. This makes socket queueing behavior and performance * not depend upon such differences. */ #define _SK_MEM_PACKETS 256 #define _SK_MEM_OVERHEAD SKB_TRUESIZE(256) #define SK_WMEM_MAX (_SK_MEM_OVERHEAD * _SK_MEM_PACKETS) #define SK_RMEM_MAX (_SK_MEM_OVERHEAD * _SK_MEM_PACKETS) /* Run time adjustable parameters. */ __u32 sysctl_wmem_max __read_mostly = SK_WMEM_MAX; EXPORT_SYMBOL(sysctl_wmem_max); __u32 sysctl_rmem_max __read_mostly = SK_RMEM_MAX; EXPORT_SYMBOL(sysctl_rmem_max); __u32 sysctl_wmem_default __read_mostly = SK_WMEM_MAX; __u32 sysctl_rmem_default __read_mostly = SK_RMEM_MAX; 

Interestingly, if you set a non-default socket buffer size, Linux doubles it to provide for the overheads. This means that if you send smaller packets (e.g. less than the 576 bytes above), you won’t be able to fit as many bytes of user data in the buffer, as you had specified for its size.

Источник

Does Linux raw socket buffer size have upper limit of 256 K?

I am using following code in Centos to change raw socket buffer size to 400 KB, however I got same result as I set buffer size to 256 KB. Anything wrong? or this is the limitation of socket layer? The kernel version is 2.6.34. Thanks!

int rawsock; socklen_t socklen; int optval; int bufsize = 400 * 1024; rawsock = socket(PF_PACKET, SOCK_RAW, htons(ETH_P_ALL)); if (rawsock < 0) < my_log(LOG_ERR, "error creating raw socket"); return rawsock; >optval = 0; socklen = 4; err = getsockopt(rawsock, SOL_SOCKET, SO_RCVBUF, &optval, &socklen); bail_error(err); my_log("socket RX original buffer size = %d", optval); optval = 0; socklen = 4; err = getsockopt(rawsock, SOL_SOCKET, SO_SNDBUF, &optval, &socklen); bail_error(err); my_log("socket TX original buffer size = %d", optval); err = setsockopt(rawsock, SOL_SOCKET, SO_RCVBUF, &bufsize, sizeof(bufsize)); bail_error(err); err = setsockopt(rawsock, SOL_SOCKET, SO_SNDBUF, &bufsize, sizeof(bufsize)); bail_error(err); optval = 0; socklen = 4; err = getsockopt(rawsock, SOL_SOCKET, SO_RCVBUF, &optval, &socklen); bail_error(err); my_log("socket RX new buffer size = %d", optval); optval = 0; socklen = 4; err = getsockopt(rawsock, SOL_SOCKET, SO_SNDBUF, &optval, &socklen); bail_error(err); my_log("socket TX new buffer size = %d", optval); 

socket RX original buffer size = 110592 socket TX original buffer size = 110592 socket RX new buffer size = 524288 socket TX new buffer size = 524288 

Источник