Tcp window sizes linux

How to Adjust the TCP Window Size Limit on Linux

The maximum bandwidth that iPerf can push between a source and a destination depends on many factors. To name a few:

1. Interface: if you are using a Raspberry Pi that has a 10/100 interface, then the upper limit is 100 Mbps.
2. CPU capabilities: pushing several Gbps of traffic needs quite a bit of CPU performance.
3. Latency: the greater the latency between source and destination, the larger the TCP window size needs to be.

The interface and the CPU put hard limits on the maximum bandwidth you can achieve. You can’t bypass them unless you get new hardware.

However, given your link’s latency, you can adjust the TCP window size in order to achieve the desired bandwidth. But it can be tricky…

How to calculate maximum achievable TCP throughput

The maximum achievable throughput is equal to:

[TCP Window Size in bits] / [Latency in seconds] = [Throughput in bits per second]

The default window size that iPerf uses varies by machine type and operating system. If it’s 64 KBytes, with a modest latency of 10 ms, the maximum bandwidth you can get is 52.43 Mbits/sec. For 500 ms the maximum is 524 Kbits/sec.

The obvious option would be to increase the window size to a larger value and get up to, let’s say, 500 Mbps. For a 50 ms latency, the window size needs to be 3.1 Mbytes.

Indeed, iPerf gives you the option to set the window size, but if you try to set it to 3.1 or 8.0 Mbytes you may get the following warning from iPerf:

“TCP window size: 250 KByte (WARNING: requested 8.00 MByte)”

Oops… Why didn’t iPerf follow your command and used only a 250 Kbyte window?

The devil is in the details!

In this case, the devil is the operating system that has a hard limit on the TCP window size that an application can use. It looks like in this case the limit is 250 Kbytes. These limits exist for good reasons. Larger TCP windows take more system memory and if you have multiple applications running, using large windows they may bog down the system. Or the system may deny TCP connections if it is running out of memory.

In your case, if you have full control of the system and you know what you are doing, you may want to increase the OS limits. As an example, to increase those limits to 8 MBytes, run the following commands as root on a Debian Linux machine:

echo 'net.core.wmem_max=4194304' >> /etc/sysctl.conf echo 'net.core.rmem_max=12582912' >> /etc/sysctl.conf echo 'net.ipv4.tcp_rmem = 4096 87380 4194304' >> /etc/sysctl.conf echo 'net.ipv4.tcp_wmem = 4096 87380 4194304' >> /etc/sysctl.conf sysctl -p

Now if you request iPerf to use a 4 Mbyte TCP window size, you will see that your request will get fulfilled like it’s supposed to!

Spot VPN, ISP, WiFi issues and more with Netbeez

Источник

How to Adjust the TCP Window Size Limit on Linux

iPerf is a bandwidth testing tool that many networking folks use for stress testing their networks, but also for troubleshooting. For some iPerf basics, take a look at some of these past posts:

The maximum bandwidth that iPerf can push between a source and a destination depends on many factors.

1. Interface: if you are using a Raspberry Pi that has a 10/100 interface, then the upper limit is 100 Mbps.
2. CPU capabilities: pushing several Gbps of traffic needs quite a bit of CPU performance.
3. Latency: the greater the latency between source and destination, the larger the TCP window size needs to be.

The interface and the CPU put hard limits on the maximum bandwidth you can achieve. You can’t bypass them unless you get new hardware.

However, given your link’s latency, you can adjust the TCP window size in order to achieve the desired bandwidth. But it can be tricky…

The maximum achievable bandwidth is equal to:

[TCP Window Size in bits] / [Latency in seconds] = [Throughput in bits per second]

Here is a calculator for getting the TCP window size.

The default window size that iPerf uses varies by machine type and operating system. If it’s 64 KBytes, with a modest latency of 10 ms, the maximum bandwidth you can get is 52.43 Mbits/sec. For 50 ms the maximum is 10.49 Mbits/sec.

The obvious option would be to increase the window size to a larger value and get up to, let’s say, 500 Mbps. For a 50 ms latency, the window size needs to be 3.1 Mbytes.

Indeed, iPerf gives you the option to set the window size, but if you try to set it to 3.1 or 4.0 Mbytes you may get the following warning from iPerf:

“TCP window size: 416 KByte (WARNING: requested 4.00 MByte)”

Oops… Why didn’t iPerf follow your command and used only a 416 Kbyte window?

The devil is in the details!

In this case, the devil is the operating system that has a hard limit on the TCP window size that an application can use. It looks like in this case the limit is 416 Kbytes. These limits exist for good reasons. Larger TCP windows take more system memory and if you have multiple applications running, using large windows they may bog down the system. Or the system may deny TCP connections if it is running out of memory.

In your case, if you have full control of the system and you know what you are doing, you may want to increase the OS limits. As an example, to increase those limits to 4 MBytes, run the following commands as root on a Debian Linux machine:

echo 'net.core.wmem_max=4194304' >> /etc/sysctl.conf echo 'net.core.rmem_max=12582912' >> /etc/sysctl.conf echo 'net.ipv4.tcp_rmem = 4096 87380 4194304' >> /etc/sysctl.conf echo 'net.ipv4.tcp_wmem = 4096 87380 4194304' >> /etc/sysctl.conf sysctl -p

Now if you request iPerf to use a 4 Mbyte TCP window size, you will see that your request will get fulfilled like it’s supposed to!

Spot VPN, ISP, WiFi issues and more with Netbeez

Источник

How to Adjust TCP Window Size to Improve Network Performance

Are you seeing poor network performance but with link utilization that’s well below 100%? You might have an issue with your TCP window size. Here’s how—and why—to fix that.

The TCP/IP protocol

The TCP/IP protocol sometimes shows its age.

It was invented in an era when networks were very slow and packet loss was high. So one of the main considerations in early protocol design was reliability.

The Transmission Control Protocol (TCP) has built-in mechanisms for reliability that include validating a checksum on every packet, as well as detection and retransmission of dropped or out-of-order packets.

These features were invented when WAN bandwidth of 56Kbps was fast and packet drop rates of 1% were not uncommon.

Today’s broadband networks are many orders of magnitude faster, as well as vastly more reliable. So it shouldn’t come as a surprise that the bulletproof reliability mechanisms designed into the protocol sometimes cause problems. One of them has to do with a feature called TCP windowing.

What’s a TCP window?

A key reliability feature of TCP is the acknowledgement (ACK) packet. Device A sends a bunch of data to device B. Then device B validates all the packet-level checksums and sends an ACK packet that says everything was received correctly.

If everything isn’t received, some or all of the data needs to be retransmitted. Each device maintains a buffer of all the data just in case it needs to send it again. Receiving the ACK packet means the device can flush that old data out of the buffer.

The TCP window is the maximum number of bytes that can be sent before the ACK must be received.

If the network is unreliable, it’s better to keep the window small. This way you don’t have to retransmit as much data if there’s a problem. But if the network is reliable, then the window can be quite large.

One of the most clever features of TCP is the concept of a sliding window. Devices can change the window size dynamically, making it smaller when there’s congestion and bigger when things are clear.

Each device sends the other a suggested window size that says how much data it wants to receive before getting an acknowledgement. You can even have different window sizes for the two traffic directions in the same conversation.

Where TCP windowing goes wrong

TCP windowing is a very clever mechanism. But consider what happens on a network with very high latency and high bandwidth.

You can easily transmit an entire window’s worth of data before the first packet is even received at the other end. Then the sender stops and waits for the acknowledgement. Eventually the receiver receives the last packet in the burst and sends an acknowledgement—a single packet that has to cross the network—taking the same amount of time again.

Each device can only send packets in these relatively short bursts. Then it must wait for the acknowledgement from the other end.

The average amount of data getting through the network is a small fraction of the total bandwidth. In modern networks, drop rates are so low, this slow transmission rate isn’t justified. All it does is drag down network performance

How to fix TCP windowing

The TCP window size is controlled by the end devices, not by the routers, switches, or firewalls that happen to be in the middle. The devices actively and dynamically negotiate the window size throughout the session.

But as I mentioned earlier, the TCP mechanism was designed for network bandwidth that’s orders of magnitude slower than what we have today. So some implementations still enforce a maximum window size of 64KB. You can get around this by enabling TCP windows scaling, which allows windows of up to 1GB.

Windows scaling was introduced in RFC 1323 to solve the problem of TCP windowing on fast, reliable networks. It’s available as an option in any modern TCP implementation. The only question is whether it’s been enabled properly.

In all recent Microsoft Windows implementations, windows scaling is enabled by default. You ‘ll find places on the Internet telling you to change registry values to increase your window size, but depending on the Windows version you’re using, these changes will have no effect. The values may no longer even exist. Bottom line, you don’t need to fix TCP windowing in Windows, either clients or servers.

On Linux systems, you can check that full TCP window scaling is enabled by looking at the value in /proc/sys/net/ipv4/tcp_window_scaling.

On Cisco devices, you can adjust the the window size using the global configuration command, “ip tcp window-size”. This command only affects sessions to the Cisco device itself. Network devices generally won’t change the parameters for sessions that merely pass through them.

About Kevin Dooley

Kevin has 15+ years of experience as a network engineer. He has designed and implemented several of the largest and most sophisticated enterprise data networks in Canada and written several highly regarded books on networking for O’Reilly and Associates, including Designing Large-Scale LANs and Cisco IOS Cookbook. Kevin holds a Ph.D. in theoretical physics and numerous industry certifications.

Источник