How to use SO_KEEPALIVE option properly to detect that the client at the other end is down?
I was trying to learn the usage of option SO_KEEPALIVE in socket programming in C language under Linux environment. I created a server socket and used my browser to connect to it. It was successful and I was able to read the GET request, but I got stuck on the usage of SO_KEEPALIVE. I checked this link keepalive_description@tldg.org but I could not find any example which shows how to use it. As soon as I detect the client’s request on accept() function I set the SO_KEEPALIVE option value 1 on the client socket. Now I don’t know, how to check if the client is down, how to change the time interval between the probes sent etc. I mean, how will I get the signal that the client is down? (Without reading or writing at the client — I thought I will get some signal when probes are not replied back from client), how should I program it after setting the option SO_KEEPALIVE on). Also if suppose the probes are sent every 3 secs and the client goes down in between I will not get to know that client is down and I may get SIGPIPE. Anyways importantly I wanna know how to use SO_KEEPALIVE in the code.
4 Answers 4
To modify the number of probes or the probe intervals, you write values to the /proc filesystem like
echo 600 > /proc/sys/net/ipv4/tcp_keepalive_time echo 60 > /proc/sys/net/ipv4/tcp_keepalive_intvl echo 20 > /proc/sys/net/ipv4/tcp_keepalive_probes Note that these values are global for all keepalive enabled sockets on the system, You can also override these settings on a per socket basis when you set the setsockopt, see section 4.2 of the document you linked.
You can’t «check» the status of the socket from userspace with keepalive. Instead, the kernel is simply more aggressive about forcing the remote end to acknowledge packets, and determining if the socket has gone bad. When you attempt to write to the socket, you will get a SIGPIPE if keepalive has determined remote end is down.
You’ll get notified when the status changes when you read from the socket. If the peer is determined to be dead due to the keepalives, select/poll will notifiy the socket as readable, and a read()/recv() will return an error. You should anyway always read or monitor a socket for reading though.
Yes, I think as far as writing, it just means you can get a SIGPIPE earlier, since the kernel detects the failiure sooner. If you are reading from the socket, it also allows read() to return an error instead of just ‘no data available’ as per @nos comment above. Getting the read event w. error on read from the select loop would probably be the best «early notice» from SO_KEEPALIVE
Note that keepalive won’t detect a failure until at least the configured keepalive_time + (keepalive_intrvl*keepalive_probes). I think by default if you don’t change the settings this can default to over an hour!
@nirudh Tomer The default keepalive probes are sent every 2 hours, and the peer is determinned dead if 9 probes with 75 seconds inbetween them all fail. How have you adjusted the defaults ? Note that tcp keepalives are not particularly designed to detect fast deaths of peers. If the peer is still reachable, send will fail, and that’s ok, you detected that it died! If somethings wrong with the network, fast detecting of a dead peer is not easy, nor usually desirable.
. If you really need a reliable way to detect this fast, you need to send heartbeats at the application level, and use some sensible timeouts on your reads/writes (note,that there’s no free lunch there, this always ends up with a lot of code to cover a lot of corner cases)
You’ll get the same result if you enable SO_KEEPALIVE, as if you don’t enable SO_KEEPALIVE — typically you’ll find the socket ready and get an error when you read from it.
You can set the keepalive timeout on a per-socket basis under Linux (this may be a Linux-specific feature). I’d recommend this rather than changing the system-wide setting. See the man page for tcp for more info.
Finally, if your client is a web browser, it’s quite likely that it will close the socket fairly quickly anyway — most of them will only hold keepalive (HTTP 1.1) connections open for a relatively short time (30s, 1 min etc). Of course if the client machine has disappeared or network down (which is what SO_KEEPALIVE is really useful for detecting), then it won’t be able to actively close the socket.
As already discussed, SO_KEEPALIVE makes the kernel more aggressive about continually verifying the connection even when you’re not doing anything, but does not change or enhance the way the information is delivered to you. You’ll find out when you try to actually do something (for example «write»), and you’ll find out right away since the kernel is now just reporting the status of a previously set flag, rather than having to wait a few seconds (or much longer in some cases) for network activity to fail. The exact same code logic you had for handling the «other side went away unexpectedly» condition will still be used; what changes is the timing (not the method).
Virtually every «practical» sockets program in some way provides non-blocking access to the sockets during the data phase (maybe with select()/poll(), or maybe with fcntl()/O_NONBLOCK/EINPROGRESS&EWOULDBLOCK, or if your kernel supports it maybe with MSG_DONTWAIT). Assuming this is already done for other reasons, it’s trivial (sometimes requiring no code at all) to in addition find out right away about a connection dropping. But if the data phase does not already somehow provide non-blocking access to the sockets, you won’t find out about the connection dropping until the next time you try to do something.
(A TCP socket connection without some sort of non-blocking behaviour during the data phase is notoriously fragile, as if the wrong packet encounters a network problem it’s very easy for the program to then «hang» indefinitely, and there’s not a whole lot you can do about it.)
Configuring TCP keepalive after accept
After calling getsockopt() for all four properties, everything seems alright. I’ve checked in Wireshark and there are no Keep alive packets being sent. I had to change SOL_TCP to IPPROTO_TCP , because when calling setsockopt() for TCP_KEEPIDLE it was returning errno 92 (Protocol not found). I’m doing the same thing after calling socket() for an outgoing connection, and it’s working perfectly. I’m using C and Linux. Is there any reason why setsockopt may not be working after an accept?
The values you’re setting may not be the actual values being used. The platform can adjust them. Use getsockopt() to find out what values are actually being used, and then see whether what’s happening agrees with that. NB I believe you can do all this once, on the listening socket, whence it will be inherited by all accepted sockets.
I tried it on the listening socket too, with no luck. I’ll double check the return values of getsockopt() . Thanks!
2 Answers 2
Here is a minimal working example. If it works for you, you can use it as a reference.
#include #include #include #include #include #include #include #include #include #define check(expr) if (!(expr)) < perror(#expr); kill(0, SIGTERM); >void enable_keepalive(int sock) < int yes = 1; check(setsockopt(sock, SOL_SOCKET, SO_KEEPALIVE, &yes, sizeof(int)) != -1); int idle = 1; check(setsockopt(sock, IPPROTO_TCP, TCP_KEEPIDLE, &idle, sizeof(int)) != -1); int interval = 1; check(setsockopt(sock, IPPROTO_TCP, TCP_KEEPINTVL, &interval, sizeof(int)) != -1); int maxpkt = 10; check(setsockopt(sock, IPPROTO_TCP, TCP_KEEPCNT, &maxpkt, sizeof(int)) != -1); >int main(int argc, char** argv) < check(argc == 2); struct sockaddr_in addr; addr.sin_family = AF_INET; addr.sin_port = htons(12345); check(inet_pton(AF_INET, argv[1], &addr.sin_addr) != -1); int server = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP); check(server != -1); int yes = 1; check(setsockopt(server, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof(int)) != -1); check(bind(server, (struct sockaddr*)&addr, sizeof(addr)) != -1); check(listen(server, 1) != -1); if (fork() == 0) < int client = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP); check(client != -1); check(connect(client, (struct sockaddr*)&addr, sizeof(addr)) != -1); printf("connected\n"); pause(); >else < int client = accept(server, NULL, NULL); check(client != -1); enable_keepalive(client); printf("accepted\n"); wait(NULL); >return 0; > Example output ( tcpdump reports keepalive packets every second):
$ ./a.out 127.0.0.1 & [1] 14010 connected accepted $ tcpdump -n -c4 -ilo port 12345 dropped privs to tcpdump tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on lo, link-type EN10MB (Ethernet), capture size 262144 bytes 18:00:35.173892 IP 127.0.0.1.12345 > 127.0.0.1.60998: Flags [.], ack 510307430, win 342, options [nop,nop,TS val 389745775 ecr 389745675], length 0 18:00:35.173903 IP 127.0.0.1.60998 > 127.0.0.1.12345: Flags [.], ack 1, win 342, options [nop,nop,TS val 389745775 ecr 389745075], length 0 18:00:36.173886 IP 127.0.0.1.12345 > 127.0.0.1.60998: Flags [.], ack 1, win 342, options [nop,nop,TS val 389745875 ecr 389745775], length 0 18:00:36.173898 IP 127.0.0.1.60998 > 127.0.0.1.12345: Flags [.], ack 1, win 342, options [nop,nop,TS val 389745875 ecr 389745075], length 0 4 packets captured 8 packets received by filter 0 packets dropped by kernel Захотелось записывать в блог, то чему обучаюсь.
Установить нужные значения можно следующими способами:
через procfs:
echo "3" > /proc/sys/net/ipv4/tcp_keepalive_probes
sysctl -w net.ipv4.tcp_keepalive_probes=3
setsockopt( sock, SOL_SOCKET, SO_KEEPALIVE, 1, sizeof(int) ); setsockopt( sock, SOL_TCP, TCP_KEEPCNT, 20, sizeof(int) ); setsockopt( sock, SOL_TCP, TCP_KEEPIDLE, 180, sizeof(int) ); setsockopt( sock, SOL_TCP, TCP_KEEPINTVL, 60, sizeof(int) );
Как это работает:
Рассмотрим на примере icq. Ваш клиент подключается к серверу загружает ваши данные и переходит в состояние ожидания входящих сообщений. Не смотря на то, что никакие данные в этот момент не передаются, соединения остается установленным. По истечении времени, установленного в tcp_keepalive_time, от последнего переданного/принятого пакета, система начинает проверять соединение. Дело в том, что если вдруг произошла ошибка, и доступ в интернет по какой-то причине пропал, ваш компьютер об этом никто не уведомит. Единственным вариантом остается самому проверить наличие соединения. И так, по истечении tcp_keepalive_time система отправляет специальный пакет по установленному соединению, если получен ответ — все в порядке, в противном случае система повторит попытку через tcp_keepalive_intvl секунд, и так tcp_keepalive_probes раз. После того как все попытки будут исчерпаны — разорвет соединение. Именно в этот момент icq выдаст сообщение о том, что связь прервана.
Время через которое будет разорвано соединение не будет превышать tcp_keepalive_time + (tcp_keepalive_intvl*tcp_keepalive_probes), что на моей машине примерно составляет 2 часа 11 минут.