httperf buffer overflow detected when rate is high
I’m running httperf 0.9.0 (downloaded from Google Code) on Ubuntu 12.04.1 LTS 64-bit with 2CPU and 4GB RAM. I’m trying to benchmark web servers but encountered the following buffer overflow problem. Terminal command:
httperf --timeout=5 --client=0/1 --server=localhost --port=9090 --uri=/?value=benchmarks --rate=1200 --send-buffer=4096 --recv-buffer=16384 --num-conns=5000 --num-calls=10 *** buffer overflow detected ***: httperf terminated ======= Backtrace: ========= /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7f1f5efa1007] /lib/x86_64-linux-gnu/libc.so.6(+0x107f00)[0x7f1f5ef9ff00] /lib/x86_64-linux-gnu/libc.so.6(+0x108fbe)[0x7f1f5efa0fbe] httperf[0x404054] httperf[0x404e9f] httperf[0x406953] httperf[0x406bd1] httperf[0x40639f] httperf[0x4054d5] httperf[0x40285e] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f1f5eeb976d] httperf[0x4038f1] ======= Memory map: ======== . . 7f1f5fd74000-7f1f5fd79000 rw-p 00000000 00:00 0 7f1f5fd91000-7f1f5fd95000 rw-p 00000000 00:00 0 7f1f5fd95000-7f1f5fd96000 r--p 00022000 08:03 4849686 /lib/x86_64-linux-gnu/ld-2.15.so 7f1f5fd96000-7f1f5fd98000 rw-p 00023000 08:03 4849686 /lib/x86_64-linux-gnu/ld-2.15.so 7fff10452000-7fff10473000 rw-p 00000000 00:00 0 [stack] 7fff1054f000-7fff10550000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] Aborted (gdb) list 198 event_signal (EV_PERF_SAMPLE, 0, callarg); 199 200 /* prepare for next sample interval: */ 201 perf_sample_start = timer_now (); 202 timer_schedule (perf_sample, regarg, RATE_INTERVAL); 203 > 204 205 int 206 main (int argc, char **argv) 207 < (gdb) bt #0 0x00007f33d4643445 in raise () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f33d4646bab in abort () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x00007f33d4680e2e in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #3 0x00007f33d4716007 in __fortify_fail () from /lib/x86_64-linux-gnu/libc.so.6 #4 0x00007f33d4714f00 in __chk_fail () from /lib/x86_64-linux-gnu/libc.so.6 #5 0x00007f33d4715fbe in __fdelt_warn () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000404054 in set_active (s=, fdset=0x612bc0) at core.c:367 #7 0x0000000000404e9f in core_connect (s=0x17e7100) at core.c:980 #8 0x0000000000406953 in make_conn (arg=. ) at conn_rate.c:64 #9 0x0000000000406bd1 in tick (t=, arg=. ) at rate.c:94 #10 0x000000000040639f in timer_tick () at timer.c:104 #11 0x00000000004054d5 in core_loop () at core.c:1255 #12 0x000000000040285e in main (argc=11, argv=) at httperf.c:971 I tracked the source code a bit and found that FD_SET seems to be the cause. Finally, for lower rates (e.g. --rate=100 or --rate=500 ) httperf works fine. I'm benchmarking different web servers, and the rate causing a crash differs. My rates vary from 100 up to 1200. For more details, actually I'm trying to repeat the experiments done by Roberto Ostinelli, and I have already tuned TCP settings and applied the patch mentioned in his blog post. Any idea about what's causing this problem? Thanks!
4 Answers 4
You are trying to use an fd that is > 1024. With low loads you don't need/use that many fds. With high loads you need more fds and eventually get to 1024 and that causes the problem.
Even when I increase __FD_SETSIZE I get this problem, so I think there is actually a bug in whatever code is doing the bound check (gcc/llvm?)
As stated in my question, I have already applied the patch mentioned in the blog post, which is to increase fd related limits in header files.
So do you have any idea about how to fix this problem? I tried this in Ubuntu 10.04 32 bit and it works fine. Also your content may be suitable for a comment instead of an answer? 🙂
Newer versions of glibc do their own checks internally for FD_SET (called from httperf), and those checks fail, causing an abort. Even though httperf was built with a different __FD_SET_SIZE, glibc was still using the original one it was compiled with.
To get around this, I dug up an old version of sys/select.h and bits/select.h, from before __FD_ELT performed checks, and dropped them into httperf's src/ directory (in sys/ and bits). This way, httperf uses the old FD_SET macros that don't perform the checks that cause the abort. I used glibc-2.14.1, but any version that doesn't have bits/select2.h should do the trick.
A friend of mine is collecting this and other fixes to httperf for our own use (and yours!) at https://github.com/klueska/httperf
I had a similar crash and in my case replacing select() with epoll() solved it.
I was able to reproduce the problem only in release compilation (I'm using eclipse as devenv, I let eclipse set the compiler options both for debug and release).
1354813976/SBNotificationServer terminated 06/12 21:13:54 > ======= Backtrace: ========= 06/12 21:13:54 > /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7f4e10f90807] 06/12 21:13:54 > /lib/x86_64-linux-gnu/libc.so.6(+0x109700)[0x7f4e10f8f700] 06/12 21:13:54 > /lib/x86_64-linux-gnu/libc.so.6(+0x10a7be)[0x7f4e10f907be] 06/12 21:13:54 > 1354813976/SBNotificationServer[0x49db90] 06/12 21:13:54 > 1354813976/SBNotificationServer[0x49de05] 06/12 21:13:54 > 1354813976/SBNotificationServer[0x4a4b07] 06/12 21:13:54 > 1354813976/SBNotificationServer[0x4a5318] 06/12 21:13:54 > 1354813976/SBNotificationServer[0x4a2628] 06/12 21:13:54 > /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7f4e10c70e9a] 06/12 21:13:54 > /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f4e10f79cbd] 06/12 21:13:54 > ======= Memory map: ======== 06/12 21:13:54 > 00400000-00507000 r-xp 00000000 ca:01 141328 /usr/share/spotbros/SBNotServer/1354813976/SBNotificationServer 06/12 21:13:54 > 00706000-00707000 r--p 00106000 ca:01 141328 /usr/share/spotbros/SBNotServer/1354813976/SBNotificationServer 06/12 21:13:54 > 00707000-00708000 rw-p 00107000 ca:01 141328 /usr/share/spotbros/SBNotServer/1354813976/SBNotificationServer 06/12 21:13:54 > 00708000-0070d000 rw-p 00000000 00:00 0 06/12 21:13:54 > 0120d000-01314000 rw-p 00000000 00:00 0 [heap] 06/12 21:13:54 > 7f49f8000000-7f49f8021000 rw-p 00000000 00:00 0 . . After this, I was able to generate a core dump and I got:
warning: Can't read pathname for load map: Input/output error. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `./SBNotificationServer ../config.xml'. Program terminated with signal 6, Aborted. 0 0x00007feb90d67425 in raise () from /lib/x86_64-linux-gnu/libc.so.6 (gdb) (gdb) bt 0 0x00007feb90d67425 in raise () from /lib/x86_64-linux-gnu/libc.so.6 1 0x00007feb90d6ab10 in abort () from /lib/x86_64-linux-gnu/libc.so.6 2 0x00007feb90da539e in ?? () from /lib/x86_64-linux-gnu/libc.so.6 3 0x00007feb90e3b807 in __fortify_fail () from /lib/x86_64-linux-gnu/libc.so.6 4 0x00007feb90e3a700 in __chk_fail () from /lib/x86_64-linux-gnu/libc.so.6 5 0x00007feb90e3b7be in __fdelt_warn () from /lib/x86_64-linux-gnu/libc.so.6 6 0x000000000049e290 in CPhpResponseReader::WaitUntilReadable(int, int&, bool&) () 7 0x000000000049e505 in CPhpResponseReader::Read(CReallocableBuffer&, int) () 8 0x00000000004a5207 in CHttpPostBufferInterface::Flush() () 9 0x00000000004a5a18 in CPhpRequestJob::Execute() () 10 0x00000000004a2d28 in CThreadPool::Worker(void*) () 11 0x00007feb90b1be9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 12 0x00007feb90e24cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6 WaitUntilReadable (see the backtrace) is a function that basically uses select() to wait for some data to read from a webserver. I rewrote that function again using epoll() instead.
My program is a server that keeps connections with thousands of clients. The clients make requests to the server, and the server then pass those requests to a webserver, and sends the response back to the clients. For the requests to the webserver, my server has a thread pool. So here comes another important detail: the crash happen if I set the threadpool number of threads to a high number, which meand that a lot of select() calls are being made concurrently (I reproduced it setting it to 1024 threads). Then I launched a client simulator that I made for testing and launched 1000 client threads making requests as fast as they could. This made it crash fast when using select() and compiled in release, but no problem after introducing epoll().
Note that each thread in the pool is making a select() call with 1 fd.
Here is the function I changed to solve this problem:
bool WaitUntilReadableDeprecated(int timeoutMSecs, int& elapsedMSecs, bool& closed) < CTimeLapsus t; fd_set fileDescriptorSet; struct timeval timeStruct; closed = false; timeStruct.tv_sec = timeoutMSecs / 1000; timeStruct.tv_usec = 0; FD_ZERO(&fileDescriptorSet); FD_SET(m_socket, &fileDescriptorSet); int sel = select(m_socket + 1, &fileDescriptorSet, NULL, NULL, &timeStruct); if(sel == 0) < LogDebug("[CPhpResponseReader::WaitUntilReadable] select() returned 0, no data available"); elapsedMSecs = t.GetElapsedMilis(); return false; >else if(sel == -1) < if(errno == EBADF) < LogDebug("[CPhpResponseReader::WaitUntilReadable] select() returned -1, errno is EBADF, connection reset by host?"); closed = true; elapsedMSecs = t.GetElapsedMilis(); return true; >throw "CPhpResponseReader::WaitUntilReadable select error"; > elapsedMSecs = t.GetElapsedMilis(); return true; > bool WaitUntilReadableEpoll(int timeoutMSecs, int& elapsedMSecs, bool& closed) < CIoPoller poller(8); CTimeLapsus t; closed = false; if(poller.Add(m_socket, EPOLLIN) == -1) LogError("[CPhpResponseReader::WaitUntilReadableEpoll] poller.Add(%d, EPOLLIN) failed", m_socket); int nfds = poller.Wait(timeoutMSecs); if (nfds >0) < int theSocket = poller.GetEvents()[0].data.fd; uint32_t event = poller.GetEvents()[0].events; if(theSocket != m_socket) < LogError("[CPhpResponseReader::WaitUntilReadableEpoll] socket is different than expected", m_socket); elapsedMSecs = t.GetElapsedMilis(); return false; >if((event & EPOLLERR) || (event & EPOLLHUP)) < LogWarning("[CPhpResponseReader::WaitUntilReadableEpoll] Disconnected socket %d (event %d)", m_socket, event); elapsedMSecs = t.GetElapsedMilis(); closed = true; return false; >if(event & EPOLLIN) < // ok >> else if (nfds == -1) < if(errno == EBADF) < LogWarning("[CPhpResponseReader::WaitUntilReadableEpoll] poller.Wait() returned -1, errno is EBADF, maybe connection reset by host"); closed = true; elapsedMSecs = t.GetElapsedMilis(); return true; >LogError("[CPhpResponseReader::WaitUntilReadableEpoll] poller.Wait() failed"); elapsedMSecs = t.GetElapsedMilis(); closed = true; return false; > else < LogDebug("[CPhpResponseReader::WaitUntilReadableEpoll] poller.Wait() returned 0, no data available"); elapsedMSecs = t.GetElapsedMilis(); return false; >elapsedMSecs = t.GetElapsedMilis(); return true; > CIoPoller is just a c++ wrapper for epoll.
Distributor ID: Ubuntu Description: Ubuntu 12.04.1 LTS Release: 12.04 Buffer overflow detected in a program that runs flawlessly ( apparently)
On a string containing as, bs and cs, we can perform the following operation. We can take any two adjacent different characters and replace them by the third. For example, 'ab' gets reduced to 'c' and so does 'ba' and so on. I wrote this code to perform the following operation on t strings (t<=100) and max length of strings = 100
#include #include #include int redlen(char string[100][100], int x) < int g, checker; checker = 1; for(; checker; ) < checker = 0; for(int i = 0;string[x][i]!='\0'; i++) < if((string[x][i]=='a' && string[x][i+1]=='b') || (string[x][i]=='b' && string[x][i+1]=='a')) < string[x][i]='c'; checker = 1; for(g = i+1; string[x][g]!='\0'; g++) < string[x][g]=string[x][g+1]; >i = 0; > else if((string[x][i]=='b' && string[x][i+1]=='c') || (string[x][i]=='c' && string[x][i+1]=='b')) < string[x][i]='a'; checker = 1; for(g = i+1; string[x][g]!='\0'; g++) < string[x][g]=string[x][g+1]; >i = 0; > else if((string[x][i]=='a' && string[x][i+1]=='c') || (string[x][i]=='c' && string[x][i+1]=='a')) < string[x][i]='b'; checker = 1; for(g = i+1; string[x][g]!='\0'; g++) < string[x][g]=string[x][g+1]; >i = 0; > > > return strlen(string[x]); > void main() < int t; char r[3]; gets(r); t = atoi(r); char string[100][100]; int i; for(i = 0; iint printval; for(i = 0; i > It worked fine on the sample case of the problem and also on the cases I developed on my own. But when I submitted it online, it passed only one of the ten cases, and in the rest, this message popped up.
*** buffer overflow detected ***: /run-DkQcMiKXhWz9LjirrRnu/solution terminated ======= Backtrace: ========= /lib/i386-linux-gnu/tls/i686/nosegneg/libc.so.6(__fortify_fail+0x45)[0xb76df045] /lib/i386-linux-gnu/tls/i686/nosegneg/libc.so.6(+0x102e1a)[0xb76dde1a] /lib/i386-linux-gnu/tls/i686/nosegneg/libc.so.6(__gets_chk+0x165)[0xb76ddd85] /run-DkQcMiKXhWz9LjirrRnu/solution[0x8048436] /lib/i386-linux-gnu/tls/i686/nosegneg/libc.so.6(__libc_start_main+0xf3)[0xb75f44d3] /run-DkQcMiKXhWz9LjirrRnu/solution[0x80484e9] ======= Memory map: ======== 08048000-08049000 r-xp 00000000 ca:02 15613970 /run-DkQcMiKXhWz9LjirrRnu/solution 08049000-0804a000 r--p 00000000 ca:02 15613970 /run-DkQcMiKXhWz9LjirrRnu/solution 0804a000-0804b000 rw-p 00001000 ca:02 15613970 /run-DkQcMiKXhWz9LjirrRnu/solution 084e2000-08503000 rw-p 00000000 00:00 0 [heap] b75ba000-b75d6000 r-xp 00000000 ca:01 394527 /lib/i386-linux-gnu/libgcc_s.so.1 b75d6000-b75d7000 r--p 0001b000 ca:01 394527 /lib/i386-linux-gnu/libgcc_s.so.1 b75d7000-b75d8000 rw-p 0001c000 ca:01 394527 /lib/i386-linux-gnu/libgcc_s.so.1 b75d8000-b75db000 rw-p 00000000 00:00 0 b75db000-b777e000 r-xp 00000000 ca:01 394522 /lib/i386-linux- gnu/tls/i686/nosegneg/libc-2.15.so b777e000-b777f000 ---p 001a3000 ca:01 394522 /lib/i386-linux-gnu/tls/i686/nosegneg/libc-2.15.so b777f000-b7781000 r--p 001a3000 ca:01 394522 /lib/i386-linux-gnu/tls/i686/nosegneg/libc-2.15.so b7781000-b7782000 rw-p 001a5000 ca:01 394522 /lib/i386-linux-gnu/tls/i686/nosegneg/libc-2.15.so b7782000-b7789000 rw-p 00000000 00:00 0 b7789000-b778a000 r-xp 00000000 00:00 0 [vdso] b778a000-b77aa000 r-xp 00000000 ca:01 400153 /lib/i386-linux-gnu/ld-2.15.so b77aa000-b77ab000 r--p 0001f000 ca:01 400153 /lib/i386-linux-gnu/ld-2.15.so b77ab000-b77ac000 rw-p 00020000 ca:01 400153 /lib/i386-linux-gnu/ld-2.15.so bfe6e000-bfe8f000 rw-p 00000000 00:00 0 [stack] Aborted (core dumped)