Creating threads fails with “Resource temporarily unavailable” with 4.3 kernel
It looks like before the last reboot I was running kernel 4.2.5-1, now I’m running 4.3.3-2. Downgrading to 4.2.5-1 fixes all the problems. Other posts mentioning the problem are this and this. I have opened a bug report for Arch Linux.
What has changed in the kernel that could be causing this?
Here are some example error messages:
Crash dump was written to: erl_crash.dump Failed to create aux thread Jan 07 14:37:25 edeltraud docker[30625]: runtime/cgo: pthread_create failed: Resource temporarily unavailable dpkg: unrecoverable fatal error, aborting: fork failed: Resource temporarily unavailable E: Sub-process /usr/bin/dpkg returned an error code (2) test -z "/usr/include" || /usr/sbin/mkdir -p "/tmp/lib32-popt/pkg/lib32-popt/usr/include" /bin/sh: fork: retry: Resource temporarily unavailable /usr/bin/install -c -m 644 popt.h '/tmp/lib32-popt/pkg/lib32-popt/usr/include' test -z "/usr/share/man/man3" || /usr/sbin/mkdir -p "/tmp/lib32-popt/pkg/lib32-popt/usr/share/man/man3" /bin/sh: fork: retry: Resource temporarily unavailable /bin/sh: fork: retry: No child processes /bin/sh: fork: retry: Resource temporarily unavailable /bin/sh: fork: retry: No child processes /bin/sh: fork: retry: No child processes /bin/sh: fork: retry: Resource temporarily unavailable /bin/sh: fork: retry: Resource temporarily unavailable /bin/sh: fork: retry: No child processes /bin/sh: fork: Resource temporarily unavailable /bin/sh: fork: Resource temporarily unavailable make[3]: *** [install-man3] Error 254 Jan 07 11:04:39 edeltraud docker[780]: time="2016-01-07T11:04:39.986684617+01:00" level=error msg="Error running container: [8] System error: fork/exec /proc/self/exe: resource temporarily unavailable" [Wed Jan 06 23:20:33.701287 2016] [mpm_event:alert] [pid 217:tid 140325422335744] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread bash fork: retry: Resource temporarily unavailable
I am trying to run a shell script which will create process using a shell script. I get Resource temporarily unavailable error. how to identify which limit (memory/process/filecount) is creating this problem. Below is my ulimit -a results.
core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 563959 max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) unlimited open files (-n) 65535 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) unlimited cpu time (seconds, -t) unlimited max user processes (-u) 10000000 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited You will have to give us more information. Perhaps post your bash script. i’ve never had trouble with this. Example: Before: ps aux | wc -l gave 183 and after running a script to create 4000 processes it gave 4183.
It’s a simple script which will invoke background process for each invocation. #!/bin/sh while read zi; do (source example.sh &) done
1 Answer 1
For the case in the comments, where you were not using much memory per thread, you were hitting the cgroup limits. You will find the default to be around 12288, but the value is writable:
$ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max 12288 $ echo 15000 | sudo tee /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max 15000 $ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max 15000 And if I use my «what is the thread limit» program (found here) to check, before:
$ ./thread-limit Creating threads . 100 threads so far . 200 threads so far . . 12100 threads so far . 12200 threads so far . Failed with return code 11 creating thread 12281 (Resource temporarily unavailable). Malloc worked, hmmm $ ./thread-limit Creating threads . 100 threads so far . 200 threads so far . 300 threads so far . . 14700 threads so far . 14800 threads so far . 14900 threads so far . Failed with return code 11 creating thread 14993 (Resource temporarily unavailable). Malloc worked, hmmm Of course, the numbers above are not exact because the «doug» user has a few other threads running, such as my SSH sessions to my sever. Check with:
$ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.current 8 /* compile with: gcc -pthread -o thread-limit thread-limit.c */ /* originally from: http://www.volano.com/linuxnotes.html */ #include #include #include #include #include #define MAX_THREADS 100000 #define PTHREAD_STACK_MIN 1*1024*1024*1024 int i; void run(void) < sleep(60 * 60); >int main(int argc, char *argv[]) < int rc = 0; pthread_t thread[MAX_THREADS]; pthread_attr_t thread_attr; pthread_attr_init(&thread_attr); pthread_attr_setstacksize(&thread_attr, PTHREAD_STACK_MIN); printf("Creating threads . \n"); for (i = 0; i < MAX_THREADS && rc == 0; i++) < rc = pthread_create(&(thread[i]), &thread_attr, (void *) &run, NULL); if (rc == 0) < pthread_detach(thread[i]); if ((i + 1) % 100 == 0) printf("%i threads so far . \n", i + 1); >else < printf("Failed with return code %i creating thread %i (%s).\n", rc, i + 1, strerror(rc)); // can we allocate memory? char *block = NULL; block = malloc(65545); if(block == NULL) printf("Malloc failed too :( \n"); else printf("Malloc worked, hmmm\n"); >> sleep(60*60); // ctrl+c to exit; makes it easier to see mem use exit(0); > EDIT May, 2020: For newer versions of Ubuntu, the default maximum PID number is now 4194304, and therefore adjusting it is not needed.
Now, if you have enough memory, the next limit will be defined by the default maximum PID number, which is 32768, but is also writable. Obvioulsy in order to have more than 32768 simultaneous processes or tasks or threads their PID will have to be allowed to be higher:
$ cat /proc/sys/kernel/pid_max 32768 $ echo 80000 | sudo tee /proc/sys/kernel/pid_max 80000 $ cat /proc/sys/kernel/pid_max 80000 Note that is quite on purpose that a number bigger than 2**16 was chosen, to see if it was actually allowed. And so now, set the cgroup max to, say 70000:
$ echo 70000 | sudo tee /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max 70000 $ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max 70000 And at this point, realize that the above listed program seems to have a limit of about 32768 threads, even if resources are still available, and so use another method. My test server with 16 gigabytes of memory seems to exhaust some other resource at about 62344 tasks, even though there does seem to still be memory available.
$ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.current 62344 top - 13:48:26 up 21:08, 4 users, load average: 281.52, 134.90, 70.93 Tasks: 62535 total, 201 running, 62334 sleeping, 0 stopped, 0 zombie %Cpu0 : 96.6 us, 2.4 sy, 0.0 ni, 1.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu1 : 95.7 us, 2.4 sy, 0.0 ni, 1.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu2 : 95.1 us, 3.1 sy, 0.0 ni, 1.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu3 : 93.5 us, 4.6 sy, 0.0 ni, 1.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu4 : 94.8 us, 3.4 sy, 0.0 ni, 1.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu5 : 95.5 us, 2.6 sy, 0.0 ni, 1.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu6 : 94.7 us, 3.5 sy, 0.0 ni, 1.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu7 : 93.8 us, 4.5 sy, 0.0 ni, 1.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 15999116 total, 758684 free, 10344908 used, 4895524 buff/cache KiB Swap: 16472060 total, 16470396 free, 1664 used. 4031160 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 37884 doug 20 0 108052 68920 3104 R 5.7 0.4 1:23.08 top 24075 doug 20 0 4360 652 576 S 0.4 0.0 0:00.31 consume 26006 doug 20 0 4360 796 720 S 0.4 0.0 0:00.09 consume 30062 doug 20 0 4360 732 656 S 0.4 0.0 0:00.17 consume 21009 doug 20 0 4360 748 672 S 0.3 0.0 0:00.26 consume Seems I finally hit my default ulimit settings for both user processes and number of timers (signals):
$ ulimit -i 62340 doug@s15:~$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 62340 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 32768 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 62340 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited If I raise those limits, in my case, I did it via /etc/security/limits.conf :
# /etc/security/limits.conf # # S18 specific edits. 2019.12.24 # also for a rediculous number of threads test. # #Each line describes a limit for a user in the form: # # # #Where: # can be: # - a user name # - a group name, with @group syntax # - the wildcard *, for default entry # - the wildcard %, can be also used with %group syntax, # for maxlogin limit # - NOTE: group and wildcard limits are not applied to root. # To apply a limit to the root user, must be # the literal username root. # # can have the two values: # - "soft" for enforcing the soft limits # - "hard" for enforcing hard limits # # can be one of the following: # - core - limits the core file size (KB) # - data - max data size (KB) # - fsize - maximum filesize (KB) # - memlock - max locked-in-memory address space (KB) # - nofile - max number of open file descriptors * - nofile 32768 # - rss - max resident set size (KB) # - stack - max stack size (KB) # - cpu - max CPU time (MIN) # - nproc - max number of processes * - nproc 200000 # - as - address space limit (KB) # - maxlogins - max number of logins for this user # - maxsyslogins - max number of logins on the system # - priority - the priority to run user process with # - locks - max number of file locks the user can hold # - sigpending - max number of pending signals * - sigpending 200000 # - msgqueue - max memory used by POSIX message queues (bytes) # - nice - max nice priority allowed to raise to values: [-20, 19] # - rtprio - max realtime priority # - chroot - change root to directory (Debian-specific) # # # #* soft core 0 #root hard core 100000 #* hard rss 10000 #@student hard nproc 20 #@faculty soft nproc 20 #@faculty hard nproc 50 #ftp hard nproc 0 #ftp - chroot /ftp #@student - maxlogins 4 # End of file I am able to go to 126020 threads, before the return of the inability to fork. This time the limit was (keep in mind that there are about `150 root owned threads on this server, before the test starts):
cat /proc/sys/kernel/threads-max 126189 O.K. so now adjusting that parameter:
echo 99999999 | sudo tee /proc/sys/kernel/threads-max 99999999 I can get to about 132,000 threads before my 16 gigabyte server starts to swap memory, and trouble errupts.
$ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.current 132016 Note: running top places a significant additional load on the system under these conditions, so I didn’t run it. However memory:
doug@s18:~/config/etc/security$ free -m total used free shared buff/cache available Mem: 15859 15509 270 1 79 137 Swap: 2047 4 2043 At some point you will get into trouble, but it is absolutely amazing how gracefully the system bogs down. Once my system starts to swap, it totally boggs down and I had many of these errors:
Feb 17 16:13:02 s15 kernel: [ 967.907305] INFO: task waiter:119371 blocked for more than 120 seconds. Feb 17 16:13:02 s15 kernel: [ 967.907335] Not tainted 4.10.0-rc8-stock #194 Feb 17 16:13:02 s15 kernel: [ 967.907357] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. And my load average ballooned to ~29000. But I just left the computer for an hour and it sorted itself out. I staggered the spin out of the threads by 200 microseconds per spin out, and that also seemed to help.
bash fork error (Resource temporarily unavailable) does not stop, and keeps showing up every time I try to kill/reboot
I mistakenly used a limited server as an iperf server for 5000 parallel connections. (limit is 1024 processes) Now every time I log in, I see this:
-bash: fork: retry: Resource temporarily unavailable -bash: fork: retry: Resource temporarily unavailable -bash: fork: retry: Resource temporarily unavailable -bash: fork: retry: Resource temporarily unavailable -bash: fork: Resource temporarily unavailable -bash-4.1$ ps -bash: fork: retry: Resource temporarily unavailable -bash: fork: retry: Resource temporarily unavailable -bash: fork: retry: Resource temporarily unavailable -bash: fork: retry: Resource temporarily unavailable -bash: fork: Resource temporarily unavailable Same happens when I do a killall or similar things. I have even tried to reboot the system but again this is what I get after reboot:
-bash-4.1$ sudo reboot -bash: fork: retry: Resource temporarily unavailable -bash: fork: retry: Resource temporarily unavailable -bash: fork: retry: Resource temporarily unavailable -bash: fork: retry: Resource temporarily unavailable -bash: fork: Resource temporarily unavailable -bash-4.1$ So Basically I cannot do anything. all the commands get this error :/ I can, however, do «exit». This is an off-site server that I do not have physical access to, so I cannot turn it off/on physically. Any ideas how I can fix this problem? I highly appreciate any help.