Increase stack size
I’m doing computations with huge arrays and for some of this computations I need an increased stack size! Is there any downside of setting the stack size to unlimited ( ulimit -s unlimited ) in my ~/.bashrc? The program is written in fortran(F77 & F90) and parallelized with MPI. Some of my arrays have more than 2E7 entries and when I use a small number of cores with MPI it crashes with segmentation fault . The array size stays the same through the whole computation therefore I setted them to fixes value:
real :: p(200,200,400) integer :: ib,ie,jb,je,kb,ke . ib=1;ie=199 jb=2;je=198 kb=2;ke=398 call SOLVE_POI_EQ(rank,p(ib:ie,jb:je,kb:ke),R) What kind of software are you running? In which programming language? Are you coding it? Could you show some source code (so edit your question to improve it)?
2 Answers 2
Setting the stacksize to unlimited likely won’t help you. You are allocating a chunk of 64MB on the stack, and likely don’t fill it from the top, but from the bottom.
This is important, because the OS grows the stack as you go. Whenever it detects a page-fault right below the stack segment, it will assume that you need more space, and silently insert a new page. The size of this trigger-region within your address-space is limited, though, and I doubt that its larger than 64 MB. Since you index variables are likely placed below your array on the stack, accessing them already does the 64 MB jump that kills your process.
Just make your array allocatable , add the corresponding allocate() statement, and you should be fine.
I always thought that allocatable arrays are used when the size of an array changes during runtime? Mine are fixed from the beginning and the stay the same size during the whole computation!
You can use allocatable to create arrays with dynamic size, but it is also useful to avoid large stack allocations. An allocatable array is always allocated on the heap which does not have the limitations of the stack. And the allocate() statement won’t complain if you pass it a compile time constant 😉
Stack size is never really unlimited, so you would still have some failures. And your code still won’t be portable to Linux systems with smaller (or normal-sized) stacks.
BTW, you should explain which kind of programs are you running, show some source code.
If coding in C++, using standard containers should help a lot (regarding actual stack consumption). For example, a local (stack allocated) std::vector v(10000); (instead of int v[10000]; ) has its data allocated on the heap (and deallocated by the destructor when you exit from the block defining it)
It would be much better to improve your programs to avoid excessive stack consumption. The need of a lot of stack space is really a bug that you should try to correct. A typical rule of thumb is to have call frames smaller than a few kilobytes (so allocate any larger data on the heap).
You might consider also using the Boehm conservative garbage collector: you would use GC_MALLOC instead of malloc (and you would heap allocate large data structure using GC_MALLOC ) but you won’t have to bother to free your (GC-heap allcoated) data.
Linux set stack size
How to change default stack size in linux?
Автор shashankagrawal, история, 3 года назад ,
We can change stack size in linux temporarily by using the command ulimit -s new_value , But is there any way to change stack size permanently, or Can we do something within C++ code, so it will change stack size by itself?
I tried changing this, but It didn’t work. I faced this issue in hackercup qualification round D2, terminal gave segfault, and later I came to know about the low value of default stack size in linux.
#linux, #stacksize
Комментарии (19)
After altering limits.conf , have you tried logging off and logging back into your account? I believe the changes won’t come into effect unless you do this.
I think you should not put @ before your username. I added this my_username soft stack 262144 and it worked well.
I feel u bro, same thing happened to me, but even worse I was on Windows using VS code, spent half an hour trying to find what’s the problem 🙁 then after a long search edited my compilation flags as shown here
Given link work only for codeblocks. Is there any way/command-flag which i can use in VScode for this purpose?
I don’t use codeblocks, just edited my compilation flags, whatever IDE u might be using, search in your setting how to edit your compilation flags and it should work fine
Just typing ulimit -s into terminal should give you the default stack size. For me, it’s 8192, i.e. 8 MiB.
To change the stack size in the code, you can use getrlimit() and setrlimit() system calls:
Note that this example will work only for UNIX-like systems, so it’s better to disable this code (using #ifdef , for example) when submitting to an online judge.
Is it a good idea to have stack size something as big as 1 gb when default size is like 8MB? Does it have any side-effects?
If you have infinite recursion, then your program crashes much later, and possibly after consuming all system memory. Suggestion: Leave it as default and raise only when needed in individual cases.
Does the same occur on increasing it not to rlim_max but to 1 Gb instead ? P.S — just curious because i added it in my snippet and made limit to 1 Gb and it should return segmentation fault after crossing the 1 Gb limit but I’m not in mood to try after reading your Comment.
If you increase it to 1 GiB, then the program will segfault if it tries to exceed this limit. If you always have 1 GiB of free memory while solving contests, everything should be fine. Also I noticed that I made a tiny mistake in the comment above, which is fixed now. setrlimit() takes stack size in bytes, so 1 GiB would be rlim.rlim_cur = 1024 * 1024 * 1024 . If you added this code into your snippet, it’s better to fix it.
bash -c 'ulimit -s 262144 && ulimit -v 2097152 && time "./%e"' ulimit -v is used to limit the amount of memory, so I won’t go out of memory in case of infinite memory allocation.
I failed D2 because of the exact same issue 🙁
Maybe you can just add ulimit -s new_value to your .bashrc file?
| № | Пользователь | Рейтинг |
|---|---|---|
| 1 | B enq | 3783 |
| 2 | t ourist | 3767 |
| 3 | c nnfls_csy | 3540 |
| 4 | — 0.5 | 3504 |
| 5 | j iangly | 3485 |
| 6 | i naFSTream | 3477 |
| 7 | f antasy | 3468 |
| 8 | e cnerwala | 3431 |
| 9 | Q AQAutoMaton | 3428 |
| 10 | m aroonrk | 3401 |
| № | Пользователь | Вклад |
|---|---|---|
| 1 | adamant | 185 |
| 2 | U m_nik | 180 |
| 3 | awoo | 173 |
| 4 | m aroonrk | 165 |
| 5 | nor | 162 |
| 6 | -is-this-fft- | 158 |
| 7 | k o_osaga | 150 |
| 7 | dario2994 | 150 |
| 9 | antontrygubO_o | 149 |
| 10 | SecondThread | 146 |
atcoder_official → freee Programming Contest 2023（AtCoder Beginner Contest 310） Announcement
kuviman → Codeforces: обновление городов и стран
visheshgautam.official → HOW TO GET RID OF STATUS STACK OVERFLOW ERROR
abhatter → Help. how to solve USACO 2017 Silver: Bovine Genomics
Ahmadsm2005 → I’m the Bluegyptian. AMA
Lonely_coder_007 → The implementation of the Rational class using operator overloading:
Krzychuo → I’ve solved all 800-1300 rated problems solvable with C++. AMA
vermish77 → Help Please
rivalq → I’m a kpop star. AMA
natalina → Codeforces Round #883 (Div. 3)
miaowtin → [cp-tricki] How to come up with a comparator?
Kalrddddd → Битовые операции
Sokol080808 → Codeforces Round #881 (Div. 3) Разбор
National_Wind → About counting the number of subarray
TheScrasse → I’m TheScrasse. AMA
TalentnotDefined → Special Subarray
don0thing → n0thing
-is-this-fft- → I’m -is-this-fft-. AMA!
flamestorm → Codeforces Round 871 (Div. 4) Editorial
tibinyte → I’m tibinyte, AMA
nor → I’m a logic gate. AMA
d uality → Codeforces Round #884 (Div. 1 + Div. 2) Editorial
BinaryGambit05 → This is how a competitive programmer’s dictionary would look like?!
pajenegod → Tutorial: A simple O(n log n) polynomial multiplication algorithm
KhaledFarhat → [GYM] The 2023 Damascus University Collegiate Programming Contest
Setting the default stack size on Linux globally for the program
So I’ve noticed that the default stack size for threads on linux is 8MB (if I’m wrong, PLEASE correct me), and, incidentally, 1MB on Windows. This is quite bad for my application, as on a 4-core processor that means 64 MB is space is used JUST for threads! The worst part is, I’m never using more than 100kb of stack per thread (I abuse the heap a LOT ;)). My solution right now is to limit the stack size of threads. However, I have no idea how to do this portably. Just for context, I’m using Boost.Thread for my threading needs. I’m okay with a little bit of #ifdef hell, but I’d like to know how to do it easily first. Basically, I want something like this (where windows_* is linked on windows builds, and posix_* is linked under linux builds)
// windows_stack_limiter.c int limit_stack_size() < // Windows impl. return 0; >// posix_stack_limiter.c int limit_stack_size() < // Linux impl. return 0; >// stack_limiter.cpp int limit_stack_size(); static volatile int placeholder = limit_stack_size(); How do I flesh out those functions? Or, alternatively, am I just doing this entirely wrong? Remember I have no control over the actual thread creation (no new params to CreateThread on Windows), as I’m using Boost.Thread.
It’s not as bad as you think. only address space is reserved, not actual RAM. So e.g. if you spawn 8 threads, and each thread actually uses 100kb of stack space, then you’ve only used up 800kb of RAM, not 8MB.
How to increase stack limit permanently without restarting
The ulimit method only works as long as I am logged in.
limits.conf will work after a restart.
Is there a possible way to increase the limit without restarting?
There must be a way to do this. The ulimit shell command doesn’t work for a user. And the current logged in user doesn’t have permission to alter their limit unless they are root.
4 Answers 4
What’s wrong with editing your .bashrc file to do a ulimit -s size every time you start a terminal session?
unfortunately that is not for me as a user, it is for the couchdb process, which crashes because a document in the DB is to big to fit in the stack.
@Simon: that means you’re currently restarting that process every time? How is that done? Can’t you splice the ulimit -s size in the scripts that control the restarts?
well, it is a subprocess of couchdb that crashes (couchjs). So all processes started by the user couchdb should have a bigger stack
If you want to do this programatically, you can use the setrlimit() function.
unfortunately I need to set the limit for couchdb, not for a program of my own. So setting it programatically is out of question
Linux (and POSIX compatible systems in general) inherit current process limits from existing processes when a new process is started (using fork() or exec() ). As a result, you would just need to change the limits for all existing processes that are already running on your system. The config files just define the values for the initial processes after the system has been booted and everything else inherits the current values of parent processes.
Why would you want to increase stack limit of everything? Do all your existing processes immediately require increased stack space for some reason?
If you actually need to change the limits of any given process after that process has already been started, you can do it with command prlimit which uses kernel interface int prlimit(pid_t pid, int resource, const struct rlimit *new_limit, struct rlimit *old_limit) behind the scenes. It requires CAP_SYS_RESOURCE for many use cases so you often need to run that as root in practice.