What is RSS and VSZ in Linux memory management
What are RSS and VSZ in Linux memory management? In a multithreaded environment how can both of these can be managed and tracked?
7 Answers 7
RSS is the Resident Set Size and is used to show how much memory is allocated to that process and is in RAM. It does not include memory that is swapped out. It does include memory from shared libraries as long as the pages from those libraries are actually in memory. It does include all stack and heap memory.
VSZ is the Virtual Memory Size. It includes all memory that the process can access, including memory that is swapped out, memory that is allocated, but not used, and memory that is from shared libraries.
So if process A has a 500K binary and is linked to 2500K of shared libraries, has 200K of stack/heap allocations of which 100K is actually in memory (rest is swapped or unused), and it has only actually loaded 1000K of the shared libraries and 400K of its own binary then:
RSS: 400K + 1000K + 100K = 1500K VSZ: 500K + 2500K + 200K = 3200K Since part of the memory is shared, many processes may use it, so if you add up all of the RSS values you can easily end up with more space than your system has.
The memory that is allocated also may not be in RSS until it is actually used by the program. So if your program allocated a bunch of memory up front, then uses it over time, you could see RSS going up and VSZ staying the same.
There is also PSS (proportional set size). This is a newer measure which tracks the shared memory as a proportion used by the current process. So if there were two processes using the same shared library from before:
PSS: 400K + (1000K/2) + 100K = 400K + 500K + 100K = 1000K Threads all share the same address space, so the RSS, VSZ and PSS for each thread is identical to all of the other threads in the process. Use ps or top to view this information in linux/unix.
There is way more to it than this, to learn more check the following references:
I believe RSS does include memory from dynamically linked libraries. If there are 3 processes using libxml2.so , the shared library will be counted in each of their RSS, so the sum of their RSS will be more than the actual memory used.
I’m on ubuntu 16.04, and there is a java process has 1.2G RES and 4.5G VIRT showing from top command. This system doesn’t have any swap, swapon —show returns nothing. How do you explain this? If vsz is swap + shared libraries, in this case, shared libraries are over than 3.3G? Is it possible? Just really confused.
I’m not really sure. Take a look at this answer on Java virtual memory usage: stackoverflow.com/a/561450/622115. Short version: VSZ can include heap space that is allocated and not used as well as memory mapped files.
Great. Just add something. if you malloc(100KB), then only use 1KB actually. The rss is 1K and vsz is 100K, even if there is no swap here.
RSS is Resident Set Size (physically resident memory — this is currently occupying space in the machine’s physical memory), and VSZ is Virtual Memory Size (address space allocated — this has addresses allocated in the process’s memory map, but there isn’t necessarily any actual memory behind it all right now).
Note that in these days of commonplace virtual machines, physical memory from the machine’s view point may not really be actual physical memory.
Minimal runnable example
For this to make sense, you have to understand the basics of paging: How does x86 paging work? and in particular that the OS can allocate virtual memory via page tables / its internal memory book keeping (VSZ virtual memory) before it actually has a backing storage on RAM or disk (RSS resident memory).
Now to observe this in action, let’s create a program that:
- allocates more RAM than our physical memory with mmap
- writes one byte on each page to ensure that each of those pages goes from virtual only memory (VSZ) to actually used memory (RSS)
- checks the memory usage of the process with one of the methods mentioned at: Memory usage of current process in C
#define _GNU_SOURCE #include #include #include #include #include #include #include typedef struct < unsigned long size,resident,share,text,lib,data,dt; >ProcStatm; /* https://stackoverflow.com/questions/1558402/memory-usage-of-current-process-in-c/7212248#7212248 */ void ProcStat_init(ProcStatm *result) < const char* statm_path = "/proc/self/statm"; FILE *f = fopen(statm_path, "r"); if(!f) < perror(statm_path); abort(); >if(7 != fscanf( f, "%lu %lu %lu %lu %lu %lu %lu", &(result->size), &(result->resident), &(result->share), &(result->text), &(result->lib), &(result->data), &(result->dt) )) < perror(statm_path); abort(); >fclose(f); > int main(int argc, char **argv) < ProcStatm proc_statm; char *base, *p; char system_cmd[1024]; long page_size; size_t i, nbytes, print_interval, bytes_since_last_print; int snprintf_return; /* Decide how many ints to allocate. */ if (argc < 2) < nbytes = 0x10000; >else < nbytes = strtoull(argv[1], NULL, 0); >if (argc < 3) < print_interval = 0x1000; >else < print_interval = strtoull(argv[2], NULL, 0); >page_size = sysconf(_SC_PAGESIZE); /* Allocate the memory. */ base = mmap( NULL, nbytes, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0 ); if (base == MAP_FAILED) < perror("mmap"); exit(EXIT_FAILURE); >/* Write to all the allocated pages. */ i = 0; p = base; bytes_since_last_print = 0; /* Produce the ps command that lists only our VSZ and RSS. */ snprintf_return = snprintf( system_cmd, sizeof(system_cmd), "ps -o pid,vsz,rss | awk ''", (uintmax_t)getpid() ); assert(snprintf_return >= 0); assert((size_t)snprintf_return < sizeof(system_cmd)); bytes_since_last_print = print_interval; do < /* Modify a byte in the page. */ *p = i; p += page_size; bytes_since_last_print += page_size; /* Print process memory usage every print_interval bytes. * We count memory using a few techniques from: * https://stackoverflow.com/questions/1558402/memory-usage-of-current-process-in-c */ if (bytes_since_last_print >print_interval) < bytes_since_last_print -= print_interval; printf("extra_memory_committed %lu KiB\n", (i * page_size) / 1024); ProcStat_init(&proc_statm); /* Check /proc/self/statm */ printf( "/proc/self/statm size resident %lu %lu KiB\n", (proc_statm.size * page_size) / 1024, (proc_statm.resident * page_size) / 1024 ); /* Check ps. */ puts(system_cmd); system(system_cmd); puts(""); >i++; > while (p < base + nbytes); /* Cleanup. */ munmap(base, nbytes); return EXIT_SUCCESS; >gcc -ggdb3 -O0 -std=c99 -Wall -Wextra -pedantic -o main.out main.c echo 1 | sudo tee /proc/sys/vm/overcommit_memory sudo dmesg -c ./main.out 0x1000000000 0x200000000 echo $? sudo dmesg - 0x1000000000 == 64GiB: 2x my computer’s physical RAM of 32GiB
- 0x200000000 == 8GiB: print the memory every 8GiB, so we should get 4 prints before the crash at around 32GiB
- echo 1 | sudo tee /proc/sys/vm/overcommit_memory : required for Linux to allow us to make a mmap call larger than physical RAM: Maximum memory which malloc can allocate
extra_memory_committed 0 KiB /proc/self/statm size resident 67111332 768 KiB ps -o pid,vsz,rss | awk '' PID VSZ RSS 29827 67111332 1648 extra_memory_committed 8388608 KiB /proc/self/statm size resident 67111332 8390244 KiB ps -o pid,vsz,rss | awk '' PID VSZ RSS 29827 67111332 8390256 extra_memory_committed 16777216 KiB /proc/self/statm size resident 67111332 16778852 KiB ps -o pid,vsz,rss | awk '' PID VSZ RSS 29827 67111332 16778864 extra_memory_committed 25165824 KiB /proc/self/statm size resident 67111332 25167460 KiB ps -o pid,vsz,rss | awk '' PID VSZ RSS 29827 67111332 25167472 Killed which by the 128 + signal number rule means we got signal number 9 , which man 7 signal says is SIGKILL, which is sent by the Linux out-of-memory killer.
- VSZ virtual memory remains constant at printf ‘0x%X\n’ 0x40009A4 KiB ~= 64GiB ( ps values are in KiB) after the mmap.
- RSS «real memory usage» increases lazily only as we touch the pages. For example:
- on the first print, we have extra_memory_committed 0 , which means we haven’t yet touched any pages. RSS is a small 1648 KiB which has been allocated for normal program startup like text area, globals, etc.
- on the second print, we have written to 8388608 KiB == 8GiB worth of pages. As a result, RSS increased by exactly 8GIB to 8390256 KiB == 8388608 KiB + 1648 KiB
- RSS continues to increase in 8GiB increments. The last print shows about 24 GiB of memory, and before 32 GiB could be printed, the OOM killer killed the process
OOM killer logs
Our dmesg commands have shown the OOM killer logs.
An exact interpretation of those has been asked at:
The very first line of the log was:
[ 7283.479087] mongod invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0So we see that interestingly it was the MongoDB daemon that always runs in my laptop on the background that first triggered the OOM killer, presumably when the poor thing was trying to allocate some memory.
However, the OOM killer does not necessarily kill the one who awoke it.
After the invocation, the kernel prints a table or processes including the oom_score :
[ 7283.479292] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [ 7283.479303] [ 496] 0 496 16126 6 172032 484 0 systemd-journal [ 7283.479306] [ 505] 0 505 1309 0 45056 52 0 blkmapd [ 7283.479309] [ 513] 0 513 19757 0 57344 55 0 lvmetad [ 7283.479312] [ 516] 0 516 4681 1 61440 444 -1000 systemd-udevdand further ahead we see that our own little main.out actually got killed on the previous invocation:
[ 7283.479871] Out of memory: Kill process 15665 (main.out) score 865 or sacrifice child [ 7283.479879] Killed process 15665 (main.out) total-vm:67111332kB, anon-rss:92kB, file-rss:4kB, shmem-rss:30080832kB [ 7283.479951] oom_reaper: reaped process 15665 (main.out), now anon-rss:0kB, file-rss:0kB, shmem-rss:30080832kBThis log mentions the score 865 which that process had, presumably the highest (worst) OOM killer score as mentioned at: https://unix.stackexchange.com/questions/153585/how-does-the-oom-killer-decide-which-process-to-kill-first
Also interestingly, everything apparently happened so fast that before the freed memory was accounted, the oom was awoken again by the DeadlineMonitor process:
[ 7283.481043] DeadlineMonitor invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0and this time that killed some Chromium process, which is usually my computers normal memory hog:
[ 7283.481773] Out of memory: Kill process 11786 (chromium-browse) score 306 or sacrifice child [ 7283.481833] Killed process 11786 (chromium-browse) total-vm:1813576kB, anon-rss:208804kB, file-rss:0kB, shmem-rss:8380kB [ 7283.497847] oom_reaper: reaped process 11786 (chromium-browse), now anon-rss:0kB, file-rss:0kB, shmem-rss:8044kBTested in Ubuntu 19.04, Linux kernel 5.0.0.
Linux kernel docs
https://github.com/torvalds/linux/blob/v5.17/Documentation/filesystems/proc.rst has some points. The term «VSZ» is not used there but «RSS» is, and there’s nothing too enlightening (surprise?!)
Instead of VSZ, the kernel seems to use the term VmSize , which appears e.g. on /proc/$PID/status .
The first of these lines shows the same information as is displayed for the mapping in /proc/PID/maps. Following lines show the size of the mapping (size); the size of each page allocated when backing a VMA (KernelPageSize), which is usually the same as the size in the page table entries; the page size used by the MMU when backing a VMA (in most cases, the same as KernelPageSize); the amount of the mapping that is currently resident in RAM (RSS); the process’ proportional share of this mapping (PSS); and the number of clean and dirty shared and private pages in the mapping.
The «proportional set size» (PSS) of a process is the count of pages it has in memory, where each page is divided by the number of processes sharing it. So if a process has 1000 pages all to itself, and 1000 shared with one other process, its PSS will be 1500.
Note that even a page which is part of a MAP_SHARED mapping, but has only a single pte mapped, i.e. is currently used by only one process, is accounted as private and not as shared.
So we can guess a few more things:
- shared libraries used by a single process appear in RSS, if more than one process has them then not
- PSS was mentioned by jmh, and has a more proportional approach between «I’m the only process that holds the shared library» and «there are N process holding the shared library, so each one holds memory/N on average»
What does maximum resident set size mean?
The above code uses ru_maxrss attribute of rusage structure. It gives the value of maximum resident set size. What does it mean? Each time the program is executed it gives a different value. So please explain the output of this code?
These are screenshots of two executions of the same code which give different outputs, how can these numbers be explained or what can be interpreted from these two outputs?For starters, do you know what resident set size means? «Maximum» means it’s the largest that the RSS ever was during the process’s life.
1 Answer 1
Resident set size (RSS) means, roughly, the total amount of physical memory assigned to a process at a given point in time. It does not count pages that have been swapped out, or that are mapped from a file but not currently loaded into physical memory.
«Maximum RSS» means the maximum of the RSS since the process’s birth, i.e. the largest it has ever been. So this number tells you the largest amount of physical memory your process has ever been using at any one instant.
It can vary from one run to the next if, for instance, the OS decided to swap out different amounts of your program’s memory at different times. This decision would depend in part on what the rest of the system is doing, and where else physical memory is needed.
These are screenshots of two executions of the same code which give different outputs, how can these numbers be explained or what can be interpreted from these two outputs?