Linux: find out what process is using all the RAM?
Before actually asking, just to be clear: yes, I know about disk cache, and no, it is not my case 🙂 Sorry, for this preamble 🙂 I’m using CentOS 5. Every application in the system is swapping heavily, and the system is very slow. When I do free -m , here is what I got:
total used free shared buffers cached Mem: 3952 3929 22 0 1 18 -/+ buffers/cache: 3909 42 Swap: 16383 46 16337 So, I actually have only 42 Mb to use! As far as I understand, -/+ buffers/cache actually doesn’t count the disk cache, so I indeed only have 42 Mb, right? I thought, I might be wrong, so I tried to switch off the disk caching and it had no effect — the picture remained the same. So, I decided to find out who is using all my RAM, and I used top for that. But, apparently, it reports that no process is using my RAM. The only process in my top is MySQL, but it is using 0.1% of RAM and 400Mb of swap. Same picture when I try to run other services or applications — all go in swap, top shows that MEM is not used (0.1% maximum for any process).
top - 15:09:00 up 2:09, 2 users, load average: 0.02, 0.16, 0.11 Tasks: 112 total, 1 running, 111 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 4046868k total, 4001368k used, 45500k free, 748k buffers Swap: 16777208k total, 68840k used, 16708368k free, 16632k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ SWAP COMMAND 3214 ntp 15 0 23412 5044 3916 S 0.0 0.1 0:00.00 17m ntpd 2319 root 5 -10 12648 4460 3184 S 0.0 0.1 0:00.00 8188 iscsid 2168 root RT 0 22120 3692 2848 S 0.0 0.1 0:00.00 17m multipathd 5113 mysql 18 0 474m 2356 856 S 0.0 0.1 0:00.11 472m mysqld 4106 root 34 19 251m 1944 1360 S 0.0 0.0 0:00.11 249m yum-updatesd 4109 root 15 0 90152 1904 1772 S 0.0 0.0 0:00.18 86m sshd 5175 root 15 0 90156 1896 1772 S 0.0 0.0 0:00.02 86m sshd Restart doesn’t help, and, by they way is very slow, which I wouldn’t normally expect on this machine (4 cores, 4Gb RAM, RAID1). So, with that — I’m pretty sure that this is not a disk cache, who is using the RAM, because normally it should have been reduced and let other processes to use RAM, rather then go to swap. So, finally, the question is — if someone has any ideas how to find out what process is actually using the memory so heavily?
Retrieve CPU usage and memory usage of a single process on Linux?
I want to get the CPU and memory usage of a single process on Linux — I know the PID. Hopefully, I can get it every second and write it to a CSV using the ‘watch’ command. What command can I use to get this info from the Linux command-line?
22 Answers 22
(You can leave off «cmd» but that might be helpful in debugging).
Note that this gives average CPU usage of the process over the time it has been running.
The assumption would be that if you care about a single processes’ memory usage enough to monitor it like this, it’s using a significant amount of memory so that the extra couple-of-megabytes due to shared mappings isn’t an issue.
Keep in mind that %cpu «is the CPU time used divided by the time the process has been running (cputime/realtime ratio), expressed as a percentage» (see manpage of ps ). This is not the real just in time CPU usage. It can also be very different from what top shows, for instance.
as said from Xebeche just above, ps -e -o pcpu,args will show the cpu average over the lifetime of the process, which is obviously not what you want if it is a long running process
This auto-refreshes the CPU usage so it’s good for monitoring.
@MatthiasBraun should be top -p $(pgrep -d’,’ process_name) please see stackoverflow.com/a/8710740/2402577
ps command (should not use):
Use top to get CPU usage in real time(current short interval):
top -b -n 2 -d 0.2 -p 6962 | tail -1 | awk »
- -b : Batch-mode
- -n 2 : Number-of-iterations, use 2 because: When you first run it, it has no previous sample to compare to, so these initial values are the percentages since boot.
- -d 0.2 : Delay-time(in second, here is 200ms)
- -p 6962 : Monitor-PIDs
- tail -1 : the last row
- awk » : the 9-th column(the cpu usage number)
This is the most accurate answer to get the current CPU usage, not an average over the lifetime of the process.
You can get the results by the name of the process using
the -C option allows you to use process name without knowing it’s pid.
e.g. to monitor these two process IDs (12345 and 11223) every 5 seconds use
$ pidstat -h -r -u -v -p 12345,11223 5 pidstat also gives a nice average. just a shame i have not found a more elegant way of pidstat -u 1 10 | grep ^Average | sort -r -n -b -k 8,8
Launch a program and monitor it
This form is useful if you want to benchmark an executable easily:
topp() ( if [ -n "$O" ]; then $* & else $* &>/dev/null & fi pid="$!" trap "kill $pid" SIGINT o='%cpu,%mem,vsz,rss' printf '%s\n' "$o" i=0 while s="$(ps --no-headers -o "$o" -p "$pid")"; do printf "$i $s\n" i=$(($i + 1)) sleep "$" done ) %cpu,%mem,vsz 0 0.0 0.0 177584 1 0.0 0.1 588024 2 0.0 0.1 607084 3 0.0 0.2 637248 4 0.0 0.2 641692 5 68.0 0.2 637904 6 80.0 0.2 642832 where vsz is the total memory usage in KiB, e.g. the above had about 600MiB usage.
If your program finishes, the loop stops and we exit topp .
Alternatively, if you git Ctrl + C, the program also stops due to the trap : How do I kill background processes / jobs when my shell script exits?
- T=0.5 topp ./myprog : change poll interval
- O=1 topp ./myprog : don’t hide program stdout/stderr. This can be useful to help correlate at which point memory usage bursts with stdout.
ps vs top on instantaneous CPU% usage
Note that the CPU usage given by ps above is not «instantaneous» (i.e. over the last N seconds), but rather the average over the processes’ entire lifetime as mentioned at: https://unix.stackexchange.com/questions/58539/top-and-ps-not-showing-the-same-cpu-result ps memory measures should be fine however.
That thread as well as: How can I determine the current CPU utilization from the shell? suggest that the Linux kernel does not store any more intermediate usage statistics, so the only way to do that would be to poll and calculate for the previous period, which is what top does.
We could therefore use top -n1 instead of ps if we wanted that:
toppp() ( $* &>/dev/null & pid="$!" trap exit SIGINT i=1 top -b n1 -d "$" -n1 -p "$pid" while true; do top -b n1 -d "$" -n1 -p "$pid" | tail -1; printf "$i "; i=$(($i + 1)); done ) as mentioned e.g. at: https://stackoverflow.com/a/62421136/895245 which produces output of type:
top - 17:36:59 up 9:25, 12 users, load average: 0.32, 1.75, 2.21 Tasks: 1 total, 1 running, 0 sleeping, 0 stopped, 0 zombie %Cpu(s): 13.4 us, 2.5 sy, 0.0 ni, 84.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st MiB Mem : 31893.7 total, 13904.3 free, 15139.8 used, 2849.7 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 16005.5 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 706287 ciro 20 0 590436 40352 20568 R 106.7 0.1 0:00.16 node 706287 ciro 20 0 607060 57172 21340 R 126.7 0.2 0:00.35 node 1 706287 ciro 20 0 642008 80276 21812 R 113.3 0.2 0:00.52 node 2 706287 ciro 20 0 641676 93108 21812 R 113.3 0.3 0:00.70 node 3 706287 ciro 20 0 647892 99956 21812 R 106.7 0.3 0:00.87 node 4 706287 ciro 20 0 655980 109564 21812 R 140.0 0.3 0:01.09 node My only problems with this is that top is not as nice for interactive usage:
- Ctrl + C does not exit the above command, not sure why trap exit is not working as it does with ps . I have to kill the command Ctrl + \ , and then that does not kill the process itself which continues to run on the background, which means that if it is an infinite loop like a server, I have to ps aux and then kill it.
- the not exit automatically when the benchmarked program exits
Maybe someone more shell savvy than me can find a solution for those.
ps memory measurements should be the same as top however if you’re just after memory.