Linux: find out what process is using all the RAM?
Before actually asking, just to be clear: yes, I know about disk cache, and no, it is not my case 🙂 Sorry, for this preamble 🙂 I’m using CentOS 5. Every application in the system is swapping heavily, and the system is very slow. When I do free -m , here is what I got:
total used free shared buffers cached Mem: 3952 3929 22 0 1 18 -/+ buffers/cache: 3909 42 Swap: 16383 46 16337 So, I actually have only 42 Mb to use! As far as I understand, -/+ buffers/cache actually doesn’t count the disk cache, so I indeed only have 42 Mb, right? I thought, I might be wrong, so I tried to switch off the disk caching and it had no effect — the picture remained the same. So, I decided to find out who is using all my RAM, and I used top for that. But, apparently, it reports that no process is using my RAM. The only process in my top is MySQL, but it is using 0.1% of RAM and 400Mb of swap. Same picture when I try to run other services or applications — all go in swap, top shows that MEM is not used (0.1% maximum for any process).
top - 15:09:00 up 2:09, 2 users, load average: 0.02, 0.16, 0.11 Tasks: 112 total, 1 running, 111 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 4046868k total, 4001368k used, 45500k free, 748k buffers Swap: 16777208k total, 68840k used, 16708368k free, 16632k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ SWAP COMMAND 3214 ntp 15 0 23412 5044 3916 S 0.0 0.1 0:00.00 17m ntpd 2319 root 5 -10 12648 4460 3184 S 0.0 0.1 0:00.00 8188 iscsid 2168 root RT 0 22120 3692 2848 S 0.0 0.1 0:00.00 17m multipathd 5113 mysql 18 0 474m 2356 856 S 0.0 0.1 0:00.11 472m mysqld 4106 root 34 19 251m 1944 1360 S 0.0 0.0 0:00.11 249m yum-updatesd 4109 root 15 0 90152 1904 1772 S 0.0 0.0 0:00.18 86m sshd 5175 root 15 0 90156 1896 1772 S 0.0 0.0 0:00.02 86m sshd Restart doesn’t help, and, by they way is very slow, which I wouldn’t normally expect on this machine (4 cores, 4Gb RAM, RAID1). So, with that — I’m pretty sure that this is not a disk cache, who is using the RAM, because normally it should have been reduced and let other processes to use RAM, rather then go to swap. So, finally, the question is — if someone has any ideas how to find out what process is actually using the memory so heavily?
11 GB of GPU RAM used, and no process listed by nvidia-smi
In my GPU #0, 11341MiB of GPU RAM is used, and no process is listed by nvidia-smi . How is that possible, and how can I get my memory back?
Thu Aug 18 14:27:58 2016 +------------------------------------------------------+ | NVIDIA-SMI 352.63 Driver Version: 352.63 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 GeForce GTX TIT. Off | 0000:02:00.0 Off | N/A | | 29% 61C P2 71W / 250W | 11341MiB / 12287MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 1 GeForce GTX TIT. Off | 0000:03:00.0 Off | N/A | | 22% 42C P0 71W / 250W | 23MiB / 12287MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 2 GeForce GTX TIT. Off | 0000:82:00.0 Off | N/A | | 22% 35C P0 69W / 250W | 23MiB / 12287MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 3 GeForce GTX TIT. Off | 0000:83:00.0 Off | N/A | | 0% 33C P0 60W / 250W | 23MiB / 12287MiB | 0% Default | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+ - pkill -9 python and sudo nvidia-smi —gpu-reset -i 0 did not help.
- sudo fuser -v /dev/nvidia* didn’t show any unwanted process.
I use a GeForce GTX Titan Maxwell with Ubuntu 14.04.4 LTS x64.
Linux: 100 Mi RAM free but no processes with high-memory usage
A Debian Bullseye server is using almost 100% RAM quantity (7.65G / 7.79 G): But there are no processes with more than 0.0% memory usage
DEV root@postgres2 ~# free -h total used free shared buff/cache available Mem: 7.8Gi 7.7Gi 100Mi 0.0Ki 42Mi 39Mi Swap: 14Gi 767Mi 14Gi DEV root@postgres2 ~# ps -eo pid,ppid,cmd,comm,%mem,%cpu --sort=-%mem | head -10 PID PPID CMD COMMAND %MEM %CPU 7072 1 /usr/bin/apt-get -o Debug:: apt-get 0.0 11.9 8424 1 /usr/bin/apt-get -o Debug:: apt-get 0.0 7.8 7252 1 /usr/bin/apt-get -o Debug:: apt-get 0.0 11.0 8265 1 /usr/bin/apt-get -o Debug:: apt-get 0.0 8.3 8085 1 /usr/bin/apt-get -o Debug:: apt-get 0.0 8.6 9806 22518 postgres: 12/main: repmgr r postgres 0.0 13.0 9703 1 /usr/bin/apt-get -o Debug:: apt-get 0.0 5.2 206 1 /lib/systemd/systemd-journa systemd-journal 0.0 0.0 9814 5382 ps -eo pid,ppid,cmd,comm,%m ps 0.0 0.0 DEV root@postgres2 ~# ps ax|wc -l 224 DEV root@postgres2 ~# cat /proc/meminfo MemTotal: 8170216 kB MemFree: 131976 kB MemAvailable: 42600 kB Buffers: 7000 kB Cached: 55200 kB SwapCached: 52900 kB Active: 82772 kB Inactive: 73988 kB Active(anon): 48652 kB Inactive(anon): 49300 kB Active(file): 34120 kB Inactive(file): 24688 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 15622140 kB SwapFree: 15211964 kB Dirty: 68 kB Writeback: 0 kB AnonPages: 92920 kB Mapped: 33280 kB Shmem: 3340 kB Slab: 7808800 kB SReclaimable: 36480 kB SUnreclaim: 7772320 kB KernelStack: 3616 kB PageTables: 14860 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 19707248 kB Committed_AS: 1162964 kB VmallocTotal: 34359738367 kB VmallocUsed: 0 kB VmallocChunk: 0 kB Percpu: 1288 kB HardwareCorrupted: 0 kB AnonHugePages: 6144 kB ShmemHugePages: 0 kB ShmemPmdMapped: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Hugetlb: 0 kB DirectMap4k: 5609320 kB DirectMap2M: 2779136 kB DEV root@postgres2 ~# ps aux --sort -rss | head -n15 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 381 0.0 0.8 1694928 68916 ? Ssl 2022 44:28 /usr/share/auditbeat/bin/auditbeat --environment systemd -c /etc/auditbeat/auditbeat.yml --path.home /usr/share/auditbeat --path.config /etc/auditbeat --path.data /var/lib/auditbeat --path.logs /var/log/auditbeat root 380 0.0 0.1 1373392 12796 ? Ssl 2022 9:33 /usr/share/metricbeat/bin/metricbeat --environment systemd -c /etc/metricbeat/metricbeat.yml --path.home /usr/share/metricbeat --path.config /etc/metricbeat --path.data /var/lib/metricbeat --path.logs /var/log/metricbeat nagios 1257 0.0 0.0 1056796 6756 ? Sl 2022 22:49 /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2 --no-stack-rlimit daemon --close-stdio -e /var/log/icinga2/error.log root 23205 0.0 0.0 159992 5004 pts/0 S 16:00 0:00 fish root 23273 0.0 0.0 10960 3348 pts/0 R+ 16:00 0:00 ps aux --sort -rss root 211 0.0 0.0 81564 2600 ? Ss 2022 16:31 /lib/systemd/systemd-journald root 1 0.0 0.0 170780 2596 ? Ss 2022 3:11 /sbin/init nagios 1262 0.0 0.0 259476 2268 ? S 2022 6:05 /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2 --no-stack-rlimit daemon --close-stdio -e /var/log/icinga2/error.log me 23170 0.0 0.0 159752 1828 pts/0 Ss 16:00 0:00 -fish me 23153 0.0 0.0 21164 1744 ? Ss 16:00 0:00 /lib/systemd/systemd --user postgres 23984 0.0 0.0 219008 1572 ? Ss 2022 15:22 /usr/lib/postgresql/12/bin/postgres -D /var/lib/postgresql/12/main -c config_file=/etc/postgresql/12/main/postgresql.conf root 23141 0.0 0.0 16616 1528 ? Ss 16:00 0:00 sshd: me**strong text** [priv] Debian-+ 382 0.0 0.0 42988 1428 ? Ss 2022 11:50 /usr/sbin/snmpd -Lsd -Lf /dev/null -u Debian-snmp -g Debian-snmp -I -smux mteTrigger mteTriggerConf -f -p /run/snmpd.pid postgres 21836 0.0 0.0 220064 1312 ? Ss 15:51 0:00 postgres: 12/main: mydb mydb 192.168.100.33(52436) idle High memory usage but no process is using it
So out of my 10GB of memory, 8.5GB is in use and only 1500MB is free (excluding cache). But I struggle to find what is using the memory. The output of ps aux | awk ‘ END ‘ , which is supposed to add up the RSS utilisation is:
In other words, my processes only use 1GB of memory but the system as a whole (excluding cache) uses 8.5GB. What could be using the other 7.5GB? ps: I have another server with a similar configuration that shows used mem of 1200 (free mem = 8.8GB) and the sum of RSS usage in ps is 900 which is closer to what I would expect.
EDIT
MemTotal: 10257656 kB MemFree: 395840 kB MemAvailable: 1428508 kB Buffers: 162640 kB Cached: 1173040 kB SwapCached: 176 kB Active: 1810200 kB Inactive: 476668 kB Active(anon): 942816 kB Inactive(anon): 176184 kB Active(file): 867384 kB Inactive(file): 300484 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 1951740 kB SwapFree: 1951528 kB Dirty: 16 kB Writeback: 0 kB AnonPages: 951016 kB Mapped: 224388 kB Shmem: 167820 kB Slab: 86464 kB SReclaimable: 67488 kB SUnreclaim: 18976 kB KernelStack: 6736 kB PageTables: 13728 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 7080568 kB Committed_AS: 1893156 kB VmallocTotal: 34359738367 kB VmallocUsed: 62284 kB VmallocChunk: 34359672552 kB HardwareCorrupted: 0 kB AnonHugePages: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 67520 kB DirectMap2M: 10418176 kB MemTotal: 12326128 kB MemFree: 8895188 kB MemAvailable: 10947592 kB Buffers: 191548 kB Cached: 2188088 kB SwapCached: 0 kB Active: 2890128 kB Inactive: 350360 kB Active(anon): 1018116 kB Inactive(anon): 33320 kB Active(file): 1872012 kB Inactive(file): 317040 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 3442684 kB SwapFree: 3442684 kB Dirty: 44 kB Writeback: 0 kB AnonPages: 860880 kB Mapped: 204680 kB Shmem: 190588 kB Slab: 86812 kB SReclaimable: 64556 kB SUnreclaim: 22256 kB KernelStack: 10576 kB PageTables: 11924 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 9605748 kB Committed_AS: 1753476 kB VmallocTotal: 34359738367 kB VmallocUsed: 62708 kB VmallocChunk: 34359671804 kB HardwareCorrupted: 0 kB AnonHugePages: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 63424 kB DirectMap2M: 12519424 kB