How to detect a memory leak?
I seem to have a larger memory leak on my current ubuntu System After reporting strange Eclipse memory errors ( https://askubuntu.com/questions/148998/eclipse-constant-different-out-of-memory-errors ) I started to get ‘Not enough Memory’ error messages in my console today — while doing simple tasks like typing in sudo -s — or even — free -m Typing in ‘free -m’ repeadetly showed me how my RAM quickly goes up from 700M to 900M, growing up to the size of 2000M in a few seconds (after freeing up memory with echo 3 > /proc/sys/vm/drop_caches ) Eclipse isnt the cause, I completly killed the process and the ram still was going up. Is there any way to detect where the leak is coming from? I cant even update my system anymore, since apt-get update fails (probably because it’s out of memory) Using Ubuntu 11.10
I am VERY happy I am not crazy. I’ve had the same problem since upgrading to 13.10, but i remember having it with 11.10. The question is: Are you using CrashPlan? I seem to have it narrowed down to that, i just don’t know how to fix it. I’ve tried the memory tweaks, but it is not working. I hope it gives you some clues
There’s no point in forcing the kernel to drop caches. They will be flushed and their space reclaimed anyway as soon as more physical memory is required otherwise. Force-flushing them most likely even detrimental to overall performance, since uncached objects need to be retrieved from much slower secondary storage. Free main memory is by no means a good thing. It’s either a sign of bad cache management or very light usage.
6 Answers 6
First, make sure to have a temp folder available which has enough free space. The following commands create dumps which can be several GBs in size.
You can create a new tmp folder using the following command. You may want to change /tmp to a different filesystem with enough space
Steps to find Memory Leak
- Find out the PID of the process which causing memory leak (you can also use e.g. htop if available) and store it in a variable called pid
cat /proc/$pid/smaps > $TMPDIR/beforeMemInc.txt cat /proc/$pid/smaps > $TMPDIR/afterMemInc.txt diff -u $TMPDIR/beforeMemInc.txt $TMPDIR/afterMemInc.txt beforeMemInc.txt afterMemInc.txt --------------------------------------------------- 2b3289290000-2b3289343000 2b3289290000-2b3289343000 #ADDRESS Shared_Clean: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Clean: 0 kB Private_Dirty: 28 kB Private_Dirty: 36 kB Referenced: 28 kB Referenced: 36 kB Anonymous: 28 kB Anonymous: 36 kB #INCREASE MEM AnonHugePages: 0 kB AnonHugePages: 0 kB Swap: 0 kB Swap: 0 kB KernelPageSize: 4 kB KernelPageSize: 4 kB MMUPageSize: 4 kB MMUPageSize: 4 kB Locked: 0 kB Locked: 0 kB VmFlags: rd wr mr mw me ac VmFlags: rd wr mr mw me ac gcore -o $TMPDIR/process $PID cd $TMPDIR gdb -p $pid dump memory memory.dump 0x2b3289290000 0x2b3289343000 I am in a Docker container, getting a Permission denied error when running cat /proc/2882/smaps > /tmp/before.txt on step 2. What did I do wrong?
This method worked for me up until step 9. The leak in my case didn’t contain any readable or recognisable data. How can I get the symbol names of the data that was leaked?
The drop_cache trick will not free memory, it will reset the cache. Uses ps command if you want to identify which processes uses the more memory.
For instance to monitor the list of the top 15 of resident memory users.
$ watch "ps --sort -rss -eo pid,pmem,rss,vsz,comm | head -16" PID %MEM RSS VSZ COMMAND 2590 13.4 136892 825000 firefox 1743 10.7 109020 300780 Xorg 2067 8.5 86764 1118140 unity-2d-shell 3307 4.1 42560 627780 unity-2d-spread 2068 2.9 29904 617644 unity-2d-panel 2092 2.5 25524 1291204 nautilus 2457 1.9 20292 530276 gnome-terminal 2351 1.9 20016 821488 unity-scope-vid 2161 1.9 19476 531968 unity-panel-ser 2034 1.7 18256 759716 gnome-settings- 2074 1.5 16176 518016 nm-applet 2273 1.5 15452 580416 unity-lens-vide 2051 1.4 15112 524260 metacity 2395 1.2 12836 407336 update-notifi You could check also the shared memory reservation but you will only know who is the owner of the segments.
$ ls -l /run/shm total 272 -r-------- 1 ed ed 67108904 Nov 29 18:17 pulse-shm-1884617860 -r-------- 1 lightdm lightdm 67108904 Nov 29 18:11 pulse-shm-2352897759 -r-------- 1 ed ed 67108904 Nov 29 18:12 pulse-shm-3444873503 -r-------- 1 ed ed 67108904 Nov 29 18:12 pulse-shm-3485341848 -r-------- 1 lightdm lightdm 67108904 Nov 29 18:11 pulse-shm-535843976 -r-------- 1 ed ed 67108904 Nov 29 19:12 pulse-shm-789046959 -r-------- 1 ed ed 67108904 Nov 29 18:38 pulse-shm-863909656 $ df /run/shm Filesystem 1K-blocks Used Available Use% Mounted on none 509332 272 509060 1% /run/shm note that reserved allocations are much higher than real the allocated pages (df ‘used’)
$ ipcs -m ------ Shared Memory Segments -------- key shmid owner perms bytes nattch status 0x00000000 294912 ed 700 122880 2 dest 0x00000000 327681 ed 700 4823040 2 dest 0x00000000 491522 ed 600 393216 2 dest 0x00000000 589827 ed 700 4578120 2 dest 0x00000000 425988 ed 700 27852 2 dest 0x00000000 458757 ed 600 393216 2 dest Edit: Need to pass —sort -rss to ps to get the processes with the most memory usage, otherwise the process list is sorted increasing numerical and gives the processes with least memory usage.