In Linux, what is the difference between «buffers» and «cache» reported by the free command?
It’s more like metadata which you find in buffers, it is not related to io buffers. Some of the kernel buffers are accounted for in the slab allocator but do not count to buffers or cache memory at all.
5 Answers 5
The «cached» total will also include some other memory allocations, such as any tmpfs filesytems. To see this in effect try:
mkdir t mount -t tmpfs none t dd if=/dev/zero of=t/zero.file bs=10240 count=10240 sync; echo 3 > /proc/sys/vm/drop_caches; free -m umount t sync; echo 3 > /proc/sys/vm/drop_caches; free -m and you will see the «cache» value drop by the 100Mb that you copied to the ram-based filesystem (assuming there was enough free RAM, you might find some of it ended up in swap if the machine is already over-committed in terms of memory use). The «sync; echo 3 > /proc/sys/vm/drop_caches» before each call to free should write anything pending in all write buffers (the sync) and clear all cached/buffered disk blocks from memory so free will only be reading other allocations in the «cached» value.
The RAM used by virtual machines (such as those running under VMWare) may also be counted in free’s «cached» value, as will RAM used by currently open memory-mapped files (this will vary depending on the hypervisor/version you are using and possibly between kernel versions too).
So it isn’t as simple as «buffers counts pending file/network writes and cached counts recently read/written blocks held in RAM to save future physical reads», though for most purposes this simpler description will do.
+1 for interesting nuances. This is the kind of information I’m looking for. In fact, I suspect that the figures are so convoluted, so involved in so many different activities, that they are at best general indicators.
I don’t think the RAM used by virtual machines is counted as «cached», at least for qemu-kvm. I notice that on my KVM host, the cache value is not only too small to be correct (at 1.9 Gig), but it doesn’t change if I destroy/start one of my VMs. It also doesn’t change if I perform the tmpfs mount trick on one of the VMs. I created an 800Meg tmpfs partition there and «cached» showed the proper values on the VM but it did not change on the VM host. But the «used» value did shrink/grow when I destroyed/started my VM.
@MikeS: How different virtualisation solutions handle memory is liable to vary, in fact how the kernel measures various uses of memory may change between major versions.
@MikeS: With regard to «perform the tmpfs mount trick on one of the VMs» — I that will not affect the host readings if they are not showing other mem used by the VM. I do see the effect in a KVM VM itself: before dd free = 2020, after dd free = 1899, after drop fs free = 2001 (the 19Mb difference will be due to other processes on the VM, it was not idle when I ran the test). The host may not see the change: the memory is probably still allocated to the VM even though it is free for use by processes in the VM.
Tricky Question. When you calculate free space you actually need to add up buffer and cache both. This is what I Could find
A buffer is something that has yet to be «written» to disk. A cache is something that has been «read» from the disk and stored for later use.
Actually, if you’re interested in reclaimable space (that is, RAM that can be used for programs if needed), you have to calculate Cached — Shmem from /proc/meminfo . This is because kernel includes Shmem into Cached for historical reasons but Shmem cannot be dropped even if system is running out of memory. Biggest users of Shmem are usually X server (including GPU), Java JVM and PostgreSQL ( shared_buffers ).
I was looking for more clear description about buffer and i found in «Professional Linux® Kernel Architecture 2008»
Chapter 16: Page and Buffer Cache
Interaction
Setting up a link between pages and buffers serves little purpose if there are no benefits for other parts of the kernel. As already noted, some transfer operations to and from block devices may need to be performed in units whose size depends on the block size of the underlying devices, whereas many parts of the kernel prefer to carry out I/O operations with page granularity as this makes things much easier — especially in terms of memory management. In this scenario, buffers act as intermediaries between the two worlds.
Explained by RedHat:
A cache is the part of the memory which transparently stores data so that future requests for that data can be served faster. This memory is utilized by the kernel to cache disk data and improve i/o performance.
The Linux kernel is built in such a way that it will use as much RAM as it can to cache information from your local and remote filesystems and disks. As the time passes over various reads and writes are performed on the system, kernel tries to keep data stored in the memory for the various processes which are running on the system or the data that of relevant processes which would be used in the near future. The cache is not reclaimed at the time when process get stop/exit, however when the other processes requires more memory then the free available memory, kernel will run heuristics to reclaim the memory by storing the cache data and allocating that memory to new process.
When any kind of file/data is requested then the kernel will look for a copy of the part of the file the user is acting on, and, if no such copy exists, it will allocate one new page of cache memory and fill it with the appropriate contents read out from the disk.
The data that is stored within a cache might be values that have been computed earlier or duplicates of original values that are stored elsewhere in the disk. When some data is requested, the cache is first checked to see whether it contains that data. The data can be retrieved more quickly from the cache than from its source origin.
SysV shared memory segments are also accounted as a cache, though they do not represent any data on the disks. One can check the size of the shared memory segments using ipcs -m command and checking the bytes column.
Buffers are the disk block representation of the data that is stored under the page caches. Buffers contains the metadata of the files/data which resides under the page cache. Example: When there is a request of any data which is present in the page cache, first the kernel checks the data in the buffers which contain the metadata which points to the actual files/data contained in the page caches. Once from the metadata the actual block address of the file is known, it is picked up by the kernel for processing.
Understanding «Buffers» and «Cached» from free command
This has been asked earlier but don’t want to update the same thread again as it was a old thread . Want to clarify myself on the «buffers» and «cache» column from the output of free command. This is what my understanding. Buffer is something where data is there in memory but yet to be flushed to disk . The data will be flushed to disk by bdflush daemon periodically or we can do it manually by running sync command . Cache on the other hand is program/data which is loaded into memory but is retained in memory so that if is needed again , it will be quickly available. To understand the concept of buffers , I tried the following experiment. This is the reading of free command in my desktop
[zama@localhost ~]$ free -m total used free shared buffers cached Mem: 2897 465 2431 0 30 230 -/+ buffers/cache: 204 2692 Swap: 4000 0 4000 [zama@localhost ~]$ sync [zama@localhost ~]$ free -m total used free shared buffers cached Mem: 2897 466 2431 0 30 230 -/+ buffers/cache: 205 2691 Swap: 4000 0 4000
Here I cannot see buffer getting reduced after executing the sync command. Next I tried the following. Tried to write a huge file to the disk .
[zama@localhost ~]$ dd if=/dev/zero of=test bs=1024k
@localhost ~]# free -m total used free shared buffers cached Mem: 2897 1466 1430 0 32 1127 -/+ buffers/cache: 306 2590 Swap: 4000 0 4000
I again executed the sync command and then checked using free . I can see that the buffer value getting decreased from the output of free command . There was no reduction in the cache . This means that the dirty pages in RAM after my execution of dd coomand has been flushed to disk .
@localhost ~]# free -m total used free shared buffers cached Mem: 2897 1466 1430 0 10 1127 -/+ buffers/cache: 306 2590 Swap: 4000 0 4000
[root@localhost ~]# cat /proc/sys/vm/drop_caches 0 [root@localhost ~]# echo "1" > /proc/sys/vm/drop_caches [root@localhost ~]# cat /proc/sys/vm/drop_caches 1
[root@localhost ~]# free -m total used free shared buffers cached Mem: 2897 299 2597 0 1 74 -/+ buffers/cache: 224 2672 Swap: 4000 0 4000
So , my initial statement that «Buffer» is RAM data which is yet to be flushed to disk looks to be correct . Please guide me whether I am in the right direction.
What do the «buff/cache» and «avail mem» fields in top mean?
Within the output of top, there are two fields, marked «buff/cache» and «avail Mem» in the memory and swap usage lines: What do these two fields mean? I’ve tried Googling them, but the results only bring up generic articles on top, and they don’t explain what these fields signify.
3 Answers 3
top ’s manpage doesn’t describe the fields, but free ’s does:
buffers
Memory used by kernel buffers ( Buffers in /proc/meminfo )
cache
Memory used by the page cache and slabs ( Cached and SReclaimable in /proc/meminfo )
buff/cache
Sum of buffers and cache
available
Estimation of how much memory is available for starting new applications, without swapping. Unlike the data provided by the cache or free fields, this field takes into account page cache and also that not all reclaimable memory slabs will be reclaimed due to items being in use ( MemAvailable in /proc/meminfo , available on kernels 3.14, emulated on kernels 2.6.27+, otherwise the same as free)
Basically, “buff/cache” counts memory used for data that’s on disk or should end up there soon, and as a result is potentially usable (the corresponding memory can be made available immediately, if it hasn’t been modified since it was read, or given enough time, if it has); “available” measures the amount of memory which can be allocated and used without causing more swapping (see How can I get the amount of available memory portably across distributions? for a lot more detail on that).