Linux ram cache buffer

What is the difference between buffer and cache memory in Linux?

Buffers are associated with a specific block device, and cover caching of filesystem metadata as well as tracking in-flight pages. The cache only contains parked file data. That is, the buffers remember what’s in directories, what file permissions are, and keep track of what memory is being written from or read to for a particular block device. The cache only contains the contents of the files themselves.

Short answer: Cached is the size of the page cache. Buffers is the size of in-memory block I/O buffers. Cached matters; Buffers is largely irrelevant.

Long answer: Cached is the size of the Linux page cache, minus the memory in the swap cache, which is represented by SwapCached (thus the total page cache size is Cached + SwapCached). Linux performs all file I/O through the page cache. Writes are implemented as simply marking as dirty the corresponding pages in the page cache; the flusher threads then periodically write back to disk any dirty pages. Reads are implemented by returning the data from the page cache; if the data is not yet in the cache, it is first populated. On a modern Linux system, Cached can easily be several gigabytes. It will shrink only in response to memory pressure. The system will purge the page cache along with swapping data out to disk to make available more memory as needed.

Buffers are in-memory block I/O buffers. They are relatively short-lived. Prior to Linux kernel version 2.4, Linux had separate page and buffer caches. Since 2.4, the page and buffer cache are unified and Buffers is raw disk blocks not represented in the page cache—i.e., not file data. The Buffers metric is thus of minimal importance. On most systems, Buffers is often only tens of megabytes.

«Buffers is largely irrelevant» — No. There are many cases, where file content caching is irrelevant, but keeping the metadata in cache speeds things up. A video streaming NAS server for instance.

Any system doing lots of I/O will use lots of memory for buffers. I’m bulkloading a 100GB MySQL/InnoDB database and buffers go above 2GB all the time.

«Buffers» represent how much portion of RAM is dedicated to cache disk blocks. «Cached» is similar like «Buffers», only this time it caches pages from file reading.

I have tested this using a simple python program that writes large amounts of blocks. What happens is that the cache gets filled up as reported by free -w -h , not the buffers column. I think the cache column counts both disk writes and disk reads and buffers is used for something else.

@CMCDragonkai thanks for the experimental evidence. The interesting question is whether you were writing faster than the disk could flush out. (e.g. would sync take a long time) which would tell us if dirty blocks are counted differently than clean ones. Linux will certainly cache both types (lru as mentioned in the question) but one is far more serious in terms of memory pressure.

It’s not ‘quite’ as simple as this, but it might help understand:

Buffer is for storing file metadata (permissions, location, etc). Every memory page is kept track of here.

Cache is for storing actual file contents.

Explained by Red Hat:

Cache Pages:

A cache is the part of the memory which transparently stores data so that future requests for that data can be served faster. This memory is utilized by the kernel to cache disk data and improve i/o performance.

The Linux kernel is built in such a way that it will use as much RAM as it can to cache information from your local and remote filesystems and disks. As the time passes over various reads and writes are performed on the system, kernel tries to keep data stored in the memory for the various processes which are running on the system or the data that of relevant processes which would be used in the near future. The cache is not reclaimed at the time when process get stop/exit, however when the other processes requires more memory then the free available memory, kernel will run heuristics to reclaim the memory by storing the cache data and allocating that memory to new process.

When any kind of file/data is requested then the kernel will look for a copy of the part of the file the user is acting on, and, if no such copy exists, it will allocate one new page of cache memory and fill it with the appropriate contents read out from the disk.

The data that is stored within a cache might be values that have been computed earlier or duplicates of original values that are stored elsewhere in the disk. When some data is requested, the cache is first checked to see whether it contains that data. The data can be retrieved more quickly from the cache than from its source origin.

SysV shared memory segments are also accounted as a cache, though they do not represent any data on the disks. One can check the size of the shared memory segments using ipcs -m command and checking the bytes column.

Buffers are the disk block representation of the data that is stored under the page caches. Buffers contains the metadata of the files/data which resides under the page cache. Example: When there is a request of any data which is present in the page cache, first the kernel checks the data in the buffers which contain the metadata which points to the actual files/data contained in the page caches. Once from the metadata the actual block address of the file is known, it is picked up by the kernel for processing.

Источник

How to Clear RAM Memory Cache, Buffer and Swap Space on Linux

Like any other operating system, GNU/Linux has implemented memory management efficiently and even more than that. But if any process is eating away your memory and you want to clear it, Linux provides a way to flush or clear ram cache.

How to Clear Cache in Linux?

Every Linux System has three options to clear cache without interrupting any processes or services.

1. Clear PageCache only.

# sync; echo 1 > /proc/sys/vm/drop_caches

2. Clear dentries and inodes.

# sync; echo 2 > /proc/sys/vm/drop_caches

3. Clear pagecache, dentries, and inodes.

# sync; echo 3 > /proc/sys/vm/drop_caches

Explanation of the above command.

sync will flush the file system buffer. Command Separated by “;” run sequentially. The shell waits for each command to terminate before executing the next command in the sequence. As mentioned in the kernel documentation, writing to drop_cache will clean cache without killing any application/service, command echo is doing the job of writing to file.

If you have to clear the disk cache, the first command is safest in enterprise and production as “. echo 1 > ….” will clear the PageCache only. It is not recommended to use the third option above “. echo 3 >” in production until you know what you are doing, as it will clear pagecache, dentries, and inodes.

Is it a good idea to free Buffer and Cache in Linux that might be used by Linux Kernel?

Free Buffer and Cache in Linux

When you are applying various settings and want to check, if it is actually implemented specially on the I/O-extensive benchmark, then you may need to clear the buffer cache. You can drop cache as explained above without rebooting the System i.e., no downtime required.

Linux is designed in such a way that it looks into the disk cache before looking onto the disk. If it finds the resource in the cache, then the request doesn’t reach the disk. If we clean the cache, the disk cache will be less useful as the OS will look for the resource on the disk.

Moreover, it will also slow the system for a few seconds while the cache is cleaned and every resource required by OS is loaded again in the disk cache.

Now we will be creating a shell script to auto clear RAM cache daily at 2 am via a cron scheduler task. Create a shell script clearcache.sh and add the following lines.

#!/bin/bash # Note, we are using "echo 3", but it is not recommended in production instead use "echo 1" echo "echo 3 > /proc/sys/vm/drop_caches"

Set execute permission on the clearcache.sh file.

Now you may call the script whenever you are required to clear the ram cache.

Now set a cron to clear RAM cache every day at 2 am. Open crontab for editing.

Append the below line, save and exit to run it at 2 am daily.

0 2 * * * /path/to/clearcache.sh

For more details on how to cron a job, you may like to check our article on 11 Cron Scheduling Jobs.

Is it a good idea to auto clear the RAM cache on the production server?

Clear RAM Cache on Linux Production Server?

No! it is not. Think of a situation when you have scheduled the script to clear ram cache every day at 2 am. Every day at 2 am the script is executed and it flushes your RAM cache. One day for whatsoever reason may be more than expected users are online on your website and seeking resources from your server.

At the same time, the scheduled script runs and clears everything in the cache. Now all the users are fetching data from the disk. It will result in a server crash and corrupt the database. So clear ram-cache only when required, and known your footsteps, else you are a Cargo Cult System Administrator.

How to Clear Swap Space in Linux?

If you want to clear Swap space, you may like to run the below command.

Also, you may add the above command to a cron script above, after understanding all the associated risks.

Now we will be combining both above commands into one single command to make a proper script to clear RAM Cache and Swap Space.

# echo 3 > /proc/sys/vm/drop_caches && swapoff -a && swapon -a && printf '\n%s\n' 'Ram-cache and Swap Cleared' OR $ su -c "echo 3 >'/proc/sys/vm/drop_caches' && swapoff -a && swapon -a && printf '\n%s\n' 'Ram-cache and Swap Cleared'" root

After testing both the above commands, we will run the command “free -h” before and after running the script and will check the cache.

That’s all for now, if you liked the article, don’t forget to provide us with your valuable feedback in the comments to let us know, what you think is a good idea to clear ram cache and buffer in production and Enterprise?

Источник

What do the «buff/cache» and «avail mem» fields in top mean?

Within the output of top, there are two fields, marked «buff/cache» and «avail Mem» in the memory and swap usage lines: What do these two fields mean? I’ve tried Googling them, but the results only bring up generic articles on top, and they don’t explain what these fields signify.

3 Answers 3

top ’s manpage doesn’t describe the fields, but free ’s does:

buffers

Memory used by kernel buffers ( Buffers in /proc/meminfo )

cache

Memory used by the page cache and slabs ( Cached and SReclaimable in /proc/meminfo )

buff/cache

Sum of buffers and cache

available

Estimation of how much memory is available for starting new applications, without swapping. Unlike the data provided by the cache or free fields, this field takes into account page cache and also that not all reclaimable memory slabs will be reclaimed due to items being in use ( MemAvailable in /proc/meminfo , available on kernels 3.14, emulated on kernels 2.6.27+, otherwise the same as free)

Basically, “buff/cache” counts memory used for data that’s on disk or should end up there soon, and as a result is potentially usable (the corresponding memory can be made available immediately, if it hasn’t been modified since it was read, or given enough time, if it has); “available” measures the amount of memory which can be allocated and used without causing more swapping (see How can I get the amount of available memory portably across distributions? for a lot more detail on that).

Источник