Linux disable disk cache

Is it possible in linux to disable filesystem caching for specific files? [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.

This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.

I have some large files and i am ok with them being read at disk I/O capacity. I wish to have file system cache free for other files. Is it possible to turn of file system caching for specific files in linux ?

SO is for programming questions, not questions about using or configuring Linux and its applications. SuperUser or Unix & Linux would be better places for questions like this.

2 Answers 2

Your question hints that you might not be the author of the program you wish to control. If that’s the case the answer is «not easily». If you are looking for something where you just mark (e.g. via extended attributes) a particular set of files «nocache» the answer is no. At best you are limited to having a LD_PRELOAD wrapper around the program and the wrapper would have to be written carefully to avoid impacting all files the program would try to open etc.

If you ARE the author of the program you should take a look at using fadvise (or the equivalent madvise if you’re using mmap ) because after you have finished reading the data you can hint to the kernel that it should discard the pieces it cached by using the FADV_DONTNEED parameter (why not use FADV_NOREUSE ? Because with Linux kernels available at the time of writing it’s a no-op).

Another technique if you’re the author would be to open the file with the O_DIRECT flag set but I do not recommend this unless you really know what you’re doing. O_DIRECT comes with a large set of usage constraints and conditions on its use (which people often don’t notice or understand the impact of until it’s too late):

• You MUST do I/O in multiples of the disk’s block size (no smaller than 512 bytes but not unusual for it to be 4Kbytes and it can be some other larger multiple) and you must only use offsets that are similarly well aligned.
• The buffers of your program will have to conform to an alignment rule.
• Filesystems can choose not to support O_DIRECT so your program has to handle that.
• Filesystems may simply choose to put your I/O through the page cache anyway ( O_DIRECT is a «best effort» hint).

NB: Not allowing caching to be used at all (i.e. not even on the initial read) may lead to the file being read in at speeds far below what the disk can achieve.

Источник

How to Disable Disk Write Caching in Ubuntu To Prevent Data Loss

This simple tutorial is going to show you how to disable disk write caching in Ubuntu to prevent data loss when you may experience power failure.

Enable write caching improves disk performance, but a power outage or equipment failure might result in data loss or corruption. It’s recommended only for disks with a backup power supply.

Some third-party programs require disk write caching to be enabled or disabled. If your disk are used for Event Store databases, it’s highly recommended to disable disk caching to help ensure that data is durable when the machine might experience a power, device or system failure.

In Ubuntu, it’s easy to check out whether disk caching is enabled on your disk or not by running below command:

Replace /dev/sda with your device and you’ll see below similar outputs:

Model=WDC WD3200BPVT-22JJ5T0, FwRev=01.01A01, SerialNo=WD-WX61EC1KZK99
Config=< HardSect NotMFM HdSw>15uSec SpinMotCtl Fixed DTR>5Mbs FmtGapReq >
RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=50
BuffType=unknown, BuffSize=8192kB, MaxMultSect=16, MultSect=off
CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=625142448
IORDY=on/off, tPIO=, tDMA=
PIO modes: pio0 pio3 pio4
DMA modes: mdma0 mdma1 mdma2
UDMA modes: udma0 udma1 udma2 udma3 udma4 udma5 *udma6
AdvancedPM=yes: unknown setting WriteCache=enabled
Drive conforms to: Unspecified: ATA/ATAPI-1,2,3,4,5,6,7

* signifies the current active mode

The words in red, WriteCache=enabled, means caching is enabled! To disable it, edit the “/etc/hdparm.conf” with your favorite editor, here I use vi as example:

Uncomment the line “#write_cache = off” (without quotes) by removing the # at its beginning. So it looks like:

# -W Disable/enable the IDE drive’s write-caching feature
write_cache = off

After that, restart your computer and check out the write caching status again to make sure it’s disabled.

Ji m

I’m a freelance blogger who started using Ubuntu in 2007 and wishes to share my experiences and some useful tips with Ubuntu beginners and lovers. Please comment to remind me outdated tutorial! And, notify me if you find any typo/grammar/language mistakes. English is not my native language. Contact me via [email protected] Buy me a coffee: https://ko-fi.com/ubuntuhandbook1

6 responses to How to Disable Disk Write Caching in Ubuntu To Prevent Data Loss

Hi Ji m,
Interesting article ! I’m right now on one my Linux distros “LMDE 201303” and I just checked on my partition LMDE thanks to : sudo hdparm -i /dev/sda2 and I found this line : # -W Disable/enable the IDE drive’s write-caching feature #write_cache = off.
Looks like the job’s already done on that distro. I’ll check that on my Ubuntu 12.04. Thanks a lot anyway !

Источник

How to disable caching in Ubuntu? Does Ubuntu consume more RAM? [closed]

I just viewed Hacker News. There is something mentioned like «Linux ate my RAM». How much is it true that linux consumes more RAM. It will be helpful if someone can provide guide to remove cache of application. At first my pc use to boot faster than it does now. After installing many programs, I’m facing this problem.

If you are going to use a source at least read it before you post a question. Everything is explained here. linuxatemyram.com

well if you read everything then you should know that this is an awesome feature and not something you should be bothered with. You wont have more lag from low amounts of RAM if you have disk-caching than if you wouldn’t. If I remember correct Windows 7 does the same thing.

It’s still the worst idea I’ve ever heard. Everything will become slow and very very lagy. DON’T DO IT!

I just want to clear cache of application. Application create cache so that they start quicker next time, isn’t it? I just want to clear those application cache?

@Habi reading your comments and question, you are mixing several types of caching. Applications cache’s aren’t the same as memory cache. There are several kind of caches for different kind of uses cases.

2 Answers 2

You didn’t read the webpage. The link you provided is totally opposite to what you are claiming, so lets bring the information here for you:

What’s going on?

Linux is borrowing unused memory for disk caching. This makes it looks like you are low on memory, but you are not! Everything is fine!

Why is it doing this?

Disk caching makes the system much faster! There are no downsides, except for confusing newbies. It does not take memory away from applications in any way, ever!

What if I want to run more applications?

If your applications want more memory, they just take back a chunk that the disk cache borrowed. Disk cache can always be given back to applications immediately! You are not low on ram!

Do I need more swap?

No, disk caching only borrows the ram that applications don’t currently want. It will not use swap. If applications want more memory, they just take it back from the disk cache. They will not start swapping.

How do I stop Linux from doing this?

You can’t disable disk caching. The only reason anyone ever wants to disable disk caching is because they think it takes memory away from their applications, which it doesn’t! Disk cache makes applications load faster and run smoother, but it NEVER EVER takes memory away from them! Therefore, there’s absolutely no reason to disable it!

Now lets go with the boring explanation:

Linux philosophy is that unused RAM is wasted RAM, hence it needs to be filled with something useful, which is the cache. Caching means that instead of reading files from the old and slow HDD/SSD, it gets loaded in the RAM which is several fold faster than any of them, and at the same time it’s translate into a faster programs start up, faster operations with frequently accessed data, a big gain in performance.

How to disable caching in Ubuntu?

You can’t, or more like, you don’t need to.

Against what we are comparing here? Linux and Windows? Ubuntu and other Linux distributions? If is Windows and Linux, Linux has better memory management and smaller memory footprint than Windows. If it’s Ubuntu and another distro, is likely that you are trying to compare oranges and apples. You must take into account the Window/Desktop Manager used, kernel version, patches applied, etc.

How much is it true that linux consumes more RAM.

Again, against what we are comparing?

It will be helpful if someone can provide guide to remove cache of application.

WHY? The memory is there for any program to use if it’s needed immediately they ask for them. The linux kernel doesn’t count «cached» memory as «free» memory because is «used», but the cached memory is recoverable and gets destroyed immediately any application need it.

At first my pc use to boot faster than it does now. After installing many programs, I’m facing this problem.

Boot times is extremally subjective, and since you could have installed several programs, is not wonder that your boot times were slower. My boot time cuts around 75% once I removed Mongo DB which reviewed a 300MB database. Just that program added around 2 minutes to my boot time, I disabled to start it at boot time and now my system boots in >1 minute. You should review what programs starts at boot times and disabled them instead. Disabling the cache will not help to make your system boot faster and on the contrary will make your system overall performance worse.

Application create cache so that they start quicker next time, isn’t it? I just want to clear those application cache?

Deppends the application, most applications doesn’t use a cache without a very good reason. Browsers for example, has a cache. But this cache is of internet’s objects (images, JavaScript, CSS, static content, etc) and the web page loads times are faster in subsequent visits. Other applications use cache’s for storing thumbnails, frequently accessed data, etc. Those applications use in-disk caching. That cache isn’t loaded in memory until the application starts, and it isn’t count as cached memory.

Applications normally can’t submit cached pages to the kernel. The memory cache is entirely controlled by the kernel and it dictates what should be cached.

Источник

ext4 отключить кэш диска

Имеется ли возможность отключения кэширования для раздела с файловой системой ext4 ? На нём расположены файлы БД, и кэш обеспечивается самой СУБД. Возникает ситуация, когда ненужный мне кэш ext4 занимает лишнюю память, провоцируя своппинг и общую потерю производительности.

CentOs 2.6.32-279.9.1.el6.x86_64 #1 SMP Tue Sep 25 21:43:11 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux СУБД Progress (OpenEdge)

На нём расположены файлы БД, и кэш обеспечивается самой СУБД

1. В ФС нет кэша по сути, есть page cache данных с диска, он независим от FS.

2. Выставь vm.swappiness=0 и кеш ни при каких условиях не будет вызывать своппинг т.к. будет очищаться при первом требовании.

В общем проблема надуманная.

blind_oracle ★★★★★ ( 09.10.12 14:03:22 MSK )
Последнее исправление: blind_oracle 09.10.12 14:03:31 MSK (всего исправлений: 1)

1. В ФС нет кэша по сути, есть page cache данных с диска, он независим от FS.

значит я неверно диагностировал причину своппинга, но для СУБД я выделяю буфер 32 гига (из семидесяти), что-то же «отжирает остальное. » может быть как раз page cache

2. Выставь vm.swappiness=0 и кеш ни при каких условиях не будет вызывать
своппинг т.к. будет очищаться при первом требовании

Имеется ли возможность отключения кэширования для раздела с файловой системой ext4 ?

. When a directory with the `D' attribute set is modified, the changes are written synchronously on the disk; this is equivalent to the `dirsync' mount option applied to a subset of the files. . When a file with the `S' attribute set is modified, the changes are written synchronously on the disk; this is equivalent to the `sync' mount option applied to a subset of the files. 

. dirsync All directory updates within the file system should be done synchronously. This affects the following system calls: creat, link, unlink, symlink, mkdir, rmdir, mknod and rename. . sync All I/O to the file system should be done synchronously. In case of media with limited number of write cycles (e.g. some flash drives) "sync" may cause life-cycle shortening. 

Но, насколько я понимаю, это не отменяет кеш чтения, хотя при записи кеширования не будет.

Источник