Linux write cache usb

Linux USB Flash direct write / write cache

Lately I’ve been using USB2 flash drives to transfer a lot of data to my company. When copying the Data to the flash drive, I noticed that the data is not written directly to the drive. It wrote with 200MB/sec to cache(?). I then umount or sync the drive and it takes hours to actually write the data to the drive. I also tried using cp instead of Nautilus, but it had the same behavior. dd is not an option for copying normal files. Is there any way to make Linux write the data directly without caching when writing on USB storage? Or maybe reduce caching so that I don’t have to wait for hours when unmounting the drive? What’s annoying me most is that the progress bar finishes copying and I then have to wait a unspecified time for sync to write the data. I’m on a Fedora 25 machine. [Update] Tried to clarify my question. [Update 2] Found a similar question without answer:Reduce cache size of flash storage devices

I’ve tried with sync. It’s the same behavior as with umount, as it writes the data after the copy process. It is really annoying when you see the progress bar finished and then have to wait a undefined time before you can remove the flash drive. I don’t mind waiting for a while when writing >10GB of data to a USB2 device. It’s just annoying not to know how long it will take and when it really is finished. The only thing I like about windows where the caching is less of a problem.

There is an answer here, which affects the entire system not just removable devices, but a number of desktop users have found it useful: unix.stackexchange.com/questions/292024/…

Although, some of the claims in the linked LWN.net article are very suspicious IMO. unix.stackexchange.com/questions/480399/…

I too find it annoying. however, i do watch the progress after issuing umount with watch -n 5 iostat /dev/your device ‘

Источник

Turn off write cache on all USB External Drives (Debian / Ubuntu / Linux)

How do you turn off write cache permanently for external USB drives in Linux (I’m using Ubuntu)? I’ve tried: Command line method : hdparm -W0 /dev/sdb Configuration file method (seems to be another way to do it permanently) : The next thing I found was modifying the «hdparm.conf» file located in /etc. However, that seems to require that you know ahead of time the drive letter, «sdb, sdc. » etc. How do you just blanket cover ALL drives no matter what, or specifically all USB externals? The key problem is.. I’m not sure of the order I will connect these USB external drives, therefore the drive letter will change, so I can’t just name «sdb» specifically since it might not be «sdb». Is this a situation where hdparm.conf can have an entry multiple times over even for drives letters that aren’t even connected? That way, no matter what drive is connected it will just turn the write_cache off preemptively. Or will that mess something up? Possible /etc/hdparm.conf example (will this work?):

/dev/sda < write_cache = off >/dev/sdb < write_cache = off >/dev/sdc < write_cache = off >/dev/sdd

Again, those drives may not even exist yet. Just covering as many letters of the alphabet preemptively so no matter what, the write_cache is off. I would think there’s got to be a better way than this though.. Any other ideas? Or better approaches? Thank you!

1 Answer 1

You need to use the -K option. From man hdparm

 -k Get/set the "keep_settings_over_reset" flag for the drive. When this flag is set, the drive will preserve the -dmu settings over a soft reset, (as done during the error recovery sequence). This option defaults to off, to prevent drive reset loops which could be caused by combinations of -dmu settings. The -k option should therefore only be set after one has achieved confidence in correct system operation with a chosen set of configuration settings. In practice, all that is typically nec‐ essary to test a configuration (prior to using -k) is to verify that the drive can be read/written, and that no error logs (kernel messages) are generated in the process (look in /var/adm/messages on most systems). -K Set the drive´s "keep_features_over_reset" flag. Setting this enables the drive to retain the settings for -APSWXZ over a soft reset (as done during the error recovery sequence). Not all drives support this feature. 

Источник

How to get Windows-like caching behavior for USB external drive

@KenMollerup I’m not sure. Would it matter? I have enough RAM that the files will stay in the disk cache so that making thumbnails from them shouldn’t cause any I/O.

Well on my system whenever i have moved or updated an external disk, and try to unmount or eject, it claims its busy, it turned out it was just creating thumbnails of pictures documents pdfs and such, and that can take a very long time, flushing the cash should be an ongoing process — and not delay you for a quarter of an hour.

Re. «Thunar claims that it’s written my files to a thumb drive, then I spend 15 minutes waiting for the drive to unmount.» Could you please explain how you monitor the time taken to unmount?

@DKBose I click the unmount icon in Thunar. A message pops up similar to «we’re writing your data, don’t disconnect your drive.» I see in my Disk Performance Monitor panel widget that the disk is being written.

3 Answers 3

The settings that control how many dirty pages are allowed to accumulate before being written to disk are controlled by the virtual memory subsystem.

For systems with a lot of RAM, they usually need to be tweaked. Unfortunately, these settings are system-wide. (Ugh, Linus.)

See the current settings with:

sudo sysctl -a | grep vm.dirty 

vm.dirty_background_bytes = 0 vm.dirty_background_ratio = 5 vm.dirty_bytes = 0 vm.dirty_expire_centisecs = 3000 vm.dirty_ratio = 10 vm.dirty_writeback_centisecs = 500 

Take a look at the documentation for these settings. On my machine, vm.dirty_ratio is set to 10 (ie, 10% of RAM). This means that the disk cache will swallow up to 10% x 64GB = 6.4GB of data before writing it to disk.

I am going to try tweaking these settings by writing the following into my /etc/sysctl.conf :

vm.dirty_bytes = 67108864 vm.dirty_background_bytes = 1048576 vm.dirty_expire_centisecs = 30 vm.dirty_writeback_centisecs = 30 

I’ll update this answer as I learn more, but I think these aggressive settings are fine for my SSD, which has its own large on-drive write cache.

I’m not sure if I can answer you what you want to hear. There is the «oldschool» option to disable that buffer — you may use the fstab sync attribute for this.

This will bring you away from the modern way of your device lying to you «your quick computer has already finished» when it hasn’t. The reality still is «it’s done when it’s done» and sync will let your computer show you when it is really done.

If you in fact are annoyed by the «progress bar» (or waiting times in the shell) and want to tweak the buffer size and behaviour you may do that by editing the sysfs/block options. Compare https://unix.stackexchange.com/questions/30286/can-i-configure-my-linux-system-for-more-aggressive-file-system-caching for this. The list of availible options may vary depending on the ioscheduler you(r kernel) use(s).

Now the question is: How to apply this for USB sticks.

AFIAK on Linux Mint udisks handles USB Stick automounting and AFAIK there is no elegant way to teach udisks sensible defaults. You may try udisks-glue to help you.

Источник

How to switch off caching for usb device when writing to it?

I have problems transferring binary programs to a micro-controller prototype board when using ubuntu/kubuntu to mount the board via usb. With MacOS and Windows there are no problems. I have found the writing from Ubuntu definitely changes the written file. I suspect that the write caching is the problem. How can I disable it, or what else might be the problem.

3 Answers 3

You can try using hdparm to set the write caching feature to off at runtime. (You will probably need to run this after every boot or every time you remove and reinsert the device)

sudo hdparm -W 0 /dev/devicename

Please ensure you know the correct device name ( sdb , sdc or so on).

You can find out the device name by running:

You can also edit /etc/fstab and add the mount options sync,dirsync however I am not very familiar with how /etc/fstab works with removable devices. I think they need a permanent mount point.

kudos to @NightwishFan for the hdparm switch to set the drive’s write-caching feature.

Note that the sync mount option decreases the speed by a magnitude (over 10 times). flush mount option ensures an effective writing with no performance cost. Also I’ve been using it with no issues for a few years on all the FAT devices (spinning or flash-memory based) I’ve used: Actually the whole set of mount options is noauto,users,noatime,flush,umask=111,dmask=000 , set manually or in /etc/fstab.

I have replaced «sync» in favor of «noatime,flush» like stated in @tuk0z answer.

This lead to the USB floppydrive behaving the same like sync, but it writes files so much faster, like the normal operationspeed you expect it to be.

Reading image files from floppy does not result in random artifacts anymore. Also reading DD floppies does not result in weird characters in filenames.

Perhaps it’s a bit unfortunate how the official documentation of «mount» put the usage in context.

Источник