File copy linux speed

Is there a faster alternative to cp for copying large files (~20 GB)?

I am a graduate student, and the group in which I work maintains a Linux cluster. Each node of the cluster has its own local disk, but these local disks are relatively small and are not equipped with automatic backup. So the group owns a fileserver with many TBs of storage space. I am a relative Linux novice, so I am not sure what are the specs of the fileserver in terms of speed, networking ability, etc. I do know from experience that the local disks are significantly faster than the fileserver in terms of I/O. About a dozen or so people use the fileserver. Using cp to copy a ~20 GB file from the fileserver to one of the local disks takes about 11.5 minutes in real time on average (according to time ). I know that this cp operation is not very efficient because (1) time tells me that the system time for such a copy is only ~45 seconds; and because (2) when I examine top during the copy, %CPU is quite low (by inspection, roughly 0-10% on average). Using cp to copy the same ~20 GB file from one folder on the local disk to another folder on the same local disk takes less time — about 9 minutes in real time (~51 seconds in system time, according to time ). So apparently the fileserver is somewhat slower than the local disk, as expected, but perhaps not significantly slower. I am surprised that copying from local to same local is not faster than 9 minutes. I need to copy ~200 large files — each ~20 GB — from the fileserver to one of the local disks. So, my question is: Is there a faster alternative to cp for copying large files in Linux? (Or are there any flags within cp that I could use which would speed up copying?) Even if I could somehow shave a minute off this copying time, that would help immensely. I am sure that buying new, faster hardware disks, but I don’t have access to such resources. I am also not a system administrator — I am only a (novice) user — so I don’t have access to more detailed information on the load that is on the disks. I do know that while about a dozen people use the fileserver daily, I am the only person using this particular node/local disk.

Источник

The faster and safer way to copy files in Linux than cp

Have you tried copying large files on Linux, sometimes it just takes ages? I was doing that when I thought there should be a faster and better way to copy files in Linux. So I started searching and came across these commands which can offer better copying speed.

As simple cp command is very useful but sometimes it can slow down the process. These commands should help you get your copying done in the fastest way.

For copying sometimes the tar command can be a better alternative. Sometimes providing a faster and safer alternative. Here is how to use tar.

How to copy files faster using tar command in Linux

To copy files open a terminal, it can generally be opened by Ctrl + Alt + T. Now in terminal change the current directory to the folder from which you want to copy files.

Now just run the command below to copy files.

tar cf - . | (cd /output/directory/ && tar xvf -)

While executing the command just replace /output/directory with the directory in which you want to copy files. All the files and subfolders are copied from the current directory to the /output/directory.

Now if you want, you can also use pv to help you monitor the progress of copying files. For example:

tar cf - . | pv | (cd /output/directory && tar xvf -)

Cp vs tar why is the speed difference?

In cp vs tar, tar sometimes has much higher copy speed than cp. The reason behind that is cp does open-read-close-open-write-close in a loop. And while tar does reading and writing in a separate process. Tar also uses multiple threads to read and write and can even several files at once.

This makes tar clearly win in comparison of cp vs tar. As tar works in a more speedy and efficient way.

Here are other alternatives that you can use to copy files in fastest way on Linux.

Another alternative command

The other command that is fast and very versatile for copying files between two locations is rsync . It can be used to copy between local as well as remote locations.

To copy files using rsync you need to enter the command below.

rsync -a Downloads/songs/abc.zip Downloads/music/

To view the progress while copying large size files you can use the command below.

rsync --info=progress2 -auvz Downloads/songs/abc.zip Downloads/music/

If you are wondering, here is what -auvz stands for.

• a: archive files and directory while synchronizing.
• u: Don’t copy files from source to destination, if destination already has newer files.
• v: Verbose output.
• z: Compress data during the transfer.

In the above example, copying is being done locally, but you can use rsync for copying over remote locations also.

You can also use like n for the dry run (to perform a trial run without synchronization) and r for recursive (sync files and directories recursively). If you are transferring from a remote location you can also use -e ssh to secure communications. Here are some other commands that you can use if you want.

If your system doesn’t come preinstalled with rsync, then you can install using the commands below.

cp vs rsync which one is better?

Although rsync is not generally faster than cp, but as it only syncs files that are modified or new. It can offer better speed when synchronizing files. The rsync also has multiple advanced options that are not available in cp.

How to install rsync

rsyn comes pre-installed on most Linux distros. But if it’s not preinstalled you can install with the command below.

On Debian and Ubuntu-based systems use the command below.

sudo apt-get install rsync

On CentOS/RHEL based systems use the command below.

For SUSE/Open based systems.

These commands will install rsync on your system. Now you can try copying files with a better speed.

SCP for copying

Secure copy or also known as SCP, can also be used for copying. Although it is not for fast copying, it can be used for the secure transmission of files between a local host and a remote host. Or between two remote hosts. So when you are looking for secure transmission of files you can use this method.

Here is how you can use SCP for file transferring from a local to a remote host.

For transferring of file from a remote host to local host you can use the command below.

scp [email protected]_host:file_name.zip /local/directory/

For transferring of a file from remote host to remote host use the command below.

scp [email protected]_host:/remote/directory/file_name.zip [email protected]_host:/remote/directory/

I hope these commands should help you. Here are some other articles that I think you might like How to add a user to a group in Linux and how can Linux NTFS file.

Источник

Why do file copy operations in Linux get slower over time?

I have 12 ~200 MB files. Copying one of them to another drive takes 20 seconds. When I try to copy all of them at once, it first says it’s going to take 5 minutes, then it keeps slowing down into the hours and days. I’m copying the files through Thunar in XFCE, but I remember having similar problems using the command line. Why does this occur and what can I do to remedy it, other than just copying the files one by one?

@tubaguy50035 The files are not in sub-folders. That being said, it’s likely that the source drive may have errors. I’ll check.

6 Answers 6

This is a known issue with the Linux kernel itself that has been known for ages, no one was tackled the issue yet.

there is currently no fix or known work-around, aside from using Windows which increases in speed and then normalizes. I find this bug to be a very high-priority issue, but the programmers do not feel the same way. It is this bug alone that is starting to drive me away from Linux.

Citation Required! It might help work our an answer if you could give some documentation that points to this bug

and here, which can be found as a link in the original citation. bugzilla.kernel.org/show_bug.cgi?id=12309

You should try rsync to monitor your transfer speed. For example:

rsync -PSauv sourcedir/* destdir/ 

If you actually see a drop in transfer speed, as opposed to files that are just big, then you need to look at a few other things.

1. If your disk is I/O-bound, you will have problems. You often see this when writing heavily to a flash drive or an external USB device.
2. Test both your disks with hdparm -Tt to see if one is dramatically slower than it should be, or dramatically slower than the other device which can lead to I/O issues.
3. Check your dmesg output and your SMART status to make sure you don’t have any obvious disk errors. You might also consider badblocks -n if you suspect disk errors on a non-SMART drive.

There are lots of reasons why a disk can thrash or perform sub-optimally, but until you can measure the problem, you won’t know what the problem is. 🙂

Remember that also the file copy process is somehow cached by Linux. If you take a single file (even if a 200MB one) and start copying it to another place, Linux caches, with great probability, the whole file in memory and writes it in its new location. Then, if you choose to copy that file again and with it a bunch of other files, then the first file is already cached and then probably skips the whole «read part», while the others are first read from and then written to their sources/destinations. This feature can even lead to infinite transfer speeds and similar oddities, so don’t trust any measurement that could be invalidated by previous chaching mechanisms.

I tried to copy a 4 GB file to a fat32 usb stick.

after 1 GB has been copied, copy speed reduced from c. 20 MB/s to c. 0.5 MB/s.

I formatted the stick to ext4 but it didn’t help.

Then I formatted the stick to NTFS as advised here, and had a dramatic improvement in speed:

$ rsync --progress movie.mkv /run/media/user/09D8E089029531DD/ movie.mkv 3,759,209,408 100% 7.90MB/s 0:08:46 (xfr#1, to-chk=0/1) 

PS: this doesn’t answer the question but hopefully may solve the problem with speed.

Источник