Linux copy changed files only

Rsync copy only changed files; ignore file modification time stamps

It would be worth reviewing whether you can revert to trusting timestamps (for example, use cp -p and scp -p instead of just cp or scp when copying files around). The consequent efficiency improvement within rsync is vast.

1 Answer 1

Description

Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon. It offers a large number of options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Rsync is widely used for backups and mirroring and as an improved copy command for everyday use.

Rsync finds files that need to be transferred using a lqquick checkrq algorithm (by default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the file’s data does not need to be updated.

So, the default behavior as we can see on the description is:

• Copy tool, that works locally or remotelly
• Lots of options
• Delta-transfer algorithm by default, that will only transfer sets of files that differ to reduces the amount of network usage
• Widely used tool for mirroring and backups
• checkrq algorithm that does what you want on condition 1: if len(f1) != len(f2) then rsync
• Destination is the one to be affected if no options are passed.

Now, it is just a matter of looking for options related to checksum. Searching at the manuals:

-c, --checksum This changes the way rsync checks if the files have been changed and are in need of a transfer. Without this option, rsync uses a lqquick checkrq that (by default) checks if each file's size and time of last modification match between the sender and receiver. This option changes this to compare a 128- bit checksum for each file that has a matching size. Generating the checksums means that both sides will expend a lot of disk I/O reading all the data in the files in the transfer (and this is prior to any reading that will be done to transfer changed files), so this can slow things down significantly. 

The description of —checksum is exactly what you want on if len(f1) == len(f2) and md5sum(f1) != md5sum(f2) then rsync . It will do a 128-bit checksum on each size matching file.

But be careful cause this option will, depending on the situation, increase you I/O significantly.

Источник

How to Use Rsync to Sync New or Changed/Modified Files in Linux

As a system administrator or Linux power user, you may have probably come across or even on several occasions, used the versatile Linux Rsync tool, which enables users to expeditiously copy or synchronize files locally and remotely. It is as well a great tool popularly used for backup operations and mirroring.

Some of its eminent features and advantages include; it is exceptionally versatile in that, it can copy locally, to/from a remote shell or remote rsync, it is also remarkably flexible, allowing users to specify any number of files to copy.

Furthermore, it permits copying of links, devices, file or directory owner, groups, and permissions. It also supports usage without root privileges coupled with many more.

One imperative differential of rsync in comparison to other file-coying commands in Linux is its use of the remote-update protocol, to transfer only the difference between files or directory content.

Therefore, in this article, we shall examine how rsync can help us only sync new or changed files or directory content while making backups and beyond in Linux.

To start with, you need to remember that the conventional and simplest form of using rsync is as follows:

# rsync options source destination

That said, let us dive into some examples to uncover how the concept above actually works.

Syncing Files Locally Using Rsync

Using the command below, being able to copy files from my Documents directory to /tmp/documents directory locally:

$ rsync -av Documents/* /tmp/documents

In the command above, the option:

1. -a – means archive mode
2. -v – means verbose, showing details of ongoing operations

By default, rsync only copies new or changed files from a source to destination, when I add a new file into my Documents directory, this is what happens after running the same command a second time:

$ rsync -av Documents/* /tmp/documents

As you can observe and notice from the output of the command, only the new file is copied to the destination directory.

The —update or -u option allows rsync to skip files that are still new in the destination directory, and one important option, —dry-run or -n enables us to execute a test operation without making any changes. It shows us what files are to be copied.

$ rsync -aunv Documents/* /tmp/documents

After executing a test run, we can then do away with the -n and perform a real operation:

$ rsync -auv Documents/* /tmp/documents

Syncing Files From Local to Remote Linux

In the example below, I am copying files from my local machine to a remote server with the IP address – 10.42.1.5. So as to only sync new files on the local machine, that do not exist on the remote machine, we can include the —ignore-existing option:

Subsequently, to sync only updated or modified files on the remote machine that have changed on the local machine, we can perform a dry run before copying files as below:

$ rsync -av --dry-run --update Documents/* [email protected]:~/all/ $ rsync -av --update Documents/* [email protected]:~/all/

To update existing files and prevent the creation of new files in the destination, we utilize the —existing option.

You can run through the man rsync page to discover additionally useful options for advanced usage, as I had mentioned earlier, rsync is a very powerful and versatile Linux tool, and many System Administrator and Linux power users know just how advantageous it is.

Most importantly, you can as well share your view on the examples we have covered here or even better still, offer us valuable tips on using this vital command line tool through the comment section below.

Источник

Shell Script (Linux): Copy and overwrite only files that have changed in destination?

I want to do the following in Shell Script: I have two directories and I want to copy over only these files from the source directory that have changed or are missing in the destination directory. Note 0: (in case the above is clear): I want to replace destination folder with source directory. Changed files in destination should be restored to the original state from the source directory. Unchanged files should not be touched (to save time). Note 1: I looked up rsynch but I cannot figure out if it can do what I want. There is only -u but that doesn’t seem to do what I want. EDIT: This is my script: (fixed)

#!/bin/sh ################################################################################ # Copies resources to build output directory (Ubuntu) ################################################################################ directoryWtSource="/usr/share/Wt" directoryBuildOutput="bin__output__" # Get the absolute path to the script scriptPath=`readlink -e $0` directoryDestination=`dirname $scriptPath` ################################################################################ # Start ################################################################################ cd .. directoryWtSourceResources=$directoryWtSource/resources directoryBuildOutputResources=$/$directoryBuildOutput if [ ! -d $/$directoryBuildOutput ] then mkdir $/$directoryBuildOutput fi #echo $directoryWtSourceResources #echo $directoryBuildOutputResources # BEGIN - REMOVE THIS: No need for cp #if [ ! -d $directoryBuildOutputResources ] #then # echo " --> No destination directory found: copying resources. " # cp -Rp $directoryWtSourceResources $directoryBuildOutputResources #else # echo " --> Destination directory found: won't copy." #fi # END - REMOVE THIS rsync -arv $directoryWtSourceResources $directoryBuildOutputResources ################################################################################ # End ################################################################################ 

1 Answer 1

rsync -av source/ destination/ 

This command will also print out the list of files that have been replaced.

@Zingam No need for -r : -a includes it. Could you give a detailed file list ( ls -ls ) of your source and destination? Edit the question to include it.

Источник

Linux how to copy but not overwrite? [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.

This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.

I want to cp a directory but I do not want to overwrite any existing files even it they are older than the copied files. And I want to do it completely noninteractive as this will be a part of a Crontab Bash script. Any ideas?

8 Answers 8

-n, --no-clobber do not overwrite an existing file (overrides a previous -i option) 

cp -n myoldfile.txt mycopiedfile.txt 

Note, this will exit with an error if the file exists. To exit with success, try cp -n source.txt destination.txt || true

@galenandrew Confirmed. Thank you. My project wasn’t building in Xcode after adding a run script to my target.

Even with Ubuntu 18.04, the behavior is the same, that when the file exists, the command exit without erro, that is, an no op! This is dangerous!

rsync -a -v --ignore-existing src dst 

As per comments rsync -a -v src dst is not correct because it will update existing files.

Complete command rsync -a -v —ignore-existing is indeed the correct answer, instead of cp -u above.

If a previous copy was interrupted and only a truncated file copied, I’m sure cp -u won’t re-copy it. but will rsync, with —ignore-existing? Probably not either. so that makes them perfectly equivalent, right?

absolutely. I don’t want to install rsync into my docker container, just to do a copy! I can’t imagine why we should use a cannon when a BB gun will suffice.

Is what you want. See the man page.

The man page: -n Do not overwrite an existing file. (The -n option overrides any previous -f or -i options.)

false | cp -i source destination 2>/dev/null 

Updating and not overwriting is something different.

Obviously, this command won’t work if you’ll try to copy more than ARG_MAX files. To work-around this case, check this link.

For people that find that don’t have an ‘n’ option (like me on RedHat) you can use cp -u to only write the file if the source is newer than the existing one (or there isn’t an existing one).

[edit] As mentioned in the comments, this will overwrite older files, so isn’t exactly what the OP wanted. Use ceving’s answer for that.

OP asked not to overwrite existing files even if they are older than copied files, so -u doesn’t actually fit purpose.

It might not be what the OP asked for, but it’s exactly what I needed for my Uberspace (Centos 7). Thanks!

Источник

How can I do a «copy if changed» operation?

I would like to copy a set of files from directory A to directory B, with the caveat that if a file in directory A is identical to a file in directory B, that file should not be copied (and thus its modification time should not be updated). Is there a way to do that with existing tools, without writing my own script to do it? To elaborate a bit on my use-case: I am autogenerating a bunch of .c files in a temporary directory (by a method that has to generate all of them unconditionally), and when I re-generate them, I’d like to copy only the ones that have changed into the actual source directory, leaving the unchanged ones untouched (with their old creation times) so that make will know that it doesn’t need to recompile them. (Not all the generated files are .c files, though, so I need to do binary comparisons rather than text comparisons.) (As a note: This grew out of the question I asked on https://stackoverflow.com/questions/8981552/speeding-up-file-comparions-with-cmp-on-cygwin/8981762#8981762, where I was trying to speed up the script file I was using to do this operation, but it occurs to me that I really should ask if there’s a a better way to do this than writing my own script — especially since any simple way of doing this in a shell script will invoke something like cmp on every pair of files, and starting all those processes takes too long.)

7 Answers 7

You can use the -u switch to cp like so:

 -u, --update copy only when the SOURCE file is newer than the destination file or when the destination file is missing 

Источник