Sync directory in linux

How to sync two folders with command line tools?

If you want the contents of folders A and B to be the same, put /home/user/A/ (with the slash) as the source. This takes not the folder A but all of it’s content and puts it into folder B. Like this:

rsync -avu --delete "/home/user/A/" "/home/user/B" 

• -a Do the sync preserving all filesystem attributes
• -v run verbosely
• -u only copy files with a newer modification time (or size difference if the times are equal)
• —delete delete the files in target folder that do not exist in the source

rsync: run rsync app, -a: do the sync preserving all filesystem attributes, -v: run verbosely, -z: compress the data during the sync (transport the data in compressed mode), —delete: delete the files in target folder that do not exist in the source, /home/user/A: source folder, /home/user/B: target folder

Hi SonicARG, I totally forgot to get back to this and put the explanation, thanks submitting the explanation, I put yours in the answer, hope you don’t mind.

Rsync is primarly meant to copy files between different computers, as explained here it can serve the purpose to sync directories as well. So the -z option is interesting to reduce network traffic and thus enhance the performance of an rsync between 2 computers: ( read data from disk -> compress) ===network===> (uncompress->write to disk) Using -z to sync 2 directories on the same host is a bit silly and waste of cpu cycles as you would get (read data from disk -> compress -> uncompress -> write to disk)

I’ve tried the command but it create a sub-dir /home/user/B/A instead of overwrite A’s content to B’s content. Could you help me to have a look on it?

Important note in response to Luke’s comment regarding trailing slashes — from the manpage: «A trailing slash on the source changes this behavior to avoid creating an additional directory level at the destination. You can think of a trailing / on a source as meaning «copy the contents of this directory» as opposed to «copy the directory by name», but in both cases the attributes of the containing directory are transferred to the containing directory on the destination.» — Bottom line, add a trailing slash to the source if you don’t want a subdirectory created on the destination

You could unison tool developed by Benjamin Pierce at U Penn.

Let us assume you have two directories,

/home/user/Documents/dirA/ and /home/user/Documents/dirB/

To synchronize these two, you may use:

~$ unison -ui text /home/user/Documents/dirA/ /home/user/Documents/dirB/

In output, unison will display each and every directory and file that is different in the two directories you have asked to sync. It will recommend to additively synchronize (replicate missing file in both locations) on the initial run, then create and maintain a synchronization tree on your machine, and on subsequent runs it will implement true synchronization (i.e., if you delete a file from . /dirA , it will get deleted from . /dirB as well. You can also compare each and every change and optionally choose to forward or reverse synchronize between the two directories.

Optionally, to launch graphical interface, simply remove the -ui text option from your command, although I find the cli simpler and faster to use.

The answer from TuxForLife is pretty good, but I strongly suggest you use -c when syncing locally. You can argue that it’s not worth the time/network penalty to do it for remote syncs, but it is totally worth it for local files because the speed is so great.

-c, --checksum This forces the sender to checksum every regular file using a 128-bit MD4 checksum. It does this during the initial file-system scan as it builds the list of all available files. The receiver then checksums its version of each file (if it exists and it has the same size as its sender-side counterpart) in order to decide which files need to be updated: files with either a changed size or a changed checksum are selected for transfer. Since this whole-file checksumming of all files on both sides of the con- nection occurs in addition to the automatic checksum verifications that occur during a file's transfer, this option can be quite slow. Note that rsync always verifies that each transferred file was correctly reconstructed on the receiving side by checking its whole-file checksum, but that automatic after-the-transfer verification has nothing to do with this option's before-the-transfer "Does this file need to be updated?" check. 

This shows how having the same size and time stamps can fail you.

The setup

$ cd /tmp $ mkdir -p /1/2/ $ echo "\___________from A" | \ tee A/1/2/x | tee A/1/2/3/y | tee A/1/2/4/z | \ tr A b | \ tee b/1/2/x | tee b/1/2/3/y | tee b/1/2/4/z | \ tee b/1/2/x0 | tee b/1/2/3/y0 > b/1/2/4/z0 $ find A b -type f | xargs -I% sh -c "echo %; cat %;" A/1/2/3/y \___________from A A/1/2/4/z \___________from A A/1/2/x \___________from A b/1/2/3/y \___________from b b/1/2/3/y0 \___________from b b/1/2/4/z \___________from b b/1/2/4/z0 \___________from b b/1/2/x \___________from b b/1/2/x0 \___________from b 

The rsync that copies nothing because the files all have the same size and timestamp

$ rsync -avu A/ b building file list . done sent 138 bytes received 20 bytes 316.00 bytes/sec total size is 57 speedup is 0.36 $ find A b -type f | xargs -I% sh -c "echo %; cat %;" A/1/2/3/y \___________from A A/1/2/4/z \___________from A A/1/2/x \___________from A b/1/2/3/y \___________from b b/1/2/3/y0 \___________from b b/1/2/4/z \___________from b b/1/2/4/z0 \___________from b b/1/2/x \___________from b b/1/2/x0 \___________from b 

The rsync that works correctly because it compares checksums

$ rsync -cavu A/ b building file list . done 1/2/x 1/2/3/y 1/2/4/z sent 381 bytes received 86 bytes 934.00 bytes/sec total size is 57 speedup is 0.12 $ find A b -type f | xargs -I% sh -c "echo %; cat %;" A/1/2/3/y \___________from A A/1/2/4/z \___________from A A/1/2/x \___________from A b/1/2/3/y \___________from A b/1/2/3/y0 \___________from b b/1/2/4/z \___________from A b/1/2/4/z0 \___________from b b/1/2/x \___________from A b/1/2/x0 \___________from b 

Источник

How to rsync a directory?

Rsync stands for “Remote Sync.” It is a synchronization method for directories and files that works both locally and remotely. This tool employs the “delta algorithm” that reduces the amount of copied data by only transferring the modified part of the files or directories. In this post, we will go through the basics of using this tool to sync a directory.

What is Rsync?

Rsync is a network-enabled utility that is highly flexible in terms of syncing files and directories. It is included in most of the Unix and Linux distributions as a popular utility for system scripts and its widespread use on these operating systems.

Now, let’s check out the basic syntax of the rsync command.

Rsync Syntax:

Here “Option” refers to rsync options, “Source” for the source directory, and “Destination” for the destination directory.

To demonstrate to you the method of syncing a directory in Linux, we created two test directories, “testdir1” and “testdir2,” in our system.

1. Rsync directory locally:

First of all, list out all the directory content to get to know the files or folders. We have selected “testdir1” for this purpose.

Rsync directory with the “-r” option:

For direct syncing, utilize the recursive “-r” option in the rsync command. The below-given syntax will sync the content from the source to its destination directory. The “/” points towards the content of the source directory. Without this slash, the rsync command will place the source directory within the destination directory.

Write out this command to start local syncing a directory to another.

$ rsync -r testdir1 / testdir2

List out the “testdir2” content to view the changes made by rsync.

Rsync directory with the “-a” option:

You can also utilize various options or flags that rsync supports. “-a” flag is one of them. This flag helps the rsync command sync a directory recursively and maintain the device files, modification times, symbolic links, owner, group, and file permissions.

$ rsync -a testdir1 / testdir2

Rsync directory with the “-v” option:

To show the rsync process on the terminal “-v” option is used with the rsync command.

To check the difference that comes by using a “/” at the end of the source directory, write out this command:

The output clearly states that now the source directory is itself transferred.

Rsync directory with the “-P” option:

The “-P” shows the progress of the syncing directory by displaying the progress bar on the terminal.

You can also utilize the same command for resuming any interrupted transfers.

To check if using the same rsync command only sync the modification we made in the directory, create some test files in your source directory.

After that, execute the same rsync command and see the intelligent behavior of the rsync command.

2. Rsync directory from local to the remote system:

The operation of syncing a local directory to any remote system is known as “push,” as it pushes the directory from your local system to the remote one.

Follow the syntax of push operation for syncing a directory:

In our case, we are going to sync the “testdir1” directory to the remote host “10.0.2.15”.

3. Rsync directory from remote to local system:

As we all know, the opposite of push is “pull.” In the context of syncing, pull operation sync a directory from the remote system to your local system.

$ sudo rsync -v username @ remote_host:Source Destination

$ sudo rsync -v linuxhint @ 10.0.2.15: / home / linuxhint / testdir1 / testdir2

Conclusion:

Rsync is a tool that can improve the reliability of local directory syncing and the file transfer process over remote systems. You can use the rsync command to create complex backups and smooth control over what and how a directory will sync. In this post, we have shown you the various forms of using rsync, which include: rsync directory within a local system, rsync directory from local to a remote system, and also from remote to the local one.

About the author

Talha Saif Malik

Talha is a contributor at Linux Hint with a vision to bring value and do useful things for the world. He loves to read, write and speak about Linux, Data, Computers and Technology.

Источник