Linux rsync to qnap

QNAP NAS Community Forum

Remote Replication via rsync from Linux to Qnap?

Remote Replication via rsync from Linux to Qnap?

Post by databala » Fri Jan 13, 2012 5:37 am

Hi,
I just need to know if I can use Remote Replication via rsync to start a scheduled rsync job so that data can backed up from a linux server to any QNAP. I want to do this only using the Qnap interface (not so good at Linux and I don’t want to play with Crontab etc). Thanks for your reply.
Regards
Databala

P3R Guru Posts: 13073 Joined: Sat Dec 29, 2007 1:39 am Location: Stockholm, Sweden (UTC+01:00)

Re: Remote Replication via rsync from Linux to Qnap?

Post by P3R » Fri Jan 13, 2012 3:34 pm

No, that isn’t possible. Remote Replication only does push rsync (from Qnap storage to a local or remote destination), not pull rsync.

I needed exactly the same thing and work at the commandline is necessary to achieve that. It’s quite fun when one have the time to learn.

RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn’t really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It’s your money and your data, spend the storage budget wisely or pay with your data!

Re: Remote Replication via rsync from Linux to Qnap?

Post by databala » Sat Jan 14, 2012 5:16 am

Hej, Tack för ditt svar.
I had a ReadyNAS NV+ which could do a pull rsync from a Linux server. ReadyNAS NV+ died for the second time that’s why I am looking for a Qnap. So there is no Qnap which can do a pull rsync for me (my TS-219P can’t do that).
Many thanks
Regards
Databala

P3R Guru Posts: 13073 Joined: Sat Dec 29, 2007 1:39 am Location: Stockholm, Sweden (UTC+01:00)

Re: Remote Replication via rsync from Linux to Qnap?

Post by P3R » Sat Jan 14, 2012 9:06 am

All Qnaps (including your TS-219 P) can do pull rsync.

They just can’t do it from the Remote Replication admin gui, because that was only designed for the purpose of making backups of the Qnap contents (push rsync).

As I said, I’m running what you want (hourly) but it requires some work at the command line. Not knowing anything about your environment or level of experience I’m not sure I can help you but an extremely short version goes like this:
1. Establish a SSH-connection
2. Make sure you have a working rsync command to do what you want
3. Add the command from step 2 into the crontab schedule to be run at the times you wish
4. Live happily ever after

Since it is a one time thing setting it up and it is less than 15 minutes work actually doing it (any education necessary not included in that time) I would recommend you to try. I’m sure you will be able to get help here in the forum should you have any questions.

RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn’t really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It’s your money and your data, spend the storage budget wisely or pay with your data!

Re: Remote Replication via rsync from Linux to Qnap?

Post by databala » Sat Jan 14, 2012 5:54 pm

Tack för dit svar igen.
Det blir nog att dyka i terminalen i alla fall.
Men först ska jag försöka få liv i ReadyNAS:en.
Har du något tips på hur rsync syntaxen kan se ut om jag vill backa upp ”/samba/ ” katalogen från linux servern till en ”/backup/” katalog på Qnap, inkrementell varje dag klockan 20:00.
Tack så mycket för dina svar.
Mvh
/databala

Re: Remote Replication via rsync from Linux to Qnap?

Post by jamestus » Sat Jan 14, 2012 7:36 pm

P3R wrote: As I said, I’m running what you want (hourly) but it requires some work at the command line. Not knowing anything about your environment or level of experience I’m not sure I can help you but an extremely short version goes like this:
1. Establish a SSH-connection
2. Make sure you have a working rsync command to do what you want
3. Add the command from step 2 into the crontab schedule to be run at the times you wish
4. Live happily ever after

I have just started to figure out a backup strategy from my linux machine to a qnap. On an abstract level your solution seems what I want, since you are mentioning the magic words rsync and ssh, which in my mind imply nice full and incremental backups over a secure connection. However, not being really familiar with rsync yet I don’t understand what you do in which machine. Could you post the sequence of commands that you use? It would really help.

Now it looks like
1. You take an ssh connection from your laptop to your NAS.
2. You do an rsync pull on the NAS.
3. You add the rsync pull to the crontab of your NAS.
4. And live happily.

(Personally I would prefer to do an rsync push from my laptop to the NAS over ssh, and I’m trying to figure out how to do this.)

P3R Guru Posts: 13073 Joined: Sat Dec 29, 2007 1:39 am Location: Stockholm, Sweden (UTC+01:00)

Re: Remote Replication via rsync from Linux to Qnap?

Post by P3R » Sat Jan 14, 2012 7:56 pm

This is an international forum so let’s stick with the official language. In short databala is asking for help with the specific settings.

The crontab entries would start like this:
0 20 * * * rsync .

I’m not good enough at rsync to know the syntax so you’d have to look that up yourself.
.

RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn’t really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It’s your money and your data, spend the storage budget wisely or pay with your data!

Re: Remote Replication via rsync from Linux to Qnap?

Post by jamestus » Sat Jan 14, 2012 7:58 pm

Answering my own question, an rsync push from linux to NAS via ssh seems to be straightforward. Here is an example command to be run in your linux box that you want to back up:

rsync -av -e ssh directory-to-back-up administrator@nas:/share/backup-directory/ 

Then it’s just to add a passwordless ssh login, both incremental and full backups and a cron entry. I can post my commands here after I have figured them all out.

Svenskan skulle också vara bra om jag mindes det bättre efter över 20 år.

P3R Guru Posts: 13073 Joined: Sat Dec 29, 2007 1:39 am Location: Stockholm, Sweden (UTC+01:00)

Re: Remote Replication via rsync from Linux to Qnap?

Post by P3R » Sat Jan 14, 2012 8:17 pm

My use of SSH is only to get a command prompt on the NAS, to be able to add the rsync command to the crontab. My rsync-connection is running over a firewall-to-firewall VPN so I don’t need to protect the rsync-traffic in the NAS. If the intention is to run this over the internet, you’ll have to use an independent VPN like I do or figure out some other way to protect it in the NAS.

My usage is to backup data on a very limited *nix-based system and that is the reason I needed rsync pull. Since both of you are talking about Linux on the other end, maybe setting the whole thing up there is an option? Maybe there is a nice user-friendly application application available for this at the other end?

RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn’t really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It’s your money and your data, spend the storage budget wisely or pay with your data!

Re: Remote Replication via rsync from Linux to Qnap?

Post by jamestus » Sun Jan 15, 2012 1:44 am

1. Enable ssh logins in your NAS. These only work for administrator user, so you have to use that user account to do the backup.
2. Instructions on how to set up a passwordless login from your computer to your NAS can be found here.
3. Below is a little shell script I wrote inspired by this page. Sorry that I had to include a long warranty text; I know how critical backups can be. Because I need to run this as root, I included the option to check for effective root uid.

#!/bin/sh # a shell script to back up a directory via rsync using ssh, with # following properties: 1. names the backup directory according to # backup time 2. performs full backups and by default uses hard links # to previously backed up data to conserve space 3. includes the # possibility to exclude certain directories and files from the backup # modify the first variables below to fit your situation; also take a # look at the manual page of rsync to see whether the option -x # specified below suits your needs # idea and much of the code from # http://blog.interlinked.org/tutorials/rsync_time_machine.html # NO WARRANTY # THE PROGRAM IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL, BUT # WITHOUT ANY WARRANTY. IT IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY # KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, # THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A # PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND # PERFORMANCE OF THE PROGRAM IS WITH THE USER. SHOULD THE PROGRAM # PROVE DEFECTIVE, THE USER ASSUMES THE COST OF ALL NECESSARY # SERVICING, REPAIR OR CORRECTION. # IN NO EVENT IS THE AUTHOR OF THIS PROGRAM LIABLE TO ANYBODY FOR # DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL # DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM # (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED # INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE # OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF THE # AUTHOR HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. # modify these so that they correspond to your setup # backed up directory; note: without tailing / to preserve top directory # in backed up structure SOURCE_DIRECTORY=/home-or-other-directory-to-back-up # backup directory in target machine TARGET_DIRECTORY_ROOT=/share/backup-or-other-existing-share-in-qnap # name / IP address of the target machine TARGET_MACHINE=your-qnap-network-name-or-address # username for accessis the target machine via ssh TARGET_USER=administrator-or-equivalent # enforce root effective uid (may be needed if your backup includes # files not accessible by your usual id) ENFORCE_ROOT=false # OPTIONAL file containing patterns (one per line) that specify files / # dirs to be excluded from backup EXCLUDED_PATTERNS_FILE=/home/user/excluded-from-backup-or-similar.lst # --------------------------------------------------------------------- # the actual program code follows # force root execution if so specified if $ && [ `id -u` -ne 0 ]; then echo "Only root (effective id) is allowed to run this program." exit 1 fi PROGRAM_NAME=`basename $0` function usage () < echo $PROGRAM_NAME " usage: " $PROGRAM_NAME " [links|nolinks]"; echo -e "\tlinks: hard links to unchanged files (default); assumes previous backup" echo -e "\tnolinks: do not make hard links to previous backup" ># read options BACKUP_MODE_LINKS=0 BACKUP_MODE_NOLINKS=1 if [ $# -eq 0 -o "$1" = "links" ]; then BACKUP_MODE=$ elif [ "$1" = "nolinks" ]; then BACKUP_MODE=$ else usage exit 1 fi # build up directory and link names BACKUP_TIME=`date "+%Y-%m-%d-%H%M%S"` TARGET_DIRECTORY=$"/"$ TARGET_DIRECTORY_INCOMPLETE=$"-incomplete" LINK_NAME_CURRENT=$"/current" # set up rsync option for employing hard links if [ $ -eq $ ]; then LINKS_OPTION="--link-dest="$ else LINKS_OPTION="" fi # set up rsync option for excluding directories / files EXCLUDED_OPTIONS="" if [ -n "$" ]; then if [ -r $ -a -f $ ]; then EXCLUDED_OPTIONS="--exclude-from="$" --delete-excluded" fi fi # run rsync: note that -x avoids crossing file system boundaries; # then, in the target machine, rename complete backup and symlink to "current" rsync -axP --delete -e ssh $ $ $ $@$":"$ \ && ssh $@$ \ "mv $TARGET_DIRECTORY_INCOMPLETE $TARGET_DIRECTORY \ && rm -f $ \ && ln -s $ $" 

Источник