- Synchronization and backup programs
- Important considerations
- Data synchronization
- Legend
- Table
- Incremental backups
- Single machine
- Chunk-based increments
- File-based increments
- Network oriented
- Version control systems
- See also
- System backup
- Using Btrfs snapshots
- Using LVM snapshots
- Using rsync
- Using tar
- Using SquashFS
- Bootable backup
- Update the fstab
- Update the boot loader’s configuration file
- First boot
- Snapshots and /boot partition
Synchronization and backup programs
This page lists and compares applications that synchronize data between two or more locations, and those that build on top of such functionality to make incremental copies of important data for backup purposes. Because of their relationship, the two groups share several traits that justify describing them in the same article.
Important considerations
In order to choose the best program for one’s own needs, the following aspects should be considered:
- The type of backup medium that is going to store the data, e.g. CD, DVD, remote server, external hard drive, etc.
- The planned frequency of backups, e.g. daily, weekly, monthly, etc.
- The features expected from the backup solution, e.g. compression, encryption, handles renames, etc.
- The planned method to restore backups if needed.
Data synchronization
These applications simply keep directories synchronized between multiple locations/machines, in a «mirror» fashion. Nonetheless, most of them still allow storing and reverting to old revisions of modified or deleted files.
Legend
- unidirectional means one-way synchronization of two locations,
- bidirectional means two-way synchronization of two locations and
- multidirectional means full synchronization of more than two locations.
Table
This article or section needs language, wiki syntax or style improvements. See Help:Style for reference.
Reason: Explicitly list supported protocols / tools. Drop Encrypted transfer column because it depends on the protocol / tool. Simplify and color Change propagation column and move it leftwards. Split unmaintained into subsection? See also User:Larivact/drafts/Synchronization and backup programs#Synchronization. (Discuss in Talk:Synchronization and backup programs)
This article or section needs expansion.
Incremental backups
Applications that can do incremental backups remember and take into account what data has been backed up during the last run (so-called «diffs») and eliminate the need to have duplicates of unchanged data. Restoring the data to a certain point in time would require locating the last full backup and all the incremental backups from then to the moment when it is supposed to be restored. This sort of backup is useful for those who do it very often.
- Name: the application name, linking to the ArchWiki article or the official website.
- Package: a link to the package.
- Implementation: the programming language, library, or utility that the application is based on.
- Compressed storage: compression is used for storage.
- Encrypted storage: encryption is used for storage.
- Delta transfer: only the modified parts of files are transferred.
- Encrypted transfer: data is encrypted by default when transferred over a network.
- FS metadata: file system permissions and attributes are backed up.
- Easy access: the backup is stored plainly in the file system, or is mountable as such.
- Resumable: the backup can be resumed without restarting it if interrupted.
- Multithreading: the backup can be done in multiple threads of execution concurrently.
- Handles renames: moved/renamed files are detected and not stored or transferred twice; it typically means that a checksum is computed for files or chunks thereof.
- CLI: the application is command-line driven, i.e. it is scriptable.
- Other interfaces: the application has the specified user interfaces, e.g. GUI, TUI, or web-based.
- Licence: the licence of the server and client applications.
- Other platforms: supported operating systems other than Linux.
- Maintained: whether the project is maintained.
- Deduplication: whether the program supports deduplicating saved files
- Specificity: brief notes about special features that notably set the application apart from the others.
Single machine
These applications are aimed at backing up data from the machine they are installed on, although the backup destination can be located on an external machine or storage media.
Chunk-based increments
This article or section needs expansion.
Reason: Fill in the blanks. FS metadata column: What does Immature mean? Resumable column: What does Pausing only / Kinda through tar mean? (Discuss in Talk:Synchronization and backup programs)
If a file is modified, these applications store only its changed parts at the next snapshot. Compared to #File-based increments applications, these are more space-efficient, especially when large files receive small modifications; on the other hand, the archived snapshots have to be opened with the backup application that created them, since the files have to be reconstructed from the stored binary diffs.
Name | Package | Implementation | Compressed storage | Encrypted storage | Delta transfer | Encrypted transfer | FS metadata | Easy access | Resumable | Multithreading | Handles renames | CLI | Other interfaces | Licence | Other platforms | Maintained | Deduplication | Specificity |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Areca Backup [dead link 2023-05-20 ⓘ] | areca AUR | Java | Zip, Zip64 | AES128, AES256 | Yes | Yes | Yes | No | Pausing only | No [27] | No | Yes | Yes | GPLv2 | Windows | Yes | ||
Borg backup | borg | Python, C (Cython) | lz4, zlib, lzma, zstd | AES256 | Yes | SSH | Yes [28] | Yes [29] | Yes [30] | No [31] | Yes | Yes | third party | BSD | *BSD, macOS, Windows (Cygwin / WSL)[32] | Yes | Yes, based on variable length chunks. | Support both local and SSH-based remote backup destination. |
bup | bup | C, Python, git | Yes | No | Yes | Yes | Immature | Yes [33] | pick up where you left off [34] | No | Yes | Yes | thesafe AUR | GPLv2 | NetBSD, Windows, macOS | Yes | Yes | Same storage format as git. |
Duplicacy | duplicacy AUR | Go | Yes | Yes | Yes | Yes | Yes | Yes [35] | Yes [36] | Yes [37] | Yes [38] | Yes | duplicacy-web AUR | Custom (non-free) | FreeBSD, macOS, Windows | Yes | Yes | |
Duplicati | duplicati-canary-bin AUR | C# | Yes | Yes | Yes | Yes | Yes | No | Pausing only | Yes [39] | No | Yes | Yes | LGPL | Windows, macOS | Yes | Yes | |
Duplicity | duplicity | librsync | gzip | gpg | Yes | Yes | ? | No | Yes | No | No | Yes | Yes | GPL | Yes | |||
Kopia | kopia AUR | Go, Javascript front-end | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | APACHE | Windows, macOS, OpenBSD | Yes | Yes | |
Kup Backup System | kup | rsync, bup front-end | Yes | Yes | Yes | Yes | Immature | Yes | No | ? | Yes | bup | Qt | GPLv2 | Yes | |||
restic | restic | Go | Yes | AES-256 [40] | Yes | Yes | Yes [41] | Yes [42] | Yes [43] | Yes [44] | Yes | Yes | No [45] | BSD | OpenBSD, Windows, macOS | Yes | Yes [46] | Supports storage on various cloud services natively and through rclone . |
ZBackup | zbackup AUR | C++ | LZMA, LZO | AES | Yes | Yes | ? | planned [47] | No | ? | Kinda through tar | Yes | No | GPLv2 | No | Yes | Repository consists of immutable files. |
File-based increments
This article or section needs expansion.
If a file is modified, these applications store its new version entirely at the next snapshot. Compared to #Chunk-based increments applications, these are less space-efficient, especially when large files receive small modifications; on the other hand, often the archived snapshots can be opened without the need to have the backup application installed.
Network oriented
This article or section needs expansion.
Reason: Fill in the blanks. Encrypted transfer column: What does Internet transfers only mean? (Discuss in Talk:Synchronization and backup programs)
These applications have been designed to centralize the backup of several machines connected to a network, through a server-client model. In general they are more complicated to deploy, compared to #Single machine solutions.
Version control systems
While version control systems are mostly used for source code, they can track any files in a directory.
See also
System backup
A system backup is the process of backing up the operating system, files and system-specific useful/essential data. [It] primarily ensures that not only the user data in a system is saved, but also the system’s state or operational condition. This helps in restoring the system to the last-saved state along with all the selected backup data. [1]
Using Btrfs snapshots
Using LVM snapshots
Using rsync
Using tar
Using SquashFS
Bootable backup
Having a bootable backup can be useful in case the filesystem becomes corrupt or if an update breaks the system. The backup can also be used as a test bed for updates, with the testing repo enabled, etc. If you transferred the system to a different partition or drive and you want to boot it, the process is as simple as updating the backup’s /etc/fstab and your boot loader’s configuration file.
This section assumes that you backed up the system to another drive or partition, that your current boot loader is working fine, and that you want to boot from the backup as well.
Update the fstab
Without rebooting, edit the backup’s fstab by commenting out or removing any existing entries. Add one entry for the partition containing the backup like the example here:
/dev/sdaX / ext4 defaults 0 1
Remember to use the proper device name and filesystem type.
Update the boot loader’s configuration file
For Syslinux, all you need to do is duplicate the current entry, except pointing to a different drive or partition.
Tip: Instead of editing syslinux.cfg , you can also temporarily edit the menu during boot. When the menu shows up, press the Tab key and change the relevant entries. Partitions are counted from one, drives are counted from zero.
For GRUB, it is recommended that you automatically re-generate the main configuration file. If you want to freshly install all GRUB files to somewhere other than /boot , such as /mnt/newroot/boot , use the —boot-directory flag.
Also verify the new menu entry in /boot/grub/grub.cfg . Make sure the UUID is matching the new partition, otherwise it could still boot the old system. Find the UUID of a partition with lsblk:
$ lsblk -no NAME,UUID /dev/sdXY
where /dev/sdXY is the desired partition (e.g. /dev/sdb3 ). To list the UUIDs of partitions GRUB thinks it can boot, use grep:
# grep UUID= /boot/grub/grub.cfg
First boot
Reboot the computer and select the right entry in the boot loader. This will load the system for the first time. All peripherals should be detected and the empty folders in / will be populated.
Now you can re-edit /etc/fstab to add the previously removed partitions and mount points.
Snapshots and /boot partition
If your file system supports snapshots (e.g., LVM or Btrfs), these will most likely exclude the /boot partition or ESP.
You can copy the boot partition automatically on a kernel update to your root partition with a pacman hook (make sure the hook file is owned by root):
/etc/pacman.d/hooks/95-bootbackup.hook
[Trigger] Operation = Upgrade Operation = Install Operation = Remove Type = Path Target = usr/lib/modules/*/vmlinuz [Action] Depends = rsync Description = Backing up /boot. When = PostTransaction Exec = /usr/bin/rsync -a --delete /boot /.bootbackup