How to create tar archive split into, or spanning, multiple files?
The problem is that this command will require you to interactively give a new filename for the next file, after the first file is filled. Anybody knows of a way to skip this interactive step, and let tar do the «splitting» automatically?
7 Answers 7
Take a look at the —new-volume-script option, which lets you replace the prompting mechanism with a different mechanism or with a generated filename. ( (tar.info)Multi-Volume Archives in the tar info page.) The problem with split is that you need to cat the pieces back together to do anything, whereas a multivolume archive should be a bit more flexible.
Thanks, this is what I was looking for! I now found out that there is actually some instructions (incl. example) available here: gnu.org/software/tar/manual/tar.html#Using-Multiple-Tapes
The problem with this is that it’s overly involved nonsense and promotes the opposite of proper Unix style apps.
You can use split for this:
tar czpvf - /path/to/archive | split -d -b 100M - tardisk This tells tar to send the data to stdout, and split to pick it from stdin — additionally using a numeric suffix ( -d ), a chunk size ( -b ) of 100M and using ‘tardisk’ as the base for the resulting filenames (tardisk00, tardisk01, tardisk02, etc.).
To extract the data afterwards you can use this:
Of course the best option to use is the —new-volume-script option.
But, if you know the size of the file (in this case, largefile.tgz), then you can do this also:
tar -c -M -L 102400 --file=disk1.tar --file=disk2.tar --file=disk3.tar largefile.tgz -c = Create -M = multi-volume -L 102400 = 100MB files (disk1.tar, disk2.tar, disk3.tar . ) (For the -L, specify as many as needed so that the total sum of the tar files is larger than largefile.tgz)
If you are trying to tar a directory tree structure
it will automatically create files of size 1.1GB, if your tar is bigger in size, you can increase the number, for an example 1000 or you can increase the input to tape-length argument.
tar --tape-length=1048576 -cMv --file=tar_archive.> backup.tar.lzma I’ve answered here with the better explanation here tar split archive
This command is creating 2GB chunks without the compression:
tar -cv --tape-length=2097000 --file=my_archive-.tar file1 file2 dir3 The easiest non-interactive way would be —
Creating multipart tar from a large file (say bigfile)
$ tar cvf small.tar -L1024 -F 'echo small.tar-$ >&$' bigfile Extracting multipart tar to a specific directory (say /output/dir)
$ tar xvf small.tar -F 'echo small.tar-$ >&$' -C /output/dir You are free to select any supported compression format, block size (1024K here) and archive name (which is small.tar here).
I got it to work with the following commands:
mkdir -p ./split rm -rf ./split/* tar -cML 102400 -F 'cp "$" \ ./split/part_$.tar' \ -f split/part_1.tar large_file.tar.gz The only problem is that part_1.tar will actually be the last file, and the others are shifted by one. I.e. part_2.tar is actually the first part, and part_k.tar is the (n — 1) th part. Fixing this with some shell script is trivial, and left as an exercise for the reader.
How to Split Large ‘tar’ Archive into Multiple Files of Certain Size
Are you worried of transferring or uploading large files over a network, then worry no more, because you can move your files in bits to deal with slow network speeds by splitting them into blocks of a given size.
In this how-to guide, we shall briefly explore the creation of archive files and splitting them into blocks of a selected size. We shall use tar , one of the most popular archiving utilities on Linux and also take advantage of the split utility to help us break our archive files into small bits.
Before we move further, let us take note of, how these utilities can be used, the general syntax of a tar and split command is as follows:
# tar options archive-name files # split options file "prefix”
Let us now delve into a few examples to illustrate the main concept of this article.
Example 1: We can first of all create an archive file as follows:
$ tar -cvjf home.tar.bz2 /home/aaronkilik/Documents/*
To confirm that out archive file has been created and also check its size, we can use ls command:
Then using the split utility, we can break the home.tar.bz2 archive file into small blocks each of size 10MB as follows:
$ split -b 10M home.tar.bz2 "home.tar.bz2.part" $ ls -lh home.tar.bz2.parta*
As you can see from the output of the commands above, the tar archive file has been split to four parts.
Note: In the split command above, the option -b is used to specify the size of each block and the «home.tar.bz2.part» is the prefix in the name of each block file created after splitting.
Example 2: Similar to the case above, here, we can create an archive file of a Linux Mint ISO image file.
$ tar -cvzf linux-mint-18.tar.gz linuxmint-18-cinnamon-64bit.iso
Then follow the same steps in example 1 above to split the archive file into small bits of size 200MB .
$ ls -lh linux-mint-18.tar.gz $ split -b 200M linux-mint-18.tar.gz "ISO-archive.part" $ ls -lh ISO-archive.parta*
Example 3: In this instance, we can use a pipe to connect the output of the tar command to split as follows:
$ tar -cvzf - wget/* | split -b 150M - "downloads-part"
In this last example, we do not have to specify an archive name as you have noticed, simply use a — sign.
How to Join Tar Files After Splitting
After successfully splitting tar files or any large file in Linux, you can join the files using the cat command. Employing cat is the most efficient and reliable method of performing a joining operation.
To join back all the blocks or tar files, we issue the command below:
# cat home.tar.bz2.parta* >backup.tar.gz.joined
We can see that after running the cat command, it combines all the small blocks we had earlier on created to the original tar archive file of the same size.
Conclusion
The whole idea is simple, as we have illustrated above, you simply need to know and understand how to use the various options of tar and split utilities.
You can refer to their manual entry pages of to learn more other options and perform some complex operations or you can go through the following article to learn more about tar command.
For any questions or further tips, you can share your thoughts via the comment section below.
Split files using tar, gz, zip, or bzip2 [closed]
I need to compress a large file of about 17-20 GB. I need to split it into several files of around 1GB per file. I searched for a solution via Google and found ways using split and cat commands. But they did not work for large files at all. Also, they won’t work in Windows; I need to extract it on a Windows machine.
Many compression programs (e.g. like 7-Zip) is able to split the compressed file into volumes of a specified size for easier distribution.
If one of the two viable solutions posted here doesn’t pan out, he’ll be needing a programming solution.
4 Answers 4
You can use the split command with the -b option:
It can be reassembled on a Windows machine using @Joshua’s answer.
copy /b file1 + file2 + file3 + file4 filetogether Edit: As @Charlie stated in the comment below, you might want to set a prefix explicitly because it will use x otherwise, which can be confusing.
split -b 1024m "file.tar.gz" "file.tar.gz.part-" // Creates files: file.tar.gz.part-aa, file.tar.gz.part-ab, file.tar.gz.part-ac, . Edit: Editing the post because question is closed and the most effective solution is very close to the content of this answer:
# create archives $ tar cz my_large_file_1 my_large_file_2 | split -b 1024MiB - myfiles_split.tgz_ # uncompress $ cat myfiles_split.tgz_* | tar xz This solution avoids the need to use an intermediate large file when (de)compressing. Use the tar -C option to use a different directory for the resulting files. btw if the archive consists from only a single file, tar could be avoided and only gzip used:
# create archives $ gzip -c my_large_file | split -b 1024MiB - myfile_split.gz_ # uncompress $ cat myfile_split.gz_* | gunzip -c > my_large_file For windows you can download ported versions of the same commands or use cygwin.