Test tar file integrity in bash
I have a bash script that creates a ‘.tar’ file. Once the file is created, I would like to test its integrity and send an email to the root user if the integrity is bad. I know I would need to use the command tar -tf /root/archive.tar to check the integrity of the file, but how would I implement this in a bash if statement and check for errors?
3 Answers 3
If tar finds errors in its input it will exit(3) ¹ with a non-zero exit value. This — with most tar implementations — is also done when listing archive contents with t . So you could simply check for the exit value of tar to determine if something has gone wrong:
if ! tar tf /root/archive.tar &> /dev/null; then write_an_email_to_root fi If your tar does not find all errors with t , you could still extract the archive to stdout and redirect stdout to /dev/null , which would be the slower but more reliable approach:
if ! tar xOf /root/archive.tar &> /dev/null; then write_an_email_to_root fi ¹ This notation denotes the manpage, not the actual call. See man 3 exit .
You’ll probably want to redirect the tar output to /dev/null , as you probably don’t actually want to see it.
Not all tar implementations detect or report all errors with t ( bsdtar doesn’t, you can use tar xOf file.tar > /dev/null there). Not all tar implementations would exit with exit status 2, star (255) ot bsdtar (1) don’t, but what matters is that the exit status is non-zero here.
I edited my post accordingly. With exit(2) I was regarding to the appropriate manpage, not the exact exit value (which is why I had »non-zero« explicitly mentioned in the following sentence). Annotated this (and changed the section to the right one -.-).
your answer is only about capability of listing and extraction, if I open tar archive with an editor and change something your solution won’t catch any kind of error . sorry
I agree with @THESorcerer because checking tar integrity meaning you need to compare the current tar file with the previous checksum. if it doesn’t match, it’s considered broken. But the answer is to validate whether the tar file is corrupt or not (normally we test it by listing and extracting at the background). So where is the integrity test ?
sorry, but you cannot, seems like tar lacks the capability of testing (for example if you forget about *nix and try to test it with winrar, result will be: «The command is not supported for this type of archive» )
how I pointed out in comments above, tar literally lacks internal CRC to have a term of comparison, therefore if you change a tar archive with an editor, listing and extraction may work flawless with no error, but extract corrupted data
in conclusion, I end up here hoping for a solution, but there is not one, fortunately there are two good news:
there are very, very, VERY rare tar archives that are not also compressed with another program (like gzip, bzip2, etc.) therefore that program will have a testing solution and people who don’t compress them are kinda lame, REALLY LAME
for my personal problem, I’m lucky, after extracting it, I just found a md5sum file with CRC of all files inside (even the person who put arhive tar on net and didn’t want to compress it, he still wanted to be able to be verified)
How to check if a Unix .tar.gz file is a valid file without uncompressing?
I have found the question How to determine if data is valid tar file without a file?, but I was wondering: is there a ready made command line solution?
8 Answers 8
What about just getting a listing of the tarball and throw away the output, rather than decompressing the file?
tar -tzf my_tar.tar.gz >/dev/null Edited as per comment. Thanks zrajm!
Edit as per comment. Thanks Frozen Flame! This test in no way implies integrity of the data. Because it was designed as a tape archival utility most implementations of tar will allow multiple copies of the same file!
@asmeurer Re: -z That’s definitely the case with GNU tar — do you know if this is this true elsewhere (BSD, etc.)?
@belacqua For others’ information, as in the manpage of BSD TAR, -z : «. In extract or list modes, this option is ignored.»
@bobwells But does successful uncompressing or content files listing imply data integrity of the tar.gz ? Any support information?
you could probably use the gzip -t option to test the files integrity
To test the gzip file is not corrupt:
To test the tar file inside is not corrupt:
gunzip -c file.tar.gz | tar -t > /dev/null As part of the backup you could probably just run the latter command and check the value of $? afterwards for a 0 (success) value. If either the tar or the gzip has an issue, $? will have a non zero value.
@IntrastellarExplorer that is a typo on my part, although it should still work without the hyphen. I’m just used to the old style options on tar. unix.stackexchange.com/questions/394060/…
At first glance I feel this is the most correct answer here, as at least the integrity of the zipped ‘file,tar’ is checked against the checksum contained in .gz file. However for individual files tared in ‘files.tar’ relying on tar -t is riskier because random byte errors may very likely not be detected. Anyhow I suppose there is nothing better than ‘tar -t» you can get here.
If you want to do a real test extract of a tar file without extracting to disk, use the -O option. This spews the extract to standard output instead of the filesystem. If the tar file is corrupt, the process will abort with an error.
Example of failed tar ball test.
$ echo "this will not pass the test" > hello.tgz $ tar -xvzf hello.tgz -O > /dev/null gzip: stdin: not in gzip format tar: Child returned status 1 tar: Error exit delayed from previous errors $ rm hello.* $ ls hello* ls: hello*: No such file or directory $ echo "hello1" > hello1.txt $ echo "hello2" > hello2.txt $ tar -cvzf hello.tgz hello[12].txt hello1.txt hello2.txt $ rm hello[12].txt $ ls hello* hello.tgz $ tar -xvzf hello.tgz -O hello1.txt hello1 hello2.txt hello2 $ ls hello* hello.tgz $ tar -xvzf hello.tgz hello1.txt hello2.txt $ ls hello* hello1.txt hello2.txt hello.tgz $ rm hello* It seems to me that the best test is this one. It truly extracts each file and makes sure there are no errors.
Really useful. I’ve made a shell script, add an argument hook to pass the path of the file and put it in my path 🙂 [ tar -xvzf $1 -O > /dev/null ]
@sleeves Why do you think that it is better than the accepted answer? tar -tvzf hello.tgz > /dev/null also gives the same error.
@dash17291 I say this because I expect it to be a tough problem to prove that for all cases, a -tvf will catch all errors or corruptions that a -xvf. In other words, -xvf will catch all that -tvf, but I cannot say the converse is true.
You can also check contents of *.tag.gz file using pigz (parallel gzip) to speedup the archive check:
pigz -cvdp number_of_threads /[. ]path[. ]/archive_name.tar.gz | tar -tv > /dev/null Mini benchmark: running it with pigz -cvd: 80s, while running with accepted answer tar -tzv: 143s on a 22G archive.
how do u remove the notification from cronjob about «tar: Removing leading `/’ from member names» when using this method ?
A nice option is to use tar -tvvf which adds a line that reports the kind of file.
Example in a valid .tar file:
> tar -tvvf filename.tar drwxr-xr-x 0 diegoreymendez staff 0 Jul 31 12:46 ./testfolder2/ -rw-r--r-- 0 diegoreymendez staff 82 Jul 31 12:46 ./testfolder2/._.DS_Store -rw-r--r-- 0 diegoreymendez staff 6148 Jul 31 12:46 ./testfolder2/.DS_Store drwxr-xr-x 0 diegoreymendez staff 0 Jul 31 12:42 ./testfolder2/testfolder/ -rw-r--r-- 0 diegoreymendez staff 82 Jul 31 12:42 ./testfolder2/testfolder/._.DS_Store -rw-r--r-- 0 diegoreymendez staff 6148 Jul 31 12:42 ./testfolder2/testfolder/.DS_Store -rw-r--r-- 0 diegoreymendez staff 325377 Jul 5 09:50 ./testfolder2/testfolder/Scala.pages Archive Format: POSIX ustar format, Compression: none > tar -tvvf corrupted.tar tar: Unrecognized archive format Archive Format: (null), Compression: none tar: Error exit delayed from previous errors.