Generate md5 checksum for all files in a directory
I would like to create a md5 checksum list for all files in a directory. I want to cat filename | md5sum > ouptput.txt . I want to do this in 1 step for all files in my directory. Any assistance would be great.
In case one wants to include subdirectories but do separate checksum files inside each subdir, this answer might be helpful: stackoverflow.com/a/71286224/111036
7 Answers 7
You can pass md5sum multiple filenames or bash expansions:
$ md5sum * > checklist.chk # generates a list of checksums for any file that matches * $ md5sum -c checklist.chk # runs through the list to check them cron: OK database.sqlite3: OK fabfile.py: OK fabfile.pyc: OK manage.py: OK nginx.conf: OK uwsgi.ini: OK If you want to get fancy you can use things like find to drill down and filter the files, as well as working recursively:
find -type f -exec md5sum "<>" + > checklist.chk ♦ how to use the above for getting md5sum of the files inside sub directories, the above md5sum * emitting going into subdirectory level by saying . is a directory
Sorry for asking by running find -type f -exec md5sum ‘<>‘ + and `find -type f -exec md5sum ‘<>‘ ` I was able to get it. Thanks 🙂
If you’re using a shell that’s happy to evaluate ** recursively (such as zsh), it’s even simpler: md5sum **/* 2>/dev/null
if the process will take a while, you can run it through pv to track progress. use it in line mode w/ pv -l : find . -type f -exec md5sum ‘<>‘ \; | pv -l -s $(find . -type f | wc -l) > ~/md5sum.txt . just make sure both of your find filters match: the main one and the one in the pv subshell
A great checksum creation/verification program is rhash .
- It can create SFV compatible files, and check them too.
- It supports md4, md5, sha1, sha512, crc32 and many many other.
- It can do recursive creation ( -r option) like md5deep or sha1deep .
- Last but not least, you can format the output of the checksum file. For example:
rhash --md5 -p '%h,%p\n' -r /home/ > checklist.csv I also find the -e option to rename files by inserting crc32 sum into the name extremely useful.
Note that you can also change md5sum with rhash in the PhoenixNL72 examples.
I think it’s an error, it certainly errors for me. The -p is the format for the output. I’ll correct it.
Here are two more extensive examples:
- Create an md5 file in each directory which doesn’t already have one, with absolute paths:
find "$PWD" -type d | sort | while read dir; do [ ! -f "$"/@md5Sum.md5 ] && echo "Processing " "$" || echo "Skipped " "$" " @md5Sum.md5 already present" ; [ ! -f "$"/@md5Sum.md5 ] && md5sum "$"/* > "$"/@md5Sum.md5 ; chmod a=r "$"/@md5Sum.md5;done find "$PWD" -type d | sort | while read dir; do cd "$"; [ ! -f @md5Sum.md5 ] && echo "Processing " "$" || echo "Skipped " "$" " @md5Sum.md5 allready present" ; [ ! -f @md5Sum.md5 ] && md5sum * > @md5Sum.md5 ; chmod a=r "$"/@md5Sum.md5 ;done What differs between 1 and 2 is the way the files are presented in the resulting md5 file.
The commands do the following:
- Build a list of directory names for the current folder. (Tree)
- Sort the folder list.
- Check in each directory if the file @md5sum.md5 exists. Output Skipped if it exists, output Processing if it doesn’t exist.
- If the @md5Sum.md5 file doesn’t exist, md5Sum will generate one with the checksums of all the files in the folder. 5) Set the generated @md5Sum.md5 file to read only.
The output of this entire script can be redirected to a file (. ;done > test.log) or piped to another program (like grep). The output will only tell you which directories where skipped and which have been processed.
After a successful run, you will end up with an @md5Sum.md5 file in each subdirectory of your current directory
I named the file @md5Sum.md5 so it’ll get listed at the top of the directory in a samba share.
Verifying all @md5Sum.md5 files can be done by the next commands:
find "$PWD" -name @md5Sum.md5 | sort | while read file; do cd "$"; md5sum -c @md5Sum.md5; done > checklog.txt Afterwards you can grep the checklog.txt using grep -v OK to get a list of all files that differ.
To regenerate an @md5Sum.md5 in a specific directory, when you changed or added files for instance, either delete the @md5Sum.md5 file or rename it and run the generate command again.
How to Get md5 Hash Of A File
Message-Digest algorithm, commonly known as md5 hash, is a type of cryptographic hash function mainly used to verify the integrity of files. Md5 is a 128-bit message digest produced after running the MD5 function against a file.
Md5 has its flaws and is therefore not a very good choice for certain encryption methods, but it is very well suited for file verification. It works by creating a checksum of a file and comparing the result to the original. That means if there are changes to a file, there is no way it can produce a digest value similar to the original. The value stays constant no matter where generated or how many times as long as the file remains unchanged.
For this guide, we shall look at ways to generate an md5 hash value of a file. That will allow you to verify the integrity of files either from remote locations or on your local machine.
Install md5sum
In Linux and almost major Unix and Unix-Like systems, they come pre-installed with an md5 tool. The most common one is md5sum. By default, you should find it available in your system.
If you do not have the tool installed, you can use the package manager of your system.
Debian/Ubuntu
On Ubuntu and other Debian based distributions, use apt as:
REHL/CentOS
On REHL and CentOS, use yum as:
Arch/Manjaro
If you are on Manjaro or other arch based distributions, use Pacman using the command:
Fedora
Finally, on Fedora systems, use the dnf command as:
Generate Md5sum of a File
With the tool installed, we can proceed and generate a md5sum for a file. You can use any basic file available in your system. In my example, I am using the /etc/hosts available in Linux systems.
To generate the md5sum of a file, simply use the md5sum command followed by the filename, which you can see in the command below:
The above command should generate a hash value of the file as shown in the output below:
Once the contents of the file change, the md5sum value becomes completely different. For example, add a value to the/etc/hosts file.
Add the following entry to the file (feel free to change to any way you see fit).
If you try to calculate the md5 value of the file with the new contents as:
The hash value is different as shown in the output below:
If you revert the file to its original contents, the md5sum value is similar to the original, allowing you to know when a file has changed.
NOTE: The md5 value will be similar to the original even if the file gets renamed. This is because md5 is calculated based on file contents and not filename.
Verify Online Files
Suppose you want to verify the integrity of a file and ensure it is tamper-proof. To do this, all you need is the original md5 value. In my example, I am using a simple deb package of MySQL from the resource below:
Download the file with wget using the command as:
Once the file has downloaded:
Let us now verify the md5 value using a command:
If the file has not been modified in any way, you should get a similar value as the original as shown:
Conclusion
This tutorial looked at a simple method to verify the md5 checksum of files and verify their modification state.
Here is a quick exercise for you.
Exercise
Create a simple bash script that checks if a file md5 value has any recorded modification every 5 minutes. If the file has changed, delete the file and shut down the system.
That should be a fun exercise!
About the author
John Otieno
My name is John and am a fellow geek like you. I am passionate about all things computers from Hardware, Operating systems to Programming. My dream is to share my knowledge with the world and help out fellow geeks. Follow my content by subscribing to LinuxHint mailing list
Learn How to Generate and Verify Files with MD5 Checksum in Linux
A checksum is a digit which serves as a sum of correct digits in data, which can be used later to detect errors in the data during storage or transmission. MD5 (Message Digest 5) sums can be used as a checksum to verify files or strings in a Linux file system.
MD5 Sums are 128-bit character strings (numerals and letters) resulting from running the MD5 algorithm against a specific file. The MD5 algorithm is a popular hash function that generates 128-bit message digest referred to as a hash value, and when you generate one for a particular file, it is precisely unchanged on any machine no matter the number of times it is generated.
It is normally very difficult to find two distinct files that results in same strings. Therefore, you can use md5sum to check digital data integrity by determining that a file or ISO you downloaded is a bit-for-bit copy of the remote file or ISO.
In Linux, the md5sum program computes and checks MD5 hash values of a file. It is a constituent of GNU Core Utilities package, therefore comes pre-installed on most, if not all Linux distributions.
Take a look at the contents of /etc/group saved as groups.cvs below.
root:x:0: daemon:x:1: bin:x:2: sys:x:3: adm:x:4:syslog,aaronkilik tty:x:5: disk:x:6: lp:x:7: mail:x:8: news:x:9: uucp:x:10: man:x:12: proxy:x:13: kmem:x:15: dialout:x:20: fax:x:21: voice:x:22: cdrom:x:24:aaronkilik floppy:x:25: tape:x:26: sudo:x:27:aaronkilik audio:x:29:pulse dip:x:30:aaronkilik
The md5sums command below will generate a hash value for the file as follows:
$ md5sum groups.csv bc527343c7ffc103111f3a694b004e2f groups.csv
When you attempt to alter the contents of the file by removing the first line, root:x:0: and then run the command for a second time, try to observe the hash value:
$ md5sum groups.csv 46798b5cfca45c46a84b7419f8b74735 groups.csv
You will notice that the hash value has now changed, indicating that the contents of the file where altered.
Now, put back the first line of the file, root:x:0: and rename it to group_file.txt and run the command below to generate its hash value again:
$ md5sum groups_list.txt bc527343c7ffc103111f3a694b004e2f groups_list.txt
From the output above, the hash value is still the same even when the file has been renamed, with its original content.
Important: md5 sums only verifies/works with the file content rather than the file name.
The file groups_list.txt is a duplicate of groups.csv, so, try to generate the hash value of the files at the same time as follows.
You will see that they both have equal hash values, this is because they have the exact same content.
$ md5sum groups_list.txt groups.csv bc527343c7ffc103111f3a694b004e2f groups_list.txt bc527343c7ffc103111f3a694b004e2f groups.csv
You can redirect the hash value(s) of a file(s) into a text file and store, share them with others. For the two files above, you can issues the command below to redirect generated hash values into a text file for later use:
$ md5sum groups_list.txt groups.csv > myfiles.md5
To check that the files have not been modified since you created the checksum, run the next command. You should be able to view the name of each file along with “OK”.
The -c or —check option tells md5sums command to read MD5 sums from the files and check them.
$ md5sum -c myfiles.md5 groups_list.txt: OK groups.csv: OK
Remember that after creating the checksum, you can not rename the files or else you get a “No such file or directory” error, when you try to verify the files with new names.
$ mv groups_list.txt new.txt $ mv groups.csv file.txt $ md5sum -c myfiles.md5
md5sum: groups_list.txt: No such file or directory groups_list.txt: FAILED open or read md5sum: groups.csv: No such file or directory groups.csv: FAILED open or read md5sum: WARNING: 2 listed files could not be read
The concept also works for strings alike, in the commands below, -n means do not output the trailing newline:
$ echo -n "Tecmint How-Tos" | md5sum - afc7cb02baab440a6e64de1a5b0d0f1b -
$ echo -n "Tecmint How-To" | md5sum - 65136cb527bff5ed8615bd1959b0a248 -
In this guide, I showed you how to generate hash values for files, create a checksum for later verification of file integrity in Linux. Although security vulnerabilities in the MD5 algorithm have been detected, MD5 hashes still remains useful especially if you trust the party that creates them.
Verifying files is therefore an important aspect of file handling on your systems to avoid downloading, storing or sharing corrupted files. Last but not least, as usual reach us by means of the comment form below to seek any assistance, you can as well make some important suggestions to improve this post.