Hashing files in linux

Computing file hash values on Linux and Windows

Just some quick notes on computing hash values (aka checksums) for one or more files on Linux and Windows.

Linux/Bash

Compute a hash value for a single file FILE with algorithm SHA256 or MD5:

$ sha256sum FILE $ md5sum FILE

To compute hash values for all files in a directory DIR (including subdirectories) you can use the good old find command. Either pipe its output to your preferred hash function via xargs or use the -execdir option of find :

$ find DIR -type f -print0 | xargs -0 md5sum $ find DIR -type f -execdir md5sum <> +

Compare all files in two different directories (e.g., after copying from one to the other) based on their names (without leading directory paths) and hash values:

Windows/PowerShell

Compute a hash value for a single file FILE with algorithm SHA256 (default) or MD5:

PS> Get-FileHash FILE PS> Get-FileHash -Algorithm MD5 FILE PS> Get-FileHash -Algorithm SHA256 FILE

To compute hash values for all files in a directory DIR (including subdirectories) you can use the Get-ChildItem commandlet. Pipe its output to the Get-FileHash commandlet:

PS> Get-ChildItem DIR -Recurse -File | Get-FileHash

Compare all files in two different directories (e.g., after copying from one to the other) based on their hash values:

PS> Compare-Object ` $(Get-ChildItem DIR1 -Recurse -File | Get-FileHash) ` $(Get-ChildItem DIR2 -Recurse -File | Get-FileHash) ` -Property Hash

Note that this will compare just hash values, nothing more, not even file names. However, if you just want to check if your copy action went smoothly, that’s usually enough.

The hashdeep tool

If you frequently compare sets of files recursively based on their hash values then the hashdeep tool may be just what you need. It is available for Unix and Windows. With hashdeep, comparing all files in two different directories based on their hash values gets as easy as this:

If you care for file paths you need a less concise version involving a temporary file (note the -l option instead of -b to use relative file paths instead of bare file names):

$ cd DIR1 $ hashdeep -lr . > /tmp/hashes $ cd DIR2 $ hashdeep -lr . -ak /tmp/hashes

Источник

An introduction to hashing and checksums in Linux

Always wondered how to make use of a checksum? This introduction shows you what they mean, and how to use the proper tools to verify the integrity of a file.

The chances are that you’ve seen references to hashes or checksums when you’ve downloaded software from the Internet. Often, the software will be displayed, and then near the link is a checksum. The checksum may be labeled as MD5, SHA, or with some other similar name. Here is an example using one of my favorite old games from the 1990s named Nethack:

Many people don’t know exactly what this information means or how to work with it. In this article, I discuss the purpose of hashing, along with how to use it.

Goals of cryptography

In this first section, I want you to unlearn something. Specifically, I want you to break the association in your head between the word encryption and the word confidential. Many of us conceive of these two words as being synonymous when that is actually not the case. Cryptography, which includes encryption, can provide confidentiality, but it can also satisfy other goals.

Cryptography actually has three goals:

• Confidentiality — to keep the file content from being read by unauthorized users
• Authenticity — to prove where a file originated
• Integrity — to prove that a file has not changed unexpectedly

Linux security

It is that third concept, integrity, that we are interested in here. In this context, integrity means to prove that data has not changed unexpectedly. Proving integrity is useful in many scenarios:

• Internet downloads such as Linux distributions, software, or data files
• Network file transfers via NFS, SSH, or other protocols
• Verifying software installations
• Comparing a stored value, such as a password, with a value entered by a user
• Backups that compare two files to see whether they’ve changed

What is hashing?

Cryptography uses hashing to confirm that a file is unchanged. The simple explanation is that the same hashing method is used on a file at each end of an Internet download. The file is hashed on the web server by the web administrator, and the hash result is published. A user downloads the file and applies the same hash method. The hash results, or checksums, are compared. If the checksum of the downloaded file is the same as that of the original file, then the two files are identical, and there have been no unexpected changes due to file corruption, man-in-the-middle attacks, etc.

Hashing is a one-way process. The hashed result cannot be reversed to expose the original data. The checksum is a string of output that is a set size. Technically, that means that hashing is not encryption because encryption is intended to be reversed (decrypted).

What kind of hash cryptography might you use with Linux?

Message Digest and Secure Hash Algorithm

In Linux, you’re likely to interact with one of two hashing methods:

These cryptography tools are built into most Linux distributions, as well as macOS. Windows does not typically include these utilities, so you must download them separately from third party vendors if you wish to use this security technique. I think it’s great that security tools such as these are part of Linux and macOS.

Message Digest versus Secure Hash Algorithm

What’s the difference between the message digest and secure hash algorithms? The difference is in the mathematics involved, but the two accomplish similar goals. Sysadmins might prefer one over the other, but for most purposes, they function similarly. They are not, however, interchangeable. A hash generated with MD5 on one end of the connection will not be useful if SHA256 is used on the other end. The same hash method must be used on both sides.

SHA256 generates a bigger hash, and may take more time and computing power to complete. It is considered to be a more secure approach. MD5 is probably good enough for most basic integrity checks, such as file downloads.

Where do you find hashing in Linux?

Linux uses hashes in many places and situations. Checksums can be generated manually by the user. You’ll see exactly how to do that later in the article. In addition, hash capabilities are included with /etc/shadow , rsync , and other utilities.

For example, the passwords stored in the /etc/shadow file are actually hashes. When you sign in to a Linux system, the authentication process compares the stored hash value against a hashed version of the password you typed in. If the two checksums are identical, then the original password and what you typed in are identical. In other words, you entered the correct password. This is determined, however, without ever actually decrypting the stored password on your system. Check the first two characters of the second field for your user account in /etc/shadow . If the two characters are $1, your password is encrypted with MD5. If the characters are $5, your password is encrypted with SHA256. If the value is $6, SHA512 is being used. SHA512 is used on my Fedora 33 virtual machine, as seen below:

How to manually generate checksums

Using the hash utilities is very simple. I will walk you through a very easy scenario to accomplish on a lab computer or whatever Linux system you have available. The purpose of this scenario is to determine whether a file has changed.

First, open your favorite text editor and create a file named original.txt with a line of text that reads: Original information.

[damon@localhost ~]$ vim original.txt [damon@localhost ~]$ cat original.txt Original information. [damon@localhost ~]$

Next, run the file through a hash algorithm. I’ll use MD5 for now. The command is md5sum . Here is an example:

[damon@localhost ~]$ md5sum original.txt 80bffb4ca7cc62662d951326714a71be original.txt [damon@localhost ~]$

Kubernetes and OpenShift

Notice the resulting checksum value. This value is large enough that it’s difficult to work with. Let’s store that value for future use by redirecting it into a file:

[damon@localhost ~]$ md5sum original.txt > hashes.txt [damon@localhost ~]$ cat hashes.txt 80bffb4ca7cc62662d951326714a71be original.txt [damon@localhost ~]$

At this point, you have an original file. Copy that file to the /tmp directory with the name duplicate.txt . Copy the file by using the following command (be sure to copy, not move):

[damon@localhost ~]$ cp original.txt /tmp/duplicate.txt [damon@localhost ~]$

Run the following command to create a checksum of the copied file:

[damon@localhost ~]$ md5sum /tmp/duplicate.txt 80bffb4ca7cc62662d951326714a71be /tmp/duplicate.txt [damon@localhost ~]$

Next, append the hash result to our hashes.txt file and then compare the two. Be very careful to use the >> append redirect operator here, because > will overwrite the hash value of the original.txt file.

Run the following command:

[damon@localhost ~]$ md5sum /tmp/duplicate.txt >> hashes.txt [damon@localhost ~]$ cat hashes.txt 80bffb4ca7cc62662d951326714a71be original.txt 80bffb4ca7cc62662d951326714a71be /tmp/duplicate.txt [damon@localhost ~]$

The two hash results are identical, so the file did not change during the copy process.

Next, simulate a change. Type the following command to change the /tmp/duplicate.txt file contents, and then rerun the md5sum command with the >> append operator:

[damon@localhost ~]$ hostname >> /tmp/duplicate.txt [damon@localhost ~]$ md5sum /tmp/duplicate.txt >> hashes.txt [damon@localhost ~]$

You know that the duplicate.txt file is no longer identical to the original.txt file, but let’s prove that:

[damon@localhost ~]$ cat hashes.txt 80bffb4ca7cc62662d951326714a71be original.txt 80bffb4ca7cc62662d951326714a71be /tmp/duplicate.txt 1f59bbdc4e80240e0159f09ecfe3954d /tmp/duplicate.txt [damon@localhost ~]$

The two checksum values are not identical, and therefore the two files from which the checksums were generated are not identical.

In the above example, you manually compared the hash values by displaying them with cat . You can use the —check option to have md5sum do the comparison for us. I’ve included both methods below:

[damon@localhost ~]$ cat hashes.txt 80bffb4ca7cc62662d951326714a71be original.txt 80bffb4ca7cc62662d951326714a71be /tmp/duplicate.txt 1f59bbdc4e80240e0159f09ecfe3954d /tmp/duplicate.txt [damon@localhost ~]$ md5sum --check hashes.txt original.txt: OK /tmp/duplicate.txt: FAILED /tmp/duplicate.txt: OK md5sum: WARNING: 1 computed checksum did NOT match [damon@localhost ~]$

You can repeat the above steps substituting sha256sum for the md5sum command to see how the process works using the SHA algorithm. The sha256sum command also includes a —check checksum option that compares the resulting hashes and displays a message for whether the files differ.

Note: If you transfer files between Linux, macOS, and Windows, you can still use hashing to verify the files’ integrity. To generate a hash value on macOS, run the md5 command. To do this in Windows, you must download a third party program. Personally, I use md5checker. Be sure to understand licensing for these utilities. You may be able to use the PowerShell cmdlet get-filehash , depending on the version of PowerShell you have installed.

[ Free course: Red Hat Satellite Technical Overview. ]

Wrap up

Hashing confirms that data has not unexpectedly changed during a file transfer, download, or other event. This concept is known as file integrity. Hashing does not tell you what changed, just that something changed. Once hashing tells you two files are different, you can use commands such as diff to discover what differences exist.

Источник