How to encode and decode data in base64 and base64URL by using unix commands?
Recent versions of coreutils include basenc(1) which supports several different encodings. From its help screen:
--base64 same as 'base64' program (RFC4648 section 4) --base64url file- and url-safe base64 (RFC4648 section 5) --base32 same as 'base32' program (RFC4648 section 6) --base32hex extended hex alphabet base32 (RFC4648 section 7) --base16 hex encoding (RFC4648 section 8) --base2msbf bit string with most significant bit (msb) first --base2lsbf bit string with least significant bit (lsb) first --z85 ascii85-like encoding (ZeroMQ spec:32/Z85); when encoding, input length must be a multiple of 4; when decoding, input length must be a multiple of 5 Here is a string that illustrates the difference:
$ printf "%s" "$s" | xxd -b -c1 | cut -d' ' -f2 | nl 1 01111000 2 01110011 3 00111111 4 00111110 5 00111110 6 00111110 And as 6 bit blocks (as base64 reads the data):
$ printf "%s" "$s" | xxd -b -c1 | cut -d' ' -f2 | tr -d '\n' | fold -w6 | nl 1 011110 2 000111 3 001100 4 111111 5 001111 6 100011 7 111000 8 111110 Note that block 4 and block 8 map to / and + respectively (Base64 table on Wikipedia):
This is the same suggestion as @jps but shorter. Also remember that echo by default always adds newline at the end, so when you want to encode it, you must add -n .
echo -n "Some_data_to_be_converted" | base64 | tr '/+' '_-' | tr -d '=' Decoding it back with bulit-in bash tools is more complicated as I didn’t find an easy way to pad the string back with ‘=’ so that the length will dividable by 4. Probably can be done with awk but I didn’t dig deep enough. If you have local ruby it becomes trivial:
2.6.2 > require 'base64' 2.6.2 > Base64.urlsafe_encode64('test', padding: false) => "dGVzdA" 2.6.2 > Base64.urlsafe_decode64('dGVzdA') => "test" I took it as a challenge to come up with a bash or standard unix commands to pad the string back in order to decode: stackoverflow.com/questions/58957358/…
Adding on to the answer by Kaplan Ilya, here is a command using standard linux/unix commands that can decode base64url , including handling missing padding.
Note: some versions of base64 can handle missing padding, such as Mac/BSD base64 -D . However, GNU base64 -d requires correct padding.
Also, I used the test string ~~~. instead of the one in the original question Some_data_to_be_converted , so that it will generate + , / , = characters.
text='~~~. ' # encode base64 echo "$text" | base64 # fn5+Pz8/Cg== # encode base64url base64url=$( echo "$text" | base64 | tr '/+' '_-' | tr -d '=' ) echo "$base64url" # fn5-Pz8_Cg # decode base64url echo "$base64url"==== | fold -w 4 | sed '$ d' | tr -d '\n' | tr '_-' '/+' | base64 -d # ~~~. Explanation of the decode base64url commands:
- echo «$str»==== appends 4 equal signs
- fold -w 4 split every 4 characters into separate lines
- sed ‘$ d’ deletes the last line (the extraneous padding)
- tr -d ‘\n’ joins all lines. Now the padding is correct.
- tr ‘_-‘ ‘/+’ converts _ to / , — to + .
(Side note: if you’re wondering why not use tr ‘-_’ ‘+/’ since that would be in alphanumeric order, it’s because that will result in invalid option because it thinks -_ is an option. You could do tr — ‘-_’ ‘+/’ , but it’s easier just to swap the order.)
If you already have a base64 encoded string you just need to replace «+» with «-» and «/» with » _» to get a base64url encoded string. To achieve this, you can use the following command:
echo Some_data_to_be_converted | base64 | sed ‘s/+/-/g; s,/,_,g’
Base64 encoding maps the input bytes (8 bit) to a 6 bit representation. 4 base64 characters can encode 4*6=24 bits, which equals 3 bytes. Whenever the number of bytes in your input can’t be divided by 3, padding is required according to the standard.
As the = character is used for key-value pairs in URLs, you can’t use it directly for padding if you intend to use the encoded value in an URL. You can either just ommit the padding, because most implementations will still work and just ignore the 2 or 4 unused bits on the end. Or, if the receiver really needs padding, you have to replace the = by it’s URL safe representation %3d .
Base64 Encode and Decode From Command Line
Encoding is the process used to convert data in a format required for effective transmission or storage. In contrast, decoding is opposite to the encoding method which converts the encoded data back to its original format. Base64 is the encoding process where the binary data is converted into ASCII. Base64 encoding is mostly required to avoid the transmission problems that occur when binary data is transmitted to text-based systems which cannot handle the binary data properly. As a result, the information is lost or corrupted during transmission.
Some of the uses of encoding are:
For encoding data, Base64 uses only alphabet, number and = symbol. For instance, c2FtcGxlCg== is a valid encoded data while b?HV3.Zh2J== is not a valid encoded data.
In this Linux Hint tutorial, we will explain how to use the base64 command to encode and decode the data in a string or a file. We have performed the commands on Ubuntu 20.04 Focal Fossa system. However, you can also run the same commands on other Linux distributions.
Base64 Syntax
Here is the syntax for encoding using Base64:
Options
Some of the command-line options that can be used with base64 command are:
Use this option to decode a file or a string.
Use this option to display help regarding the usage of base64.
Use this option while decoding to ignore non-alphabet characters
Use this option to display version information
Encoding String
You can easily encode a string using the base64 command. For instance, to encode a sample text “Welcome to Linux” to base64, the command would be:
This command will encode the text in the string using base64 and print the encoded text to standard output as shown in the Terminal Output above.
You can also save the encoded output to a file rather than printing to standard output using the redirection operator (>). The following command will encode the text and save the output to a file named “encodedfile.txt:
To view the encoded file, you can use the cat command:
Decoding String
You can also decode the base64 encoded text using the –decode or -d option. For instance to decode base64 encoded text “V2VsY29tZSB0byBMaW51eAo=”, the command would be:
This command will decode the base64 encoded text and print the original text on the standard output as shown in the following Terminal Output.
Terminal Output:
linuxhint@hp34:~$ echo 4 oCcV2VsY29tZSB0byBMaW51eOKAnQo = | base64 —decode
“Welcome to Linux”
Encoding Text File
The base64 command can also be used to encode a text file. For this example lets create a text file with some content first using this command on the terminal, or use any textfile.
To encode this or any text file named “testfile.txt”, the command would be:
This command will encode the specified text file and print its encoded form on the standard output as shown in the following Terminal Output:
linuxhint@hp34:~$ base64 testfile.txt
SSBMb3ZlIExpbnV4CkkgTG92ZSBMaW51eApJIExvdmUgTGludXgKTGludXhIaW50IGlzIG15IGhv
bWVwYWdlCg ==
You can also save the encoded output to a file rather than printing to standard output using the redirection operator (>). The following command will convert the text in the file using base64 and save the output to another file named “encodedfile.txt”. The command does not print anything to the screen.
To view the encoded file, you can use the cat command:
Terminal Output:
linuxhint@hp34:~$ base64 testfile.txt > encodedfile.txt
linuxhint@hp34:~$ cat encodedfile.txt
SSBMb3ZlIExpbnV4CkkgTG92ZSBMaW51eApJIExvdmUgTGludXgKTGludXhIaW50IGlzIG15IGhv
bWVwYWdlCg ==
Decoding Text File
To decode an encoded text file, use the –decode or -d option. For instance to decode base64 encoded text file “encodedfile.txt”, the command would be:
This command will decode the base64 encoded text file and print the original text on the standard output as shown in the following Terminal Output:
Terminal Output:
linuxhint@hp34:~$ base64 -d encodedfile.txt
I Love Linux
I Love Linux
I Love Linux
LinuxHint is my homepage
linuxhint@hp34:~$
Encoding User Input
Using the base64 encoding, we can encode any user-provided data. For this purpose, we will need to create a script that will take user input, encode it using base64 encoding, and print the encoded data on standard output. Create a script “test.sh” with the following code:
#!/bin/bash
# Print message to ask for input
echo «Provide Some data to encode»
# Save the input to a variable named “data”
read data
# Encode using base64 encoding and save the output to a variable “encod_data”
encod_data = ` echo -n $data | base64 `
# Print encoded output
echo «Encoded text is : $encod_data «
Run the script as follows:
After running the script, you will be asked to input the data that you want to encode. Type some data and press Enter, and you will receive the encoded output on the screen as shown below:
Terminal Output:
linuxhint@hp34:~$ chmod 755 test.sh; . / test.sh
Provide Some data to encode
I love linux
Encoded text is : SSBsb3ZlIGxpbnV4
This encoded text can be sent over the internet or to another program and then decoded later using a simple command. For this command we assume the receiving program, decode.sh, has put the encoded data into a variable in BASH called RECEIVED_ENCODING.
#!/bin/bash
RECEIVED_ENCODING =SSBsb3ZlIGxpbnV4
RECEIVED_STRING = ` echo $RECEIVED_ENCODING | base64 —decode `
echo $RECEIVED_STRING
Here you can see the results of the receiving program in the Terminal Output:
Conclusion
This is how you can use the base64 to encode and decode a string or a file from the command line. The results can be printed on the standard output, saved in a variable or a file, or passed over the network to another program. However, remember that encoding is not similar to encryption, and one can easily reveal the encoded data, so it is not recommended to use encoding for the transmission of sensitive data unless its also encrypted.
About the author
Linux Wolfman
Linux Wolfman is interested in Operating Systems, File Systems, Databases and Analytics and always watching for new technologies and trends. Reach me by tweeting to @linuxhint and ask for the Wolfman.