How to convert file data to plain hex?
How to easily convert to/from plain machine-readable hexadecimal data (without any paddings/offsets/character view) with xdd or hexdump ? I’m tired of digging of some special format strings (and finding out that it suddenly starts wrapping lines after N characters or skip lines) or writing Perl one-liners every time. Why is it not as simple as base64 / base64 -d ?
3 Answers 3
If you get tired of writing this every time, create an alias.
I’m tired of looking at man page and gettings confused either by necessity to compose hexdump’s formt strings or by xxd’s «postscript» thing. (Is this «postscript» something about printers or about letters?)
@Vi, think of -p as «plain». I surmise the Postscript comment is because the Postscript language allows for in-line data (typically images) encoded exactly this way. So Postscript programmers may use xxd to convert binary images into a convenient form for embedding in a Postscript file.
How to easily convert to/from plain machine-readable hexadecimal data
$ xxd -plain test.txt > test.hex $ xxd -plain -revert test.hex test2.txt $ diff test.txt test2.txt $
$ xxd -plain test.txt > test.hex This writes a hex encoding of the data in test.txt into new file test.hex. The -p or -plain option makes xxd use «plain» hex format with no spaces between pairs of hex digits (i.e. no spaces between byte values). This converts «abc ABC» to «61626320414243». Without the -p it would convert the text to a 16-bit word oriented traditional hexdump format, which is arguably easier to read but less compact and therefore less suitable as a transmission format and slightly harder to reverse.
$ xxd -plain -revert text.hex test2.txt This uses the -r or -revert option for reverse operation. The -plain option is used again to indicate that the input hex file is in plain format.
I make the output filename different from the original filename so we can later compare the results with the original file.
The diff command outputs nothing — this means there is no difference between the original and reconstituted file contents.
I’m tired of digging of some special format strings
Use alias or declare functions in your .profile to create mnemonics so you don’t have to remember or dig about in man pages.
or just remember -plain and -revert .
Yes, there are new-line characters in the output. You want to avoid that. You can use the -c or -cols option to specify how long you want the output lines to be to attempt to avoid line-wrapping of the output. -c 0 gives the default length and the man page suggests 256 is the limit but it seems to work beyond that.
$ xxd -plain -cols 9999 test.txt > test.hex $ wc test.txt test.hex 121 880 4603 test.txt 1 1 9207 test.hex The wc wordcount command tells us how many lines, words and characters are in each file.
So 121 lines (880 words, 4603 bytes) of ASCII text were encoded as 1 line of hex digits.
Binary to hexadecimal and decimal in a shell script
I have a context where I need to convert binary to hexadecimal and decimal and viceversa in a shell script. Can someone suggest me a tool for this?
Seeing the recent answers, maybe you should specify whether “binary” means binary number or binary file.
7 Answers 7
It’s fairly straightforward to do the conversion from binary in pure bash ( echo and printf are builtins):
Binary to decimal
Binary to hexadecimal
Going back to binary using bash alone is somewhat more complex, so I suggest you see the other answers for solutions to that.
What’s the point of using printf %d for bin2dec? $(( . )) already gives a decimal result, echo is enough.
@AndreaCorbellini — Personal preference. In general, I use printf ‘%s\n’ foo instead of echo for a variety of reasons (mostly portability), for the same reason, I don’t use it here.
@ChrisDown: I though this question was explicitly about Bash (which has a well-implemented echo builtin). My bad!
Assuming that by binary, you mean binary data as in data with any possible byte value including 0, and not base-2 numbers:
To convert from binary, od (standard), xxd (comes with vim ) or perl ‘s unpack come to mind.
od -An -vtu1 # for decimal od -An -vtx1 # for hexadecimal xxd -p # for hexa perl -pe 'BEGIN;$_=unpack("H*",$_)' # like xxd -p # for decimal: perl -ne 'BEGIN; print unpack("C*",$_)' Now, to convert back to binary, awk (standard), xxd -r or perl ‘s pack :
From the decimal output from od -tu1 or perl above:
LC_ALL=C awk '' perl -ape '$_=pack("C*",@F)' From the hexa perl or xxd -p above:
xxd -r -p perl -pe 'chomp;$_=pack("H*",$_)' xxd -r gives binary data, not ascii equivalent of hex data. ie 11111111 in binary file is FF in hex data. The question is asking to give 11111111 in ascii (displayable on screen)
You can use bc for this by manipulating the ibase and obase parameters:
The trick is that you need to be explicit about the bases. So if your ibase is 2, then if you set your obase to 10, it won’t do anything, as 10 in binary is 2. Hence you need to use hexadecimal notation.
So binary to decimal would be (watch that obase is A)
$> echo 'ibase=2;obase=A;11110001011010'|bc 15450 $> echo 'ibase=2;obase=10000;11110001011010'|bc 3C5A If the ‘output base’ obase is changed first, it should be easier:
$> echo 'obase=10;ibase=2;11110001011010'|bc 15450 $> echo 'obase=16;ibase=2;11110001011010'|bc 3C5A This second example is wrong — ‘F’ would be base 15, not 16 (decimal 16 in hex is 10, not F). Try echo ‘ibase=2;obase=F;1111 which should equal decimal 15, ie. F in hex. It comes out as 10, which it is in base 15 (digits 0-E). It is also easier if you state the obase first, eg: echo ‘obase=16;ibase=2;1111’|bc . No confusion.
Now it is worst. Once you set ibase , you have to provide input in that base, even for obase . So in your example would be echo ‘ibase=2;obase=10000;11110001011010’|bc . Better listen to goldilocks’s advice and reverse the order – first set obase , then ibase .
If you mean converting numbers from base-2 to 10 or 16 and back, bc is the standard tool to do that as already mentioned by psarossy.
decimal=123 binary=$(echo "obase=2;$decimal" | bc) hex=$(echo "obase=16;ibase=2;$binary" | bc) Some shells like zsh have builtin support for base conversion as part of their arithmetic expansion operators:
decimal=123 binary=$(([##2]decimal)) hex=$(([##16]decimal)) hex=$(([##16]2#$binary)) decimal=$((2#$binary)) Both ksh93 and zsh also support:
typeset -i2 binary=123 typeset -i16 dec2hex=123 bin2hex='2#1111' But note that, when expanded, $binary will have a 2# or 16# prefix (which you can strip with $ .
For binary to hexadecimal use: xxd tool in linux and for binary to decimal you can use qalculate tool.
For help regarding xxd type xxd —help or man xxd in Linux.
As mentioned in previous answer you can do Binary to Decimal and Hexa Decimal as follows in Bash using echo and printf. I am just adding here how to convert from Decimal and Hex to binary using pure Bash.
Binary to Decimal using echo
Binary to Hexa-Decimal using printf
Integer decimal to Binary conversion using only Bash
Using only Bash, if you would like to convert decimal to binary, you can do it like the following:
touch dec2bin && chmod +x "$_" && vim "$_" And then copy and paste the following:
#!/bin/bash ## converting decimal integer to binary, pass int as a parameter num=$1; dec2bin() < [ "$num" == "" ] && < printf "Error: Pass an integer\n"; exit 1; >; op=2; ## Since we're converting to binary quo=$(( $num / $op)); rem=$(( $num % $op)); ## quotient and remainder remarray=(); ## array for putting remainder inside array remarray+=("$rem"); ## array expansion until [[ $quo -eq 0 ]]; do num=$quo; quo=$(( $num / $op)); ## looping to get all remainder untill 0 rem=$(( $num % $op)); remarray+="$rem"; ## array expansion done binary=$(echo "$" | rev); ## reversing array printf "$binary\n"; ## printing reversed array > main() < [[ -n $]] && < printf "Error: $num is not an integer bruv!\n"; return 1; >|| < dec2bin $num; >> main; And then try it from where you saved:
./dec2bin 420.py 420.py is not an integer bruv! Hexadecimal to Binary conversion using only Bash
Similarly, hexadecimal to binary, as follows using only bash:
#!/usr/local/bin/bash ## converting hexadecimal to binary, pass hex as a parameter hex=$1; hex2bin() < [ "$hex" == "" ] && < printf "Error: Pass a hex\n"; exit 1; >; op=2; num=$((16#$hex)); ## converting hex to integer quo=$(( $num/ $op)); rem=$(( $num% $op)); ## quotient and remainder remarray=(); remarray+=("$rem"); ## array expansion until [[ $quo -eq 0 ]]; do num=$quo; quo=$(( $num / $op)); ## looping to get all remainder untill 0 rem=$(( $num % $op)); remarray+="$rem"; ## array expansion done binary=$(echo "$" | rev); ## reversing array printf "$binary\n"; ## printing reversed array > main() < [[ -n $]] && < printf "Error: $hex is not a hex bruv!\n"; return 1; >|| < hex2bin $hex; >> main; ./hex2bin.bash XyZ XyZ is not a hexa decimal number bruv! Convert binary data to hexadecimal in a shell script
I want to convert binary data to hexadecimal, just that, no fancy formatting and all. hexdump seems too clever, and it «overformats» for me. I want to take x bytes from the /dev/random and pass them on as hexadecimal. Preferably I’d like to use only standard Linux tools, so that I don’t need to install it on every machine (there are many).
@AquariusPower random blocks when it does not have enough random data, while urandom does not (IIRC loops over what it has)
9 Answers 9
% xxd -l 16 -p /dev/random 193f6c54814f0576bc27d51ab39081dc Note that you can use -c to change the number of bytes per line. Unfortunately you can only set it to 256 after which you need to have some newlines.
I just wanted to say thank you and that this is the fastest ive ever seen for generating data without doing overthe top work. i literally did a 2gb file in a few seconds I needed 32 byte strings in hex and this did the trick time xxd -c 32 -l 1024000000 -ps /dev/urandom 32bytehexnewtest1.txt ; real 0m17.484s
hexdump and xxd give the results in a different endianness!
$ echo -n $'\x12\x34' | xxd -p 1234 $ echo -n $'\x12\x34' | hexdump -e '"%x"' 3412 $ echo abc | od -A n -v -t x1 | tr -d ' \n' 6162630a $ echo abc | hexdump -ve '/1 "%02x"' 6162630a «Depending on your system type, either or both of these two utilities will be available—BSD systems deprecate od for hexdump, GNU systems the reverse.»
After 4 years (not much, I know), most times I read a thread about *nix, I learn about something really old and still completely new for me. +1 for the od, never heard about it, very useful and present even on Cygwin. 😉
Perhaps you could write your own small tool in C, and compile it on-the-fly:
int main (void) < unsigned char data[1024]; size_t numread, i; while ((numread = read(0, data, 1024)) >0) < for (i = 0; i < numread; i++) < printf("%02x ", data[i]); >> return 0; > And then feed it from the standard input:
You can even embed this small C program in a shell script using the heredoc syntax.
@user405725 — This seems to be the simplest solution for a C programmer. The best I can tell, all the other solutions do not hex encode a binary file. I am befuddled at how difficult this task has become. Is it really that f**k’ing hard to hex encode a binary file?
All the solutions seem to be hard to remember or too complex. I find using printf the shortest one:
But as noted in comments, this is not what author wants, so to be fair, below is the full answer.
. to use above to output actual binary data stream:
printf '%x\n' $(cat /dev/urandom | head -c 5 | od -An -vtu1) - printf ‘%x\n’ . — prints a sequence of integers , i.e. printf ‘%x,’ 1 2 3 , will print 1,2,3,
- $(. ) — this is a way to get output of some shell command and process it
- cat /dev/urandom — it outputs random binary data
- head -c 5 — limits binary data to 5 bytes
- od -An -vtu1 — octal dump command, converts binary to decimal
As a testcase (‘a’ is 61 hex, ‘p’ is 70 hex, . ):
$ printf '%x\n' $(echo "apple" | head -c 5 | od -An -vtu1) 61 70 70 6c 65 Or to test individual binary bytes, on input let’s give 61 decimal (‘=’ char) to produce binary data ( ‘\\x%x’ format does it). The above command will correctly output 3d (decimal 61):
$printf '%x\n' $(echo -ne "$(printf '\\x%x' 61)" | head -c 5 | od -An -vtu1) 3d @МалъСкрылевъ I have updated my answer, you are right, I missed the main point of question — I was probably looking for simplest way of converting decimal to hex in shell, and this answer shows up in google as first.
hey, I know this is old but for the second code box couldnt you just skip the cat and just do the head directly? printf ‘%x\n’ $(head -c 5 /dev/urandom | od -An -vtu1) or is there a specific reason for the cat command? Also, if you wanted to have the hex only separated by a new line at the end I came up with this echo $(printf «%x» $(head -c 5 /dev/urandom | od -An -vtu1)) although I dont know the performance costs for these and mainly why im asking about the cat command being cut out. I’m pretty new to command line but this information all helps.