Convert file to hex linux

Convert a list of decimal values in a text file into hex format

I have a need to convert a list of decimal values in a text file into hex format, so for example test.txt might contain:

131072 196608 262144 327680 393216 . 

00020000 00030000 00040000 . 

EDIT #1

I missed one extra operation: I need to add 80000000 hex to each of the created hex values. (arithmetic addition, to apply to already created list of hex values).

Do you mean add 0x80000000 to number before converting? If not, are you adding using 2’s complement; how do you want overflows to be handled. i.e., what output do you want for -2147483648, -1, 0, 1 and 2147483647, (I suspect you actually want offset binary not 2’s complement)

6 Answers 6

You can do this using printf and bash :

Or using printf and bc. just. because?

In order to print the output to a text file just use the shell redirect > like:

You should use bc as primary (only?) answer. bc is arbitrary precision, printf will fail for large numbers.

@Dani_l: I don’t know of any way to reliably pad the bc output with zeros the way OP wants without using printf.

@Dani_l, the OP’s requirement to output in 8 hex digit format will fail for large numbers. I assume he meant 8 hexit when he used 8 bit.

@Grump the OP’s requirement is satisfied automatically by the addition operation, as the ouput would always b 8 or more digits, and reducing the number of digits doesn’t make sense in an addition operation.

$ awk '/7/ < printf("%08x\n", $0) >' file 00020000 00030000 00040000 00050000 00060000 

@minto Did you run the command on a file that had . on the last line? If so, then yes. It would convert all values to hexadecimal, even empty lines. Data that can not be converted to an integer would be taken as zero.

@minto See updated answer. The updated answer outputs only if the input line contains a number. How big is your file?

Three possible solutions (assuming each line is a set of only digits):

For shells like ksh, bash, zsh:

For simpler shells: dash (the default sh in Debian based systems), ash (busybox emulated shell), yash and some default shells in AIX and Solaris:

In fact, for a shell like the heirloom version (Bourne like) it needs to be written like (which do work on all posix shells listed above but I strongly recommend to do not use it ):

Answer to EDIT #1

Understand that an hex value of 80000000 will cause overflow on a 32 bit computer (not common this days, but possible). Check that echo «$((0x80000000))» do not print a negative value.

As I said in the first line: «assuming each line is a set of only digits» it makes no difference to add the complexity of read. @Dani_I

@Isaac maybe some clarification should be done regarding hex value of 80000000 that may cause potential overflow on a 32 bit computer. 80000000 is the base address of the flash chip in the embedded system. Base address chosen by device manufacturer, so likely allowed by the processor’s memory controller. I’m not sure it can cause overflow in this case.

@minto In answer to your comment I’m not sure it can cause overflow in this case I have to say: I do not know if you are running the shell inside the same system that use that memory address. You may be developing and compiling on one computer and then sending the compiled code to the embedded system. Or . you may be running the shell inside the embedded system, in which case it seems reasonable to expect that the shell has been compiled to run in that system and that the shell has been compiled with the same bit size. But I can not know that both facts are true.

Источник

How to convert file data to plain hex?

How to easily convert to/from plain machine-readable hexadecimal data (without any paddings/offsets/character view) with xdd or hexdump ? I’m tired of digging of some special format strings (and finding out that it suddenly starts wrapping lines after N characters or skip lines) or writing Perl one-liners every time. Why is it not as simple as base64 / base64 -d ?

3 Answers 3

If you get tired of writing this every time, create an alias.

I’m tired of looking at man page and gettings confused either by necessity to compose hexdump’s formt strings or by xxd’s «postscript» thing. (Is this «postscript» something about printers or about letters?)

@Vi, think of -p as «plain». I surmise the Postscript comment is because the Postscript language allows for in-line data (typically images) encoded exactly this way. So Postscript programmers may use xxd to convert binary images into a convenient form for embedding in a Postscript file.

How to easily convert to/from plain machine-readable hexadecimal data

$ xxd -plain test.txt > test.hex $ xxd -plain -revert test.hex test2.txt $ diff test.txt test2.txt $

$ xxd -plain test.txt > test.hex 

This writes a hex encoding of the data in test.txt into new file test.hex. The -p or -plain option makes xxd use «plain» hex format with no spaces between pairs of hex digits (i.e. no spaces between byte values). This converts «abc ABC» to «61626320414243». Without the -p it would convert the text to a 16-bit word oriented traditional hexdump format, which is arguably easier to read but less compact and therefore less suitable as a transmission format and slightly harder to reverse.

$ xxd -plain -revert text.hex test2.txt 

This uses the -r or -revert option for reverse operation. The -plain option is used again to indicate that the input hex file is in plain format.

I make the output filename different from the original filename so we can later compare the results with the original file.

The diff command outputs nothing — this means there is no difference between the original and reconstituted file contents.

I’m tired of digging of some special format strings

Use alias or declare functions in your .profile to create mnemonics so you don’t have to remember or dig about in man pages.

or just remember -plain and -revert .

Yes, there are new-line characters in the output. You want to avoid that. You can use the -c or -cols option to specify how long you want the output lines to be to attempt to avoid line-wrapping of the output. -c 0 gives the default length and the man page suggests 256 is the limit but it seems to work beyond that.

$ xxd -plain -cols 9999 test.txt > test.hex $ wc test.txt test.hex 121 880 4603 test.txt 1 1 9207 test.hex 

The wc wordcount command tells us how many lines, words and characters are in each file.

So 121 lines (880 words, 4603 bytes) of ASCII text were encoded as 1 line of hex digits.

Источник

Converting from binary to hex and back

A hexdump command comes with busybox , but it is different from the one that comes with util-linux .

I’m looking for a script or command to convert a file to a hex string, and a corresponding one for converting it back to binary. The intermediate format doesn’t have to be hex, it can be base64 or something else.

This is for an embedded device with limited disk space.

Out of interest, why do you need to? What constraints are there on the string, does it just have to be printable?

It’s for an old system, for copying a file over telnet in an expect script, for debugging. The string does not have to be printable, but it’s an advantage that it doesn’t have to be escaped when using expect. (I know, I know, telnet is pretty bad. Luckily, it was removed in all later versions).

From the source, it looks like 1.18.4 should have both uuencode and makemime . Are they not available?

3 Answers 3

Here’s what I came up with (based on several online sources and some experimentation).

Converting from hex to bin ( hex2bin ):

#!/bin/sh sed 's/\([0-9A-F]\\)/\\\\\\x\1/gI' "$1" | xargs printf 

Converting from bin to hex ( bin2hex ):

#!/bin/sh hexdump -v -e '1/1 "%02x"' "$1" 

Example use:

./bin2hex binary_file_1 | ./hex2bin - > binary_file_2 diff -s binary_file_1 binary_file_2 

This works with busybox, but hex2bin is unfortunately limited by the maximum length of the argument given to xargs , so this method will only work for small files (less than 32 KiB on my desktop system).

Thanks, removed cat frrom bin2hex . When using sed -i instead of cat and sed , however, sed complained that it could not read — (while cat had no such complaint).

@Alexander, -i tries to edit in place, which is hard to do for a stream. You can still do without the cat and use sed < "$1" 's/. '" | .

@Alexander, ah yes, I missed you wanted that. Though of course you could explicitly test $1 against a dash but of course it’s easier since cat has the logic built-in.

POSIXly (and only using a common subset compatible with busybox (or at least the busybox as built for the current busybox Debian package):

( export LC_ALL=C od -An -vtx1 | tr -s ' \t\n' '\n\n\n' | grep . ) 

( export LC_ALL=C awk ' BEGIN < for (i = 0; i < 256; i++) c[sprintf("%02x", i)] = sprintf("%o", i) >NR % 200 == 1  END ' | sh ) 

If your busybox , contrary to Debian’s one has been built without the DESKTOP option, then the -An and -tx1 option to od won’t be available. You can use od -b instead which will give a one-byte octal dump with octal offsets. od -b is Unix but not POSIX however so won’t work on every Unix-like system.

( export LC_ALL=C od -b | awk ' BEGIN < for(i = 0; i < 256; i++) hex[sprintf("%03o", i)] = sprintf("%02x", i) >NF > 1 ' ) 

Again, tested only with Debian’s busybox, I can’t tell how much of that is dependant on some busybox compile-time option or another. You’d have to test on the target system.

When running this bin2hex script on the hardware device, I get: «` od: invalid option — ‘A’ BusyBox v1.26.2 (2017-03-13 14:50:51 CET) multi-call binary. Usage: od [-aBbcDdeFfHhIiLlOovXx] [FILE] Print FILE (or stdin) unambiguously, as octal bytes by default «` I get the same result on my desktop system.

@Alexander, yes, as always with busybox, how much of the POSIX standard is implemented depends very much on the version and the selected configuration options at build time. Here, for it to support od -A , it seems you need to tick the DESKTOP option («Enable options for full-blown desktop systems») at build time (as is done with my Debian busybox it would seem). You can always leave out the -An and filter out the address with sed , but it looks like it would still not support -tx1 , so you’re probably out of luck with your busybox od . Or you’d need awk again to translate oct to hex

The new version seems to freeze on my desktop system (64-bit Arch Linux, busybox 1.26.2). On the test hardware with the new version of busybox (1.26.2) it also times out and produces no output. I’m not well versed enough in awk to see why. od -b works as expected. busybox awk is present. Same result with busybox 1.18.4.

Источник

Linux script to convert byte data into a hex string

Now, I want to create one Linux script that will convert that data into a hex string. How can I do that?

2 Answers 2

Should you want only the hex strings:

$ echo '0Y0*†HÎ*†HÎ=B¬`9E>ÞÕ?ÐŽ·‹ñ6ì­Â‰&ÉÐL_cüsyxoú¢'|od -vt x1|awk '' 30 59 30 2a e2 80 a0 48 c3 8e 2a e2 80 a0 48 c3 8e 3d 42 c2 ac 60 39 45 3e c3 9e c3 95 3f c3 90 c5 bd c2 b7 e2 80 b9 c3 b1 36 c3 ac c2 ad c3 82 e2 80 b0 26 c3 89 c3 90 4c 5f 63 c3 bc 73 79 78 6f c3 ba c2 a2 0a 

You can avoid the awk part by just using od -vt x1 -A n . Thanks @Stefan van den Akker.

My GNU od version 9.0 has the flags -A n which skips the offset so you don’t have to filter through awk .

Use the hd (hex dump) command:

$ echo '0Y0*†HÎ*†HÎ=B¬`9E>ÞÕ?ÐŽ·‹ñ6ì­Â‰&ÉÐL_cüsyxoú¢' | hd 00000000 30 59 30 2a e2 80 a0 48 c3 8e 2a e2 80 a0 48 c3 |0Y0*. H..*. H.| 00000010 8e 3d 42 c2 ac 60 39 45 3e c3 9e c3 95 3f c3 90 |.=B..`9E>. | 00000020 c5 bd c2 b7 e2 80 b9 c3 b1 36 c3 ac c2 ad c3 82 |. 6. | 00000030 e2 80 b0 26 c3 89 c3 90 4c 5f 63 c3 bc 73 79 78 |. &. L_c..syx| 00000040 6f c3 ba c2 a2 0a |o. | 00000046 

Or, if you don’t have hd , hexdump :

$ echo '0Y0*†HÎ*†HÎ=B¬`9E>ÞÕ?ÐŽ·‹ñ6ì­Â‰&ÉÐL_cüsyxoú¢' | hexdump 0000000 5930 2a30 80e2 48a0 8ec3 e22a a080 c348 0000010 3d8e c242 60ac 4539 c33e c39e 3f95 90c3 0000020 bdc5 b7c2 80e2 c3b9 36b1 acc3 adc2 82c3 0000030 80e2 26b0 89c3 90c3 5f4c c363 73bc 7879 0000040 c36f c2ba 0aa2 0000046 

Источник