What is escape character in linux

Which characters need to be escaped when using Bash?

and worked fine, without escaping % . Does it mean % does not need to be escaped? Was this a good way to check the necessity? And more general: are they the same characters to escape in shell and bash ?

7 Answers 7

There are two easy and safe rules which work not only in sh but also bash .

1. Put the whole string in single quotes

This works for all chars except single quote itself. To escape the single quote, close the quoting before it, insert the single quote, and re-open the quoting.

'I'\''m a s@fe $tring which ends in newline ' 

2. Escape every char with a backslash

This works for all characters except newline. For newline characters use single or double quotes. Empty strings must still be handled — replace with «»

\I\'\m\ \a\ \s\@\f\e\ \$\t\r\i\n\g\ \w\h\i\c\h\ \e\n\d\s\ \i\n\ \n\e\w\l\i\n\e" " 

2b. More readable version of 2

There’s an easy safe set of characters, like [a-zA-Z0-9,._+:@%/-] , which can be left unescaped to keep it more readable

I\'m\ a\ s@fe\ \$tring\ which\ ends\ in\ newline" " 

Note that in a sed program, one can’t know whether the last line of input ends with a newline byte (except when it’s empty). That’s why both above sed commands assume it does not. You can add a quoted newline manually.

Note that shell variables are only defined for text in the POSIX sense. Processing binary data is not defined. For the implementations that matter, binary works with the exception of NUL bytes (because variables are implemented with C strings, and meant to be used as C strings, namely program arguments), but you should switch to a «binary» locale such as latin1.

(You can easily validate the rules by reading the POSIX spec for sh . For bash, check the reference manual linked by @AustinPhillips)

Note: a good variation on #1 can bee seen here: github.com/scop/bash-completion/blob/…. It does not require running sed , but does require bash .

Note for anyone else (like me!) who struggles to get these working. looks like the flavour of sed you get on OSX doesn’t run these sed commands properly. They work fine on Linux though!

Seems you missed-out should the string start with a ‘-‘ (minus), or does that only apply to filenames? — in latter case need a ‘./’ in front.

For macOS users without GNU sed : @fd0 has a sed option to escape every character: apple.stackexchange.com/a/363400/409134 And I wrote a solution that only escapes the control characters using perl : apple.stackexchange.com/a/458279/409134

format that can be reused as shell input

Edit february 2021: bash $

Under bash, you could store your variable content with Parameter Expansion‘s @ command for Parameter transformation:

$ Parameter transformation. The expansion is either a transforma‐ tion of the value of parameter or information about parameter itself, depending on the value of operator. Each operator is a single letter: Q The expansion is a string that is the value of parameter quoted in a format that can be reused as input. . A The expansion is a string in the form of an assignment statement or declare command that, if evaluated, will recreate parameter with its attributes and value. 

$ var=$'Hello\nGood world.\n' $ echo "$var" Hello Good world. $ echo "$" $'Hello\nGood world.\n' $ echo "$" var=$'Hello\nGood world.\n' 

Old answer

There is a special printf format directive ( %q ) built for this kind of request:

 %q causes printf to output the corresponding argument in a format that can be reused as shell input. 

Some samples:

read foo Hello world printf "%q\n" "$foo" Hello\ world printf "%q\n" $'Hello world!\n' $'Hello world!\n' 

This could be used through variables too:

printf -v var "%q" "$foo " echo "$var" $'Hello world\n' 

Quick check with all (128) ascii bytes:

Note that all bytes from 128 to 255 have to be escaped.

for i in ;do printf -v var \\%o $i printf -v var $var printf -v res "%q" "$var" esc=E [ "$var" = "$res" ] && esc=- printf "%02X %s %-7s\n" $i $esc "$res" done | column 

This must render something like:

00 E '' 1A E $'\032' 34 - 4 4E - N 68 - h 01 E $'\001' 1B E $'\E' 35 - 5 4F - O 69 - i 02 E $'\002' 1C E $'\034' 36 - 6 50 - P 6A - j 03 E $'\003' 1D E $'\035' 37 - 7 51 - Q 6B - k 04 E $'\004' 1E E $'\036' 38 - 8 52 - R 6C - l 05 E $'\005' 1F E $'\037' 39 - 9 53 - S 6D - m 06 E $'\006' 20 E \ 3A - : 54 - T 6E - n 07 E $'\a' 21 E \! 3B E \; 55 - U 6F - o 08 E $'\b' 22 E \" 3C E \ < 56 - V 70 - p 09 E $'\t' 23 E \# 3D - = 57 - W 71 - q 0A E $'\n' 24 E \$ 3E E \>58 - X 72 - r 0B E $'\v' 25 - % 3F E \? 59 - Y 73 - s 0C E $'\f' 26 E \& 40 - @ 5A - Z 74 - t 0D E $'\r' 27 E \' 41 - A 5B E \[ 75 - u 0E E $'\016' 28 E \( 42 - B 5C E \\ 76 - v 0F E $'\017' 29 E \) 43 - C 5D E \] 77 - w 10 E $'\020' 2A E \* 44 - D 5E E \^ 78 - x 11 E $'\021' 2B - + 45 - E 5F - _ 79 - y 12 E $'\022' 2C E \, 46 - F 60 E \` 7A - z 13 E $'\023' 2D - - 47 - G 61 - a 7B E \ < 14 E $'\024' 2E - . 48 - H 62 - b 7C E \| 15 E $'\025' 2F - / 49 - I 63 - c 7D E \>16 E $'\026' 30 - 0 4A - J 64 - d 7E E \~ 17 E $'\027' 31 - 1 4B - K 65 - e 7F E $'\177' 18 E $'\030' 32 - 2 4C - L 66 - f 19 E $'\031' 33 - 3 4D - M 67 - g 

Where first field is hexa value of byte, second contain E if character need to be escaped and third field show escaped presentation of character.

Why , ?

You could see some characters that don’t always need to be escaped, like , , > and < .

So not always but sometime:

echo test 1, 2, 3 and 4,5. test 1, 2, 3 and 4,5. 

echo test test1 test2 test3 echo test\ test 1 test 2 test 3 echo test\ test 1 test 2 test 3 echo test\ test 1, 2 test 3 

@ThorSummoner, not if you pass the string as a literal argument to the shell from a different language (where you presumably already know how to quote). In Python: subprocess.Popen([‘bash’, ‘-c’, ‘printf «%q\0» «$@»‘, ‘_’, arbitrary_string], stdin=subprocess.PIPE, stdout=subprocess.PIPE).communicate() will give you a properly shell-quoted version of arbitrary_string .

FYI bash’s %q was broken for a long time — If my mind serves me well, an error was fixed (but might still be broken) in 2013 after being broken for ~10 years. So don’t rely on it.

Thank you to add special notes about , . I was surprised to learn that built-in Bash printf — %q ‘,’ gives \, , but /usr/bin/printf — %q ‘,’ gives , (un-escapted). Same for other chars: < , | , >, ~ .

@ilkkachu Depending on local config, using utf or iso as default, playing with bytes between 128 to 255 could lead to strange behaviour

To save someone else from having to RTFM. in bash:

Enclosing characters in double quotes preserves the literal value of all characters within the quotes, with the exception of $ , ` , \ , and, when history expansion is enabled, ! .

. so if you escape those (and the quote itself, of course) you’re probably okay.

If you take a more conservative ‘when in doubt, escape it’ approach, it should be possible to avoid getting instead characters with special meaning by not escaping identifier characters (i.e. ASCII letters, numbers, or ‘_’). It’s very unlikely these would ever (i.e. in some weird POSIX-ish shell) have special meaning and thus need to be escaped.

This is a short, sweet and mostly correct answer (+1 for that) but maybe it’s even better to use single quotes — see my longer answer.

Using the print ‘%q’ technique, we can run a loop to find out which characters are special:

#!/bin/bash special=$'`!@#$%^&*()-_+=<>|[]\\;\':",.<>?/ ' for ((i=0; i < $; i++)); do char="$" printf -v q_char '%q' "$char" if [[ "$char" != "$q_char" ]]; then printf 'Yes - character %s needs to be escaped\n' "$char" else printf 'No - character %s does not need to be escaped\n' "$char" fi done | sort 

No, character % does not need to be escaped No, character + does not need to be escaped No, character - does not need to be escaped No, character . does not need to be escaped No, character / does not need to be escaped No, character : does not need to be escaped No, character = does not need to be escaped No, character @ does not need to be escaped No, character _ does not need to be escaped Yes, character needs to be escaped Yes, character ! needs to be escaped Yes, character " needs to be escaped Yes, character # needs to be escaped Yes, character $ needs to be escaped Yes, character & needs to be escaped Yes, character ' needs to be escaped Yes, character ( needs to be escaped Yes, character ) needs to be escaped Yes, character * needs to be escaped Yes, character , needs to be escaped Yes, character ; needs to be escaped Yes, character < needs to be escaped Yes, character >needs to be escaped Yes, character ? needs to be escaped Yes, character [ needs to be escaped Yes, character \ needs to be escaped Yes, character ] needs to be escaped Yes, character ^ needs to be escaped Yes, character ` needs to be escaped Yes, character < needs to be escaped Yes, character | needs to be escaped Yes, character >needs to be escaped 

Some of the results, like , look a little suspicious. Would be interesting to get @CharlesDuffy's inputs on this.

Источник

Escaping Characters in Bash on Linux

Characters are used in source code, command lines, and the majority of computer interaction. The majority of characters, on the other hand, are not represented by keys on a standard keyboard, and many are not even readable. Another category of characters is complicated control characters.

We'll talk about character escape in Bash in this tutorial. We'll start off by briefly outlining how computers represent characters. First, we examine the various string types in Bash. The character escape in only Bash is then thoroughly described.

Note − Linux commands are case-sensitive.

Strings in Bash

Since a key does not represent each sign on a keyboard, writing and storing characters are two distinct processes. Strings are used in Bash to store text. Indeed, all variables in Bash are essentially character strings. Usually, they consist of single- or double-quoted, straight sequences.

We interpret or extrapolate particular character combinations in one context and take them literally in another, a crucial distinction between these two approaches.

Single Quotes

The primary goal of utilizing single quotes in Bash (') is to maintain each character's literal value. When you want the input to preserve its literal value without any data interpolation, use single quotes to extract any special value from a character. When you don't want to utilize the escape characters to alter how Bash understands the input string, single quotes are a useful alternative.

$ text='a $(echo b) c' $ echo "$" a $(echo b) c

Take note of how the single quotes keep the content intact. No interpolation is performed, but as a result, we are also unable to have a single quotation placed inside another single quote at any time.

Double Quotes

When history expansion is allowed, double quotes (") maintain the literal value of all characters with the exception of ( $ ), ('), ("), (), and the (!) character.

$ text="a" $ text="$ $(echo "b") c" $ echo "$" a b c

Initially, we use the [a] key alone to directly assign the character to text. Next, using the $ variable expansion, we obtain its value. The $ are translated in this sentence. Lastly, we assign back to text by joining this value to an expression and still another character.

Escaping Characters in Bash

Characters in Bash having specific meanings must be escaped outside of single quotes in order to maintain their literal values. In actuality, this is mostly accomplished using the backslash escape character. Sometimes we may need to use alternative techniques.

Let's examine when and how each approach is used.

1.Double Quotes

A backslash character () inside of double quotes only has a specific significance when it is followed by ( $ ), ('), ("), (), or a newline character. The backslash does not show up in the output if any of those characters are used after it within double quotes. If a backslash comes before it, a double quotation is acceptable between double quotes. To escape text inside double-quoted strings, we add the backslash character before the character.

$ text1="a $(echo b) c" $ text2="a \$(echo b) c" $ echo "$" a b c $ echo "$" a $(echo b) c

In the above example, See how the dollar symbol is escaped in text2 so that it retains its literal meaning while losing its special functions

No Quotes

There is a cost associated with doing away with the quotes, as we have already demonstrated. The following characters must all be escapes in order for any sequence without quotes to be unified: comma, period, underscore, plus sign, colon, commercial at, percent sign, slash, and hyphen.

$ text=a\ \&\ b\ \&\ c $ echo "$" a & b & c

As a good practice, it is better not to use quotes.

Conclusion

We spoke about character escape in Bash in this tutorial. The various character encoding tables were the first thing we discovered. Also, we observed that some characters are only used as command or marker text and cannot be printed. We need a way to get away from such characters in order to use them literally. We looked at pure Bash as well as several typical built-in characters escaping situations in Bash.

This tutorial demonstrated how to use both single and double quotes in bash and clarified the differences between them. You should be able to tell the shell how to read the input strings and whether to output the input as is or to interpolate data once you have learned the distinctions.

Источник