Environment variable substitution in sed
I read in tutorials that to substitute environment variables from shell you need to stop, and ‘out quote’ the $varname part so that it is not substituted directly, which is what I did, and which works only if the variable is defined immediately before. How can I get sed to recognize a $var as an environment variable as it is defined in the shell?
Use set -x in the shell to get the shell to echo each command just before it executes them. This can clear up a lot of confusion. (Also, I often use set -u to make de-referencing unset variables a hard error. (See set -e too.))
I was hoping to find a way for sed to handle the environment variables as not to leak the values into the process table, seems like sed is the wrong tool for installing secrets according to all the answers in this thread
12 Answers 12
Your two examples look identical, which makes problems hard to diagnose. Potential problems:
- You may need double quotes, as in sed ‘s/xxx/'»$PWD»‘/’
- $PWD may contain a slash, in which case you need to find a character not contained in $PWD to use as a delimiter.
To nail both issues at once, perhaps
You can have several candidates like @#%! and check with a case expression to find if $PWD has them. E.g., case «$PWD» of @) ;; *) delim=»@» ;; esac; repeat until $delim is not empty.
You can use another delimiter for sed. You dont need to use / you can use , as well if your environment variable is an url.
What if the string contains a \ followed by an n — how to stop sed from converting that into a single newline character?
In addition to Norman Ramsey’s answer, I’d like to add that you can double-quote the entire string (which may make the statement more readable and less error prone).
So if you want to search for ‘foo’ and replace it with the content of $BAR, you can enclose the sed command in double-quotes.
sed 's/foo/$BAR/g' sed "s/foo/$BAR/g" In the first, $BAR will not expand correctly while in the second $BAR will expand correctly.
this is what I had to use to get the environment variable to expand correctly in this command: sed -i «s/127.0.0.1/127.0.0.1 localhost $HOSTNAME/» hosts
This is neat but it is not working when you want to do some more complex substitutions, such as «2,$s/^/$foo/» as $s gets interpreted as a variable too and it should not.
This alone didn’t work as I still needed a different delimiter for paths like the other answers suggest.
Since $PWD will usually contain a slash / , use | instead of / for the sed statement:
Nice clean pattern but the -e (—expression) is not needed here, since this example is not stringing several expressions together.
This has been better using | than / as some of the value in my variable were having special characters like preview:1.2.7-2208.190204
You can use other characters besides «/» in substitution:
In my specific case, $2 was a file path, so sed was barfing due to interpreting the / in the contents of $2. This was exactly what I needed to get past it. Thanks fora great tip!
一. bad way: change delimiter
sed 's/xxx/'"$PWD"'/' sed 's:xxx:'"$PWD"':' sed 's@xxx@'"$PWD"'@' maybe those not the final answer,
you can not known what character will occur in $PWD , / : OR @ .
if delimiter char in $PWD, they will break the expression
the good way is replace(escape) the special character in $PWD .
二. good way: escape delimiter
for example: try to replace URL as $url (has : / in content)
A. use / as delimiter
escape / as \/ in var (before use in sed expression)
B. use : as delimiter (more readable than / )
escape : as \: in var (before use in sed expression)
With your question edit, I see your problem. Let’s say the current directory is /home/yourname . in this case, your command below:
which is not valid. You need to put a \ character in front of each / in your $PWD if you want to do this.
Actually, the simplest thing (in GNU sed, at least) is to use a different separator for the sed substitution ( s ) command. So, instead of s/pattern/’$mypath’/ being expanded to s/pattern//my/path/ , which will of course confuse the s command, use s!pattern!’$mypath’! , which will be expanded to s!pattern!/my/path! . I’ve used the bang ( ! ) character (or use anything you like) which avoids the usual, but-by-no-means-your-only-choice forward slash as the separator.
Dealing with VARIABLES within sed
[root@gislab00207 ldom]# echo domainname: None > /tmp/1.txt [root@gislab00207 ldom]# cat /tmp/1.txt domainname: None [root@gislab00207 ldom]# echo $ dcsw-79-98vm.us.oracle.com [root@gislab00207 ldom]# cat /tmp/1.txt | sed -e 's/domainname: None/domainname: $/g' --- Below is the result -- very funny. domainname: $ --- You need to single quote your variable like this . [root@gislab00207 ldom]# cat /tmp/1.txt | sed -e 's/domainname: None/domainname: '$'/g' --- The right result is below domainname: dcsw-79-98vm.us.oracle.com I was struggling to get this working in Azure DevOps YAML pipelines. This comment helped me to successfully use this trick to tokenize some configuration files.
VAR=8675309 echo "abcde:jhdfj$jhbsfiy/.hghi$jh:12345:dgve::" |\ sed 's/:4*:/:'$VAR':/1' where VAR contains what you want to replace the field with
How to use variable in sed search pattern
I know it’s wrong, because the variable identifier «$» is conflict with perl pattern $ which means the end of the line. So my question is: is there any way to use variable in sed patter search?
the shell expands $VARIABLE inside double quotes before spawning sed . Your problem here is /$VARIABLE,+4/ should be /$VARIABLE/,+4 probably.
1 Answer 1
It looks like you’re trying to use the sed range notation, i.e. /start/,/end/?. Is that correct?
If so all you need to do is add the additional ‘/’ chars that are missing, i.e.
sed -n "/$VARIABLE/,/test/p" aFile A range can be composed of line numbers, strings/regexs, and/or relative line numbers (with no negative look back).
sed -n "1,20p" aFile # prints lines 1-20 sed -n '/start/,/end/p' aFile # prints any text (include current line) # with start until it finds the word end sed -n '/start/,+2p' aFile # prints line matching start and 2 more lines after it I am using strings in a /regex/ pattern to simplfy the explanation. You can do real rex-ex’s like /^[A-Za-z_][A-Za-z0-9_]*/ etc. as you learn about them.
Also per your comment about ‘$’, sed will also use ‘$’ as an indicator of ‘end-of-line’ IF the character is visible in the reg-ex that is being evaluated. Also note how some of my examples use dbl-quotes OR single-quotes. Single quotes mean that your expression sed -n ‘/$VARIABLE/,/test/p’ aFile would match the literal chars AND because the $ is not at the end of a reg-ex, it will be used as a regular character. The ‘$’ only applies as end-of-line, when it is at the end of a reg-ex (part); for example you could do /start$|start2$/ , and both of those would signify end-of-line.
As others have pointed out, your use of
sed -n "/$VARIABLE/,/test/p" aFile is being converted via shell variable explasion to
sed -n "/some text/,/test/p" aFile SO if you wanted to ensure your text was anchored at the end-of-line, you could write
sed -n "/$VARIABLE$/,/test/p" aFile sed -n "/some text$/,/test/p" aFile Using variables in sed
I’m a n00b trying to return my IP address to a variable, which is then used in the sed command within a bash script. I’m trying to replace the text ‘mycomputer’ in a file with my IP address without much luck. Here are my attempts:
localip=`ipconfig getifaddr en0` sed -i '' “s/mycomputer/$localip/” config.txt sed: 1: "“s/mycomputer/192.168 . ": invalid command code ? localip=`ipconfig getifaddr en0` sed -i '' 's/mycomputer/$localip/g' config.txt localip=`ipconfig getifaddr en0` sed -i '' 's/mycomputer/‘“$localip”’/g’ config.txt ./mytest.sh: line 5: unexpected EOF while looking for matching `'' ./mytest.sh: line 6: syntax error: unexpected end of file #!/bin/bash cd "`dirname "$0"`" localip=`ipconfig getifaddr en0’ sed -i '' "s/mycomputer/$localip/" config.txt Dn’t use “”’ quotes. they’re not ‘quotes’ as far as the shell is concerned. e.g. don’t use a wordprocessor as your code editor.
2 Answers 2
You got the double-quotes wrong:
sed -i '' “s/mycomputer/$localip/” config.txt This should work (notice the difference):
sed -i '' "s/mycomputer/$localip/" config.txt Actually you have similar problems on other lines too. So the full script, corrected:
#!/bin/bash cd $(dirname "$0") localip=$(ipconfig getifaddr en0) sed -i '' "s/mycomputer/$localip/" config.txt Note that -i » is for the BSD version of sed (in BSD systems and MAC). In Linux, you’d write the same thing this way:
sed -i "s/mycomputer/$localip/" config.txt thanks for this, not sure about the curly double quotes and where they came from. Your answer is still returning errors here. Maybe my shell script is the issue: #!/bin/bash cd » dirname «$0″ » localip=`ipconfig getifaddr en0’ sed -i » «s/mycomputer/$localip/» config.txt Error is: ./mytest.sh: line 4: unexpected EOF while looking for matching «’ ./mytest.sh: line 6: syntax error: unexpected end of file
The formatting doesn’t come out right in comments. Add that inside your question and I can have a look. Are you in Linux or what OS?
aaagghh fixed it with your help! I re-edited with the wrong type of quote.. of course.. never knew about the standard vs curly single and double quotes.. thanks for your help!