Using grep and sed to find and replace a string
I am using the following to search a directory recursively for specific string and replace it with another:
grep -rl oldstr path | xargs sed -i 's/oldstr/newstr/g' This works okay. The only problem is that if the string doesn’t exist then sed fails because it doesn’t get any arguments. This is a problem for me since i’m running this automatically with ANT and the build fails since sed fails. Is there a way to make it fail-proof in case the string is not found? I’m interested in a one line simple solution I can use (not necessarily with grep or sed but with common unix commands like these).
the reason i want to keep it as simple as possible is because my script connects to a remote server and runs this line with SSH. if i would use a shell script for this, i would have to copy the shell script to the server before and then run it there. i’m trying to avoid it and keep it simple. thanks.
Coming here from Google, I was looking for the exact string given in the question! (I didn’t have the problem the OP was suffering from)
8 Answers 8
You can use find and -exec directly into sed rather than first locating oldstr with grep . It’s maybe a bit less efficient, but that might not be important. This way, the sed replacement is executed over all files listed by find , but if oldstr isn’t there it obviously won’t operate on it.
find /path -type f -exec sed -i 's/oldstr/newstr/g' <> \; @Vlad: It is, because you’re running a separate sed for each file instead of letting xargs batch them. That said, unless you’re talking about several thousand tiny files, you’re unlikely to notice a difference.
My problem with this: the timestamp of every file will get reset to «right now» as if they were all «touched», even if no edits occurred in the file.
I’ve seen this «Hello world» example of using find all over the place, but nothing that constrains it from running sed on every file, including binaries. I have yet to find a working example with pipes, and I’ve tried a number of ways to hack the simplistic version of very limited utility.
@JerryMiller you can of course limit find to specific filename patterns, which is what I often do: find -name «*.c» -o -name «*.h» -type f -exec sed. would modify all the .c and .h files for example. To exclude binary files, it can get more complicated: unix.stackexchange.com/questions/46276/…
Your solution is ok. only try it in this way:
files=$(grep -rl oldstr path) && echo $files | xargs sed. so execute the xargs only when grep return 0 , e.g. when found the string in some files.
That is a nice underestimated solution! This is what I needed to execute sed over a list of files output of grep, and avoid the error: «can’t read . No such file or directory». Thanks for your answer!
Agree with @Leopoldo. This looks like the most Posixy and portable (for OS like Solaris) and the one with the least side effects (not touching every file mtime, etc). files=$(grep -rl oldstr path | cut -f 1 -d ‘:’ | sort | uniq) should build a smaller list where the files are listed once.
I have taken Vlad’s idea and changed it a little bit. Instead of
grep -rl oldstr path | xargs sed -i 's/oldstr/newstr/g' /dev/null sed: couldn't edit /dev/null: not a regular file I’m doing in 3 different connections to the remote server
touch deleteme grep -rl oldstr path | xargs sed -i 's/oldstr/newstr/g' ./deleteme rm deleteme Although this is less elegant and requires 2 more connections to the server (maybe there’s a way to do it all in one line) it does the job efficiently as well
Standard xargs has no good way to do it; you’re better off using find -exec as someone else suggested, or wrap the sed in a script which does nothing if there are no arguments. GNU xargs has the —no-run-if-empty option, and BSD / OS X xargs has the -L option which looks like it should do something similar.
thanks, i tried to use —no-run-if-empty but it still returns nonzero code (returns 1) and that would also trigger a build fail for me. how generic and common is find command ?
+1 I’m surprised the xargs -r fix is not mentioned in more answers. For this limited use case, it’s simple and sufficient.
I think that without using -exec you can simply provide /dev/null as at least one argument in case nothing is found:
grep -rl oldstr path | xargs sed -i 's/oldstr/newstr/g' /dev/null @michael: Turns out — running anything like this will actually crash the shell because of too many arguments. See superuser.com/questions/257250/…
there are a lot of files in my directory but only few of them contain the string i want to change. so i am not so worried about too many arguments to sed.
My use case was I wanted to replace foo:/Drive_Letter with foo:/bar/baz/xyz In my case I was able to do it with the following code. I was in the same directory location where there were bulk of files.
find . -name "*.library" -print0 | xargs -0 sed -i '' -e 's/foo:\/Drive_Letter:/foo:\/bar\/baz\/xyz/g' Not sure if this will be helpful but you can use this with a remote server like the example below
ssh example.server.com «find /DIR_NAME -type f -name «FILES_LOOKING_FOR» -exec sed -i ‘s/LOOKINGFOR/withThisString/g’ <> ;»
replace the example.server.com with your server replace DIR_NAME with your directory/file locations replace FILES_LOOKING_FOR with files you are looking for replace LOOKINGFOR with what you are looking for replace withThisString with what your want to be replaced in the file
If you are to replace a fixed string or some pattern, I would also like to add the bash builtin pattern string replacement variable substitution construct. Instead of describing it myself, I am quoting the section from the bash manual:
$
The pattern is expanded to produce a pattern just as in pathname expansion. parameter is expanded and the longest match of pattern against its value is replaced with string. If pattern begins with / , all matches of pattern are replaced with string. Normally only the first match is replaced. If pattern begins with # , it must match at the beginning of the expanded value of parameter. If pattern begins with % , it must match at the end of the expanded value of parameter. If string is null, matches of pattern are deleted and the / following pattern may be omitted. If parameter is @ or * , the substitution operation is applied to each positional parameter in turn, and the expansion is the resultant list. If parameter is an array variable subscripted with @ or * , the substitution operation is applied to each member of the array in turn, and the expansion is the resultant list.
Find and Replace string in all files recursive using grep and sed [duplicate]
You’ll also have an issue with the / in your patterns. They should be escaped. Or you could use another delimiter for your s command (e.g. @ ).
5 Answers 5
As @Didier said, you can change your delimiter to something other than / :
grep -rl $oldstring /path/to/folder | xargs sed -i s@$oldstring@$newstring@g The above would fail in SO many interesting ways given so many possible contents of oldstring or newstring.
@EdMorton, would you please put an example of how does it fail? I know you posted the traditional form of doing below, but just saying it can fail leave the question of how. Then I could understand why I should your solution. These solution just seems much more easy. Thanks
Try the above with any shell globbing character or an @ in either string variable. The globbing chars can/will match file names in your directory, if not today then some day in future when you last expect it, and the @ will terminate your sed command. It’ll also fail if oldstring contains spaces in various locations, not to mention if either string contains newline characters. The OP specifically said the «oldstring» has special characters so that’s a big heads up that any of what I just described and more is likely to happen.
Also take care that in some environments, notably Darwin/MacOSX you need to give an explicit backup extension like ‘sed -i .bak’.
grep -rl $oldstring . | xargs sed -i "s/$oldstring/$newstring/g" So the complication with this, is files with spaces in their names won’t be picked up. If you had «/files/File Two.txt», sed would try «/files/File» and «Two.txt».
The GNU guys REALLY messed up when they introduced recursive file searching to grep. grep is for finding REs in files and printing the matching line (g/re/p remember?) NOT for finding files. There’s a perfectly good tool with a very obvious name for FINDing files. Whatever happened to the UNIX mantra of do one thing and do it well?
Anyway, here’s how you’d do what you want using the traditional UNIX approach (untested):
find /path/to/folder -type f -print | while IFS= read -r file do awk -v old="$oldstring" -v new="$newstring" ' BEGIN < rlength = length(old) >rstart = index($0,old) < $0 = substr($0,rstart-1) new substr($0,rstart+rlength) > < print >' "$file" > tmp && mv tmp "$file" done Not that by using awk/index() instead of sed and grep you avoid the need to escape all of the RE metacharacters that might appear in either your old or your new string plus figure out a character to use as your sed delimiter that can’t appear in your old or new strings, and that you don’t need to run grep since the replacement will only occur for files that do contain the string you want. Having said all of that, if you don’t want the file timestamp to change if you don’t modify the file, then just do a diff on tmp and the original file before doing the mv or throw in an fgrep -q before the awk.
Caveat: The above won’t work for file names that contain newlines. If you have those then let us know and we can show you how to handle them.
Grep: search and replace full line
prints all matching lines in my file. Now I want to replace the full line with another string. How can I do that?
2 Answers 2
If you’re matching a substring of the whole line, you can either use sed’s s command with a regex to mop up the rest of the line:
sed -i 's/^.*foo.*$/another string/' myfile.txt or use the c command to replace the matched line in one go:
If you don’t want to type multiline commands at the prompt, you can put it in a script instead:
$ cat foo.sed /foo/ < c \ another string >$ sed -i -f foo.sed myfile.txt at the end of the first line, you need to type \ and then hit Return — it’s important that the \ escapes that first newline. Then, make sure the >’ is on a seperate line from another string : the replacement is everything on that second line, including the > if you put it there.
You can use inplace functionnality of sed :
sed -i -e 's/foo/bar/' myfile.txt funny enough, I did multiple searches and this question came up as my top result hence for some reason even though the question is to replace the full line I still believe that this answer helps other lost souls out there. I’d update your comment to clarify and leave the original response.
You must log in to answer this question.
Related
Hot Network Questions
Subscribe to RSS
To subscribe to this RSS feed, copy and paste this URL into your RSS reader.
Site design / logo © 2023 Stack Exchange Inc; user contributions licensed under CC BY-SA . rev 2023.7.14.43533
Linux is a registered trademark of Linus Torvalds. UNIX is a registered trademark of The Open Group.
This site is not affiliated with Linus Torvalds or The Open Group in any way.
By clicking “Accept all cookies”, you agree Stack Exchange can store cookies on your device and disclose information in accordance with our Cookie Policy.