Find and Replace Inside a Text File from a Bash Command
What’s the simplest way to do a find and replace for a given input string, say abc , and replace with another string, say XYZ in file /tmp/file.txt ? I am writting an app and using IronPython to execute commands through SSH — but I don’t know Unix that well and don’t know what to look for. I have heard that Bash, apart from being a command line interface, can be a very powerful scripting language. So, if this is true, I assume you can perform actions like these. Can I do it with bash, and what’s the simplest (one line) script to achieve my goal?
18 Answers 18
The easiest way is to use sed (or perl):
sed -i -e 's/abc/XYZ/g' /tmp/file.txt Which will invoke sed to do an in-place edit due to the -i option. This can be called from bash.
If you really really want to use just bash, then the following can work:
while IFS='' read -r a; do echo "$" done < /tmp/file.txt >/tmp/file.txt.t mv /tmp/file.txt
This loops over each line, doing a substitution, and writing to a temporary file (don’t want to clobber the input). The move at the end just moves temporary to the original name. (For robustness and security, the temporary file name should not be static or predictable, but let’s not go there.)
For Mac users:
sed -i '' 's/abc/XYZ/g' /tmp/file.txt (See the comment below why)
Except that invoking mv is pretty much as ‘non Bash’ as using sed. I nearly said the same of echo, but it’s a shell builtin.
The -i argument for sed doesn’t exist for Solaris (and I would think some other implementations) however, so keep that in mind. Just spent several minutes figuring that out.
Note for Mac users who get an invalid command code C error. For in-place replacements, BSD sed requires a file extension after the -i flag because it saves a backup file with the given extension. For example: sed -i ‘.bak’ ‘s/find/replace/’ /file.txt You can skip the backup by using an empty string like so: sed -i » ‘s/find/replace/’ /file.txt
File manipulation isn’t normally done by Bash, but by programs invoked by Bash, e.g.:
perl -pi -e 's/abc/XYZ/g' /tmp/file.txt The -i flag tells it to do an in-place replacement.
See man perlrun for more details, including how to take a backup of the original file.
The purist in me says you can’t be sure Perl will be available on the system. But that’s very seldom the case nowadays. Perhaps I’m showing my age.
Can you show a more complex example. Something like replacing «chdir /blah» with «chdir /blah2». I tried perl -pi -e ‘s/chdir (?:\\/[\\w\\.\\-]+)+/chdir blah/g’ text , but I keep getting an error with Having no space between pattern and following word is deprecated at -e line 1. Unmatched ( in regex; marked by
This is an old post but for anyone wanting to use variables as @centurian said the single quotes mean nothing will be expanded.
A simple way to get variables in is to do string concatenation since this is done by juxtaposition in bash the following should work:
sed -i -e "s/$var1/$var2/g" /tmp/file.txt Problem solved, well, . $li comes from a file line, so there is e.g. \n and the error is there. So either awk or another language like python comes.
I was surprised when I stumbled over this.
There is a replace command which ships with the «mysql-server» package, so if you have installed it try it out:
# replace string abc to XYZ in files replace "abc" "XYZ" -- file.txt file2.txt file3.txt # or pipe an echo to replace echo "abcdef" |replace "abc" "XYZ" See man replace for more on this.
Two things are possible here: a) replace is a useful independent tool and the MySQL folks should release it separately and depend on it b) replace requires some bit of MySQL o_O Either way, installing mysql-server to get replace would be the wrong thing to do 🙂
That’s because you don’t have mysql-server package installed. As pointed by @rayro, replace is part of it.
Be careful not to run the REPLACE command on Windows! On Windows the REPLACE command is for a fast replication of files. Not relevant to this discussion.
Bash, like other shells, is just a tool for coordinating other commands. Typically you would try to use standard UNIX commands, but you can of course use Bash to invoke anything, including your own compiled programs, other shell scripts, Python and Perl scripts etc.
In this case, there are a couple of ways to do it.
If you want to read a file, and write it to another file, doing search/replace as you go, use sed:
If you want to edit the file in place (as if opening the file in an editor, editing it, then saving it) supply instructions to the line editor ‘ex’
echo "%s/abc/XYZ/g w q " | ex file Example is like vi without the fullscreen mode. You can give it the same commands you would at vi ‘s : prompt.
Not as far as I know. for f in report*.txt; do echo «%s/abc/XYZ/g \n w \n q \n» | ex file; done is clean and simple. Why put functionality into ex that the shell already has?
I found this thread among others and I agree it contains the most complete answers so I’m adding mine too:
- sed and ed are so useful. by hand. Look at this code from @Johnny:
sed -i -e 's/abc/XYZ/g' /tmp/file.txt x='abc' y='XYZ' sed -i -e 's/$x/$y/g' /tmp/file.txt #or, sed -i -e "s/$x/$y/g" /tmp/file.txt but, what can we do? As, @Johnny said use a while read. but, unfortunately that’s not the end of the story. The following worked well with me:
#edit user's virtual domain result= #if nullglob is set then, unset it temporarily is_nullglob=$( shopt -s | egrep -i '*nullglob' ) if [[ is_nullglob ]]; then shopt -u nullglob fi while IFS= read -r line; do line="$'/$server>" line="$'/$alias>" line="$'/$user>" line="$'/$group>" result="$result""$line"'\n' done < $tmp echo -e $result >$tmp #if nullglob was set then, re-enable it if [[ is_nullglob ]]; then shopt -s nullglob fi #move user's virtual domain to Apache 2 domain directory . The most suitable solution I can think of with the given assumptions of the problem.
How to replace all the contents from one folder with another one
I have folders old and new . I want to replace all the contents in old/* to new/* . There can be many subdirectories in those folders. But there can be few files which are not in new/* but are in old/* , so I want them to stay as they are. How can I do that from a Linux shell?
3 Answers 3
rsync would probably be a better option here. It’s as simple as rsync -a subdir/ ./. check this unix.stackexchage answer for better solutions
use -f with the cp command
suppress cp to overwrite» prompt..
To override cp’s alias you can simply enclose it in quotes:
for more information follow these links:
Don’t use rsync for a one time copy, the overhead of hashing all dir contents is substantial, it only pays when keeping directories synchronized. For a gigantic one time copy, use a tarpipe.
Use rsync . It will synchronize the directories in one direction. So, if you want to update your old folder with everything from new , but keep what’s in there, just use:
rsync -avh --dry-run /path/to/new/ /path/to/old/ This will, in a first instance, just output the list of files that would be transferred. In that case: Everything found in new will be copied to old , unless it’s already there. Everything in old stays as it is.
If it looks fine to you, remove the —dry-run argument to transmit them for real.
The -avh flags just enable archive mode (which will preserve timestamps, etc.), verbosity and human-readable file-sizes. Nothing will be deleted from the destination unless you specify the —delete flag. Consult man rsync for more information.
replace a whole file with another file in bash
Ive learned how to replace a line using bash script but I am wanting to learn how to replace a whole file with another file in a different folder with the same name. Is this possible??
4 Answers 4
cp -f [original file] [new file] Copies the original file and overwrites the target file (hence -f which stands for «force»).
Looks like you switched up the arguments, It should be the other way around. cp -f [new file] [original file]
In case you are attempting to copy just the content of the file try
cat /first/file/same_name > /second/file/same_name This will overwrite all the content of the second file with the content from the first. However, your owner, group, and permissions of the second file will be unchanged.
«overwrite content» and «copy» are kinda different operations if the destination file is a symlink 🙂
This is exactly what I needed to deal with hardlinked files updated in a way the caused the link to be broken (saved as a separate file.)
If file2 does not exist, then file1 is renamed file2. If file2 exists, its contents are replaced with the contents of file1.
Like above however, since the «-i» (interactive) option is specified, if file2 exists, the user is prompted before it is overwritten with the contents of file1.
mv file1 file2 file3 dir1 The files file1, file2, file3 are moved to directory dir1. dir1 must exist or mv will exit with an error.
If dir2 does not exist, then dir1 is renamed dir2. If dir2 exists, the directory dir1 is created within directory dir2.