Replace two lines in a file with one new line with bash
Updated the example file. The first line can be a lot of times in the files but it should be replaced only when the second line matches as well.
3 Answers 3
There are a lot of ways to interpret your question. If you are trying to replace lines at a fixed position, eg lines 2 and 3, do:
If you want to replace a matching line and the line following:
To replace the two consecutive lines in which the second line matches a pattern:
For the first, lines are not in a fixed position. For the second, it would be better if I could replace the line matching and the previous, not the following.
For the final example; sed -n ‘$p; N; /pattern/d; P; D’ , obviously pattern is the pattern to match, but which part is the replacement string?
@Andy J There is no replacement string here. This just deletes the two lines. (oops!) To do a replacement, you could do: sed -n ‘$p; N; /pattern/s/.*/repl/; P; D’
Even if this question should be already answered in this thread, I didn’t manage to make the «one-line-one-command» solution work.
perl -pe 's/START.*STOP/replace_string/g' file_to_change seems not to work for me and doesn’t perform a multi-line replace. I had to split it in two different perl scripts, like this:
perl -pe 's/bbb\tbbb\n.*/placeholderstring/g' foo1.txt | perl -pe 's/placeholderstring ccc\tccc/eee\teee/g' Try to see what works best for you.
With the new sample text, the only solution that works is the one by William Pursell
Replace space with new line
How can I replace spaces with new lines on an input like: /path/to/file /path/to/file2 /path/to/file3 /path/to/file4 /path/to/file5 etc. To obtain the following:
/path/to/file /path/to/file2 /path/to/file3 /path/to/file4 /path/to/file5 Note
I’m posting this question to help other users, it was not easy to find a useful answer on UNIX SE until I started to type this question. After that I found the following:
Related question
What produces that text, and what do you want to use it for? If it’s a list of actual pathnames, just replacing spaces by newlines may obviously mangle pathnames that contain embedded spaces, such as /User/myself/VirtualBox VMs/ .
@Kusalananda Gilles has written an excelent answer here (see unix.stackexchange.com/a/105655/49721) explaining why «A space-separated list of file names doesn’t really work: what if one of the file names contained spaces?»
13 Answers 13
Use the tr command
echo "/path/to/file /path/to/file2 /path/to/file3 /path/to/file4 /path/to/file5" \ | tr " " "\n" tr was my first thought as well, although there are so many ways to do it! This is pretty much EXACTLY what tr is for, though!
I think it’s worth noting that the sequence \n represents a newline because. it’s in the documentation. I mean : when writing ‘\n’ (or «\n» ) in the shell, tr and sed do not see an actual newline, they see «a backslash then the letter n». In fact, giving an actual newline (can be obtained with $’\n’ in bash) would work with tr (e.g. tr » » $’\n’ works) but not with sed (e.g. sed $’s/ /\n/g’ does not work)
@YoungFrog sed $’s/ /\\n/g’ . A literal newline has to be escaped with a backslash. But sed ‘y/ /\n/’ would be faster and also more portable.
In this case I would use printf:
printf '%s\n' /path/to/file /path/to/file2 /path/to/file3 /path/to/file4 /path/to/file5 If there are spaces within the one of the paths, you can quote that filepath in order to prevent it from being split on the spaces:
printf '%s\n' /path/to/file '/path/to/file with spaces' /path/to/another/file To transform text in general, tr is your best bet, as covered in an existing answer.
A quick note from the future: thanks for posting this solution, it’s been very reliable for me, compared to using «echo».
The above says to substitute one or more whitespace characters ( \s\+ ) with a newline ( \n ).
The -i parameter stands for changing the file «in place».
This is more or less «substitute /one space or more/ for /newline/ globally».
If you have multiple spaces between text then you will get blank lines. You need to add a ‘+’ after the \s to indicate that you want to match at least one spaces as apposed to exactly one space. ie. sed ‘s/\s+/\n/g’
@Timo no, every space will be translated into a linebreak, you could use ` /` as the break point and then inject the / again but that’ll be buggy with some edge cases like /path/with space before slash /and then more .
Assuming you have a string with spaces as separators:
However, you’re probably asking the wrong question. (Not necessarily, for example this might come up in a makefile.) A space-separated list of file names does not really work: what if one of the file names contained spaces?
If a program receives file names as arguments, don’t join them with spaces. Use «$@» to access them one by one. Although echo «$@» prints the arguments with spaces in between, that’s due to echo : it prints its arguments with spaces as separators. somecommand «$@» passes the file names as separate arguments to the command. If you want to print the arguments on separate lines, you can use
If you do have space-separated file names and you want to put them in an array to work on them, you can use an unquoted variable expansion to split the value at characters on IFS (you’ll need to disable wildcard expansion with set -f , otherwise glob patterns will be expanded in the value):
space_separated_list='/path/to/file1 /path/to/file2 /path/to/file3' IFS=' '; set -f eval "array=(\$space_separated_list)" for x in "$"; do … You can encapsulate this in a function that restores the -f setting and the value of IFS when it is done:
split_list () < local IFS=' ' flags='+f' if [[ $- = *f* ]]; then flags=; fi set -f eval "$1=(\$2)" set $flags >split_list array '/path/to/file1 /path/to/file2 /path/to/file3' for x in "$"; do … How to replace a word with new line
What you attempted doesn’t work because sed uses basic regular expressions, and your sed implementation has a \| operator meaning “or” (a common extension to BRE), so what you wrote replaces (empty string or END or empty string) by a newline.
@Baazigar No, what A.B. wrote is correct (for Linux at least, some sed implementations would emit \n ). The question asks how to replace |END| by a newline, not by \n .
The characters for newline are ‘\n’. The \\n is needed because \ also is an escape character, so if you do only \n, you are saying ‘escape this n character’. When you do \\n you are saying ‘don’t treat this next \ as an escape.’.
The following worked fine for me:
$ sed 's/|END|/\ /g' foobar T|somthing|something T|something2|something2 Notice that I just put a backslash followed by the enter key.
That’s the standard syntax. Using \n as in @A.B.’s answer wouldn’t work with some sed implementations.
@StéphaneChazelas What sed implementation supports \| for alternation in a regexp but not \n meaning newline in an s replacement?
$ awk -F'\\|END\\|' '1' OFS='\n' file T|somthing|something T|something2|something2 - -F’\\|END\\|’ set field separator to |END|
- OFS=’\n’ set ouput field separator to newline
- $1=$1 cause awk reconstruct $0 with OFS as field separator
- 1 is a true value, cause awk print the whole input line
Another possibly awk command and using its RS option would be:
Will print those records (based on awk’s Record Separator) which are not empty( has at least one field) to prevent printing empty lines.
T|somthing|something|END|T|something2|something2|END| Test|END| |END| T|somthing|something T|something2|something2 Test That cleared all empty lines 🙂 If you want to have newlines too, replace $1=$1 with $0 in command:
$1=$1 condenses sequences of blanks into one space character and returns false if the first field is 0. Doesn’t make sense. You probably want awk 1 RS=’\\|END\\|’ or awk NF RS=’\\|END\\|’ or awk length RS=’\\|END\\|’ here. Note that a regexp RS requires gawk or mawk
Another way with sed that doesn’t print empty lines:
T|one|two|END|T|three|four|END| T|five|six|END|T|seven|eight|END| T|nine|ten|END|T|eleven|twelve|END| T|one|two T|three|four T|five|six T|seven|eight T|nine|ten T|eleven|twelve As mentioned here by Walter Mundt, we can achieve this using an ANSI C quoted string
sed $'s/|END|/\\\n/g' ~ $ echo 'T|somthing|something|END|T|something2|something2|END|' | sed $'s/|END|/\\\n/g' T|somthing|something T|something2|something2 ~ $ Check out the above here link for other alternatives.
You can use the following syntax as well, I’m not sure if it works on all flavours of Unix/Linux
sed 's/|END|/\'$'\n''/g' ~ $ echo 'T|somthing|something|END|T|something2|something2|END|' | sed 's/|END|/\'$'\n''/g' T|somthing|something T|something2|something2 ~ $ How do I replace a string with a newline using a bash script and sed?
In my bash script I would like to replace the @@ with a newline. I have tried various things with sed but I’m not having any luck:
Ultimately I need to parse this whole input into rows and values. Maybe I am going about it wrong. I was planning to replace the @@ with newlines and then loop through the input with setting IFS=’|’ to split up the values. If there is a better way please tell me, I am still a beginner with shell scripting.
7 Answers 7
Using pure BASH string manipulation:
eol=$'\n' line="$" echo "$line" Value1|Value2|Value3|Value4 Value5|Value6|Value7|Value8 Value9|etc. Or else do it in single step:
Variation that doesn’t require the separate eol definition is to replace $eol with a literal newline (as in, hit Enter, and the final >» appears at the beginning of the next line). Not as clear, but useful for code golf. The » around it are mandatory though, even if nothing else in the string would require them (since the newlines will get turned back into spaces in most [all?] contexts if it’s not quoted).
Another way to avoid defining eol would be to move $’. ‘ into the curly brackets: line=»$
I recommend using the tr function
[itzhaki@local ~]$ X="Value1|Value2|Value3|Value4@@ Value5|Value6|Value7|Value8@@" [itzhaki@local ~]$ X=`echo "$X" | tr '@@' '\n'` [itzhaki@local ~]$ echo "$X" Value1|Value2|Value3|Value4 Value5|Value6|Value7|Value8 This did not work in my script, it still has the ‘@@ ‘ in the output: echo $x | tr ‘@@ ‘ ‘\n’ >> $OUTPUT
I know for a fact it works. I’m using «tr» all the time. Make sure you’re using BASH and not SH, and that you’re looking at the right output. Here, run this: pastebin.com/gxvKUwS9
As far as I know, tr works on individual characters, not strings. As in, tr ‘Ab’ ‘aB’ turns Abracadabra into aBracadaBra .
Finally got it working with:
Adding the single quotes around \\n seemed to help for whatever reason
If you don’t mind to use perl:
echo $line | perl -pe 's/@@/\n/g' Value1|Value2|Value3|Value4 Value5|Value6|Value7|Value8 Value9|etc for line in `echo $longline | sed 's/@@/\n/g'` ; do $operation1 $line $operation2 $line . $operationN $line for field in `echo $each | sed 's/|/\n/g'` ; do $operationF1 $field $operationF2 $field . $operationFN $field done done Edited answer to make solution compatible with performing operations on both rows and fields in those rows.
This wraps up using perl to do it, and gives some simple help.
$ echo "hi\nthere" hi there $ echo "hi\nthere" | replace_string.sh e hi th re $ echo "hi\nthere" | replace_string.sh hi there $ echo "hi\nthere" | replace_string.sh hi bye bye there $ echo "hi\nthere" | replace_string.sh e super all hi thsuperrsuper replace_string.sh
#!/bin/bash ME=$(basename $0) function show_help() < IT=$(cat e.g. $ME : -> replaces first instance of ":" with a new line $ME : b -> replaces first instance of ":" with "b" $ME a b all -> replaces ALL instances of "a" with "b" ) echo "$IT" exit > if [ "$1" == "help" ] then show_help fi if [ -z "$1" ] then show_help fi STRING="$1" TIMES=$ WITH=$ if [ "$TIMES" == "all" ] then TIMES="g" else TIMES="" fi perl -pe "s/$STRING/$WITH/$TIMES"