In string, character replacement / deletion
Under bash, there are some bashisms:
The tr command could be replaced by $ bashism:
COMMAND=$'\nREBOOT\r \n' echo "|$|" | OOT | echo "|$|" |REBOOT | echo "|$|" |REBOOT| See Parameter Expansion and QUOTING in bash’s man page:
man -Pless\ +/parameter/pattern/string bash man -Pless\ +/\/pattern bash man -Pless\ +/\\\'string\\\' bash man -Pless\ +/^\\\ *Parameter\\\ Exp bash man -Pless\ +/^\\\ *QUOTING bash $ If there are two slashes separating parameter and pattern, all matches of pattern are replaced with string. Further.
As asked by @AlexJordan, this will suppress all specified characters. So what if $COMMAND do contain spaces.
COMMAND=$' \n RE BOOT \r \n' echo "|$COMMAND|" | BOOT | read -r COMMAND " echo "|$COMMAND|" |RE BOOT| Explanation
Why does this work when the pattern is empty?
- The 1st slashe / mean: Pattern substitution
- The pattern is /[$’\r\n ‘] , begin with / then all matches of pattern are replaced with string
- Then, the string is empty.
Avoid fork to tr for single string!
COMMAND=$'\nREBOOT\r \n' echo $ $'\nREBOOT\r \n' COMMAND=$(echo $COMMAND|tr -d '\n\t\r ') echo $ 'REBOOT' time for i in ;do COMMAND=$'\nREBOOT\r \n' COMMAND=$(echo $COMMAND|tr -d '\n\t\r ') done;echo $ real 0m2.785s user 0m2.296s sys 0m0.774s 'REBOOT' COMMAND=$'\nREBOOT\r \n' COMMAND="$" echo $ time for i in ;do COMMAND=$'\nREBOOT\r \n' COMMAND="$" done;echo $ real 0m0.006s user 0m0.001s sys 0m0.004s 'REBOOT' Doing 1’000 forks to tr take more than 2700ms on my host, while same job is done in 6ms ( 464.2x faster!! ) by using built-in bash Parameter Expansion!!
This is the way to strip linefeeds from a variable in BASH. Other answers are needlessly invoking an extra TR process. BTW, it has the added benefit of removing ending spaces!
Note that it also removes internal spaces from a string as well. COMMAND=»RE BOOT»; echo «|$|» returns |REBOOT|
@AlexJordan Yes, it’s a wanted feature: You could whipe the space after \n to prevent this: COMMAND=»RE BOOT»; echo «|$|» will return |RE BOOT| .
How is the pattern working in $
Clean your variable by removing all the linefeeds:
COMMAND=$(echo $COMMAND|tr -d '\n') Echoing an uncommented variable removes all IFS characters (newline, space, tab by default). So if you’re going to do this, you should be aware that all IFS characters are dropped, and you don’t need the tr . Simply COMMAND=$(echo $COMMAND) will give a similar effect. Which is, presumably, devil spawn as it invokes a new process, but it’s still pretty short and sweet for the human’s eyes and if you have a second or two to spare in your life you may be willing to take the hit 🙂 .
I’ve updated the question to say «linefeed», as its example did show that it was a linefeed, not a carriage return. This answer is still correct, but should maybe be updated to say «linefeed» instead of «carriage return»?
@MikeS +1 for noticing the antipattern echo $COMMAND (missing quotes). But it’s an antipattern and shouldn’t be used since it will also expand glob characters if there are any (e.g., * , or [. ] or ? ). So never use an unquoted expansion, unless your variable is in fact a glob that you want to actually expand.
will replace the newline (in POSIX/Unix it’s not a carriage return) with a space.
To be honest I would think about switching away from bash to something more sane though. Or avoiding generating this malformed data in the first place.
Hmmm, this seems like it could be a horrible security hole as well, depending on where the data is coming from.
The date is coming from a curl request which is on the same server, How would i go about putting that into a new var ? newvar=$(echo «|$COMMAND|»|tr ‘\n’ ‘ ‘)
Yes. But please tell me you’re not allowing arbitrary people to reboot your server remotely without a password.
(Note that the control character in this question is a ‘newline’ ( \n ), not a carriage return ( \r ); the latter would have output REBOOT| on a single line.)
Explanation
The pattern is expanded to produce a pattern just as in filename expansion. Parameter is expanded and the longest match of pattern against its value is replaced with string. [. ] If string is null, matches of pattern are deleted and the / following pattern may be omitted.
Also uses the $» ANSI-C quoting construct to specify a newline as $’\n’ . Using a newline directly would work as well, though less pretty:
Full example
#!/bin/bash COMMAND="$'\n'REBOOT" echo "|$|" # Outputs |REBOOT| #!/bin/bash COMMAND=" REBOOT" echo "|$|" # Outputs |REBOOT| Adding answer to show example of stripping multiple characters including \r using tr and using sed. And illustrating using hexdump.
In my case I had found that a command ending with awk print of the last item |awk » in the line included a carriage-return \r as well as quotes.
I used sed ‘s/[«\n\r]//g’ to strip both the carriage-return and quotes.
I could also have used tr -d ‘»\r\n’ .
Interesting to note sed -z is needed if one wishes to remove \n line-feed chars.
$ COMMAND=$'\n"REBOOT"\r \n' $ echo "$COMMAND" |hexdump -C 00000000 0a 22 52 45 42 4f 4f 54 22 0d 20 20 20 0a 0a |."REBOOT". ..| $ echo "$COMMAND" |tr -d '"\r\n' |hexdump -C 00000000 52 45 42 4f 4f 54 20 20 20 |REBOOT | $ echo "$COMMAND" |sed 's/["\n\r]//g' |hexdump -C 00000000 0a 52 45 42 4f 4f 54 20 20 20 0a 0a |.REBOOT ..| $ echo "$COMMAND" |sed -z 's/["\n\r]//g' |hexdump -C 00000000 52 45 42 4f 4f 54 20 20 20 |REBOOT | bash/sed/awk/etc remove every other newline
Are you looking for every other (single) newline at the end of a line, or multiple newlines like \n\n , or both?
8 Answers 8
+1 — works and is elegant. (It puts a tab between the lines — paste -d ‘ ‘ — — will add a space instead if required)
paste is used to concatenate corresponding lines from files: paste file1 file2 file3 . . If one of the «file» arguments is «-«, then lines are read from standard input. If there are 2 «-» arguments, then paste takes 2 lines from stdin. And so on. See the man page.
If the line number is evenly divisible by 2, end with a new line, otherwise, end with a space.
(Tested on: CentOS 6, GNU Awk 3.1.7)
If you want to use sed , there’s no reason to read the whole file into memory. You can merge every other line like this:
Use any character you’d like instead of the space.
Here’s another way using awk:
The if/else handles the case where there is an odd number of lines in the file. Without it, the odd last line gets printed twice. Otherwise, for comparison, you could do:
A late comment: 1) always use a format specifier for printf in case the string has percent signs, 2) to avoid the duplicated last line, set $0 to «» — awk ‘
The most idiomatic way to do it in awk is as follows:
Runtime Name: vmhba2:C0:T3:L14 Group State: active Runtime Name: vmhba3:C0:T0:L14 Group State: active unoptimized Runtime Name: vmhba2:C0:T1:L14 Group State: active unoptimized Runtime Name: vmhba3:C0:T3:L14 Group State: active Runtime Name: vmhba2:C0:T2:L14 Group State: active To explain it we need to define each one of the built-in variables:
- RS record separator. Defaults to \n (new line).
- ORS output record separator. Defaults to \n (new line).
- FS field separator. Defaults to (space).
- NR number of record.
As the default record separator is the new line, a record is, as default, a line.
NR%2 is the modulus of NR/2 , so that it will be either 0 or 1 . 0 for even lines and 1 for odd lines.
var=condition?condition_if_true:condition_if_false is the ternary operator.
All together, saying ORS=NR%2?FS:RS we are defining the output record separator:
- if the number of record is on the form 2k + 1 , that is, on even lines, then the output record separators is set to FS , that is, a space.
- if the number of record is on the form 2k , that is, on odd lines, then the output record separators is set to RS , that is, a new line.
This way, odd lines end with a space, that is then joined with the next line. After that line, a new line is printed.
Remove empty lines in a text file via grep
grep . FILE doesn’t work for me. It’s probably better to stick with grep for searching file contents, and sed for editing file contents.
sed -ne/./p works too, and awk /./ is shorter (action is
For those that don’t understand, the . is a regular expression that matches any character except for newline.
grep . FILE works with the given example, but not necessarily when the file can have bytes not part of the charset. For instance, with GNU grep 2.20, printf «\x80\n» | grep . outputs nothing.
«grep» looks for any line that matches the pattern. «.» matches any character. «grep . FILE» matches any line with at least 1 character. Whereas «grep -v» excludes lines matching the pattern. OP said «remove all the empty new lines». If you want to exclude lines with just spaces, «grep -v ‘^ $'». The «» will match zero or more of the preceding pattern, in this case a space. Though you might prefer to match and exclude other whitespace characters (tabs, form-feeds, etc) too.
This method allowed me to combine multiple excludes more easily than just «grep . FILE». For example, I was looking at a conf file and wanted to exclude all commented lines and all empty lines. So I used «grep -v -e ‘#’ -e ‘^$’ squid.conf». Worked a treat.
this one is a lot faster than the ‘grep . FILE’. This is due to the more complex tasks of verifying the regex ‘.’ than excluding as soon as ^$ does not matches.
grep -v -e ‘^$’ always works, which is not the case of grep . . For instance, with GNU grep 2.20, printf «\x80\n» | grep . outputs nothing, while printf «\x80\n» | grep -v ‘^$’ outputs the non-empty line.
with awk, just check for number of fields. no need regex $ more file hello world foo bar $ awk 'NF' file hello world foo bar its just my good practice to put quotes, since you are running it from shell.. for composite awk statements, you still have to put quotes. so why not cultivate this habit.
Here is a solution that removes all lines that are either blank or contain only space characters:
If removing empty lines means lines including any spaces, use:
$ printf "line1\n\nline2\n \nline3\n\t\nline4\n" > FILE $ cat -v FILE line1 line2 line3 line4 $ grep '\S' FILE line1 line2 line3 line4 $ grep . FILE line1 line2 line3 line4 [root@node1 ~]# cat /etc/sudoers | grep -v -e ^# -e ^$ Defaults !visiblepw Defaults always_set_home Defaults match_group_by_gid Defaults always_query_group_plugin Defaults env_reset Defaults env_keep = "COLORS DISPLAY HOSTNAME HISTSIZE KDEDIR LS_COLORS" Defaults env_keep += "MAIL PS1 PS2 QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE" Defaults env_keep += "LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES" Defaults env_keep += "LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE" Defaults env_keep += "LC_TIME LC_ALL LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY" Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin root ALL=(ALL) ALL %wheel ALL=(ALL) ALL [root@node1 ~]# Try this: sed -i ‘/^[ \t]*$/d’ file-name
It will delete all blank lines having any no. of white spaces (spaces or tabs) i.e. (0 or more) in the file.
Note: there is a ‘space’ followed by ‘\t’ inside the square bracket.
The modifier -i will force to write the updated contents back in the file. Without this flag you can see the empty lines got deleted on the screen but the actual file will not be affected.
THIS IS THE FILE EOF_MYFILE it gives as output only lines with at least 2 characters.
THIS IS THE FILE EOF_MYFILE See also the results with grep ‘^’ my_file outputs
THIS IS THE FILE EOF_MYFILE and also with grep ‘^.’ my_file outputs
THIS IS THE FILE EOF_MYFILE