Remove blank lines with grep
The single quotes around ^$ makes it work for Cshell. Other shells will be happy with either single or double quotes.
UPDATE: This works for me for a file with blank lines or «all white space» (such as windows lines with \r\n style line endings), whereas the above only removes files with blank lines and unix style line endings:
That egrep would only work for files with zero or 1 space on the line, not for files with 2 or more spaces. Change ? to *.
grep -v -e ‘^[[:space:]]*$’ -e ‘^#’ file will give you all non-blank, non-comment lines in a script or config file (or any file type that uses the hash character for comments).
«The -e option allows regex patterns for matching.» That is very misleading. -e is a (POSIX-)definition for: This can be used to specify multiple search patterns, or to protect a pattern beginning with a hyphen (-). (from the manual). Grep already expects a (basic) regular expression by default. For this pattern, you may leave out -e entirely: grep -v ‘^[[:space:]]*$’ foo.txt .
If you’re dealing with files that might have windows-style CR+LF line breaks but don’t want to also exclude lines with other whitespace, then use this regex: ‘^[[:cnrl:]]?$’ .
This works for me on files from a linux based system but not on files from Windows. Presumably because of Windows line-ending characters.
I’m upvoting this even though it doesn’t quite solve the OP’s problem of handling a file with Windows line endings, but since I don’t have that issue, this turned out to be the perfect solution for me.
$ dos2unix file $ grep -v "^$" file If you don’t have dos2unix, then you can use tools like tr:
Good point about converting to UNIX-style line endings otherwise regular expressions may not work as expected. Nothing here worked for me until I converted the line endings.
grep -v "^[[:space:]]*$" The -v makes it print lines that do not completely match ===Each part explained=== ^ match start of line [[:space:]] match whitespace- spaces, tabs, carriage returns, etc. * previous match (whitespace) may exist from 0 to infinite times $ match end of line $ echo " > hello > > ok" | > grep -v "^[[:space:]]*$" hello ok To understand more about how/why this works, I recommend reading up on regular expressions. http://www.regular-expressions.info/tutorial.html
How and why does this work? Your answer would be much better if you could explain. For instance your regular expression matches the beginning of the string then one or more spaces using the POSIX standard then the end of the string, i.e. with grep -v it removes all lines that are only spaces. Right? What happens if there are no spaces; it’s simply a newline character?
As my example shows, even only an empty line is removed (the first line). I added more information, so hopefully that helps. 🙂
If you have sequences of multiple blank lines in a row, and would like only one blank line per sequence, try
grep -v "unwantedThing" foo.txt | cat -s cat -s suppresses repeated empty output lines.
Your output would go from
The three blank lines in the original output would be compressed or «squeezed» into one blank line.
The same as the previous answers:
Here, grep -e means the extended version of grep. ‘^$’ means that there isn’t any character between ^(Start of line) and $(end of line). ‘^’ and ‘$’ are regex characters.
So the command grep -v will print all the lines that do not match this pattern (No characters between ^ and $).
This way, empty blank lines are eliminated.
-e does not mean «the extended version of grep», maybe you are confused with -E ? The manual clearly says that -e just explicitly says that a pattern follows. Since the pattern does not start with a dash, and you are only defining one pattern anyway, you might as well leave it out as by default grep expects one regex pattern: grep -v ‘^$’ foo.txt (no need for extended regex functionality). Also it is worth mentioning that this does not eliminate the blank lines in the file, only that which is piped through the output. For that case, sed -i would be the right tool.
Do lines in the file have whitespace characters?
I prefer using egrep , though in my test with a genuine file with blank line your approach worked fine (though without quotation marks in my test). This worked too:
This code removes blank lines and lines that start with «#»
grep -v "^#" file.txt | grep -v ^[[:space:]]*$ Remove empty lines in a text file via grep
grep . FILE doesn’t work for me. It’s probably better to stick with grep for searching file contents, and sed for editing file contents.
sed -ne/./p works too, and awk /./ is shorter (action is
For those that don’t understand, the . is a regular expression that matches any character except for newline.
grep . FILE works with the given example, but not necessarily when the file can have bytes not part of the charset. For instance, with GNU grep 2.20, printf «\x80\n» | grep . outputs nothing.
«grep» looks for any line that matches the pattern. «.» matches any character. «grep . FILE» matches any line with at least 1 character. Whereas «grep -v» excludes lines matching the pattern. OP said «remove all the empty new lines». If you want to exclude lines with just spaces, «grep -v ‘^ $'». The «» will match zero or more of the preceding pattern, in this case a space. Though you might prefer to match and exclude other whitespace characters (tabs, form-feeds, etc) too.
This method allowed me to combine multiple excludes more easily than just «grep . FILE». For example, I was looking at a conf file and wanted to exclude all commented lines and all empty lines. So I used «grep -v -e ‘#’ -e ‘^$’ squid.conf». Worked a treat.
this one is a lot faster than the ‘grep . FILE’. This is due to the more complex tasks of verifying the regex ‘.’ than excluding as soon as ^$ does not matches.
grep -v -e ‘^$’ always works, which is not the case of grep . . For instance, with GNU grep 2.20, printf «\x80\n» | grep . outputs nothing, while printf «\x80\n» | grep -v ‘^$’ outputs the non-empty line.
with awk, just check for number of fields. no need regex $ more file hello world foo bar $ awk 'NF' file hello world foo bar its just my good practice to put quotes, since you are running it from shell.. for composite awk statements, you still have to put quotes. so why not cultivate this habit.
Here is a solution that removes all lines that are either blank or contain only space characters:
If removing empty lines means lines including any spaces, use:
$ printf "line1\n\nline2\n \nline3\n\t\nline4\n" > FILE $ cat -v FILE line1 line2 line3 line4 $ grep '\S' FILE line1 line2 line3 line4 $ grep . FILE line1 line2 line3 line4 [root@node1 ~]# cat /etc/sudoers | grep -v -e ^# -e ^$ Defaults !visiblepw Defaults always_set_home Defaults match_group_by_gid Defaults always_query_group_plugin Defaults env_reset Defaults env_keep = "COLORS DISPLAY HOSTNAME HISTSIZE KDEDIR LS_COLORS" Defaults env_keep += "MAIL PS1 PS2 QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE" Defaults env_keep += "LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES" Defaults env_keep += "LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE" Defaults env_keep += "LC_TIME LC_ALL LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY" Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin root ALL=(ALL) ALL %wheel ALL=(ALL) ALL [root@node1 ~]# Try this: sed -i ‘/^[ \t]*$/d’ file-name
It will delete all blank lines having any no. of white spaces (spaces or tabs) i.e. (0 or more) in the file.
Note: there is a ‘space’ followed by ‘\t’ inside the square bracket.
The modifier -i will force to write the updated contents back in the file. Without this flag you can see the empty lines got deleted on the screen but the actual file will not be affected.
THIS IS THE FILE EOF_MYFILE it gives as output only lines with at least 2 characters.
THIS IS THE FILE EOF_MYFILE See also the results with grep ‘^’ my_file outputs
THIS IS THE FILE EOF_MYFILE and also with grep ‘^.’ my_file outputs
THIS IS THE FILE EOF_MYFILE Delete empty lines using sed
You may have spaces or tabs in your «empty» line. Use POSIX classes with sed to remove all lines containing only whitespace:
A shorter version that uses ERE, for example with gnu sed:
(Note that sed does NOT support PCRE.)
@BernieReiter ^\s*$ will match all «empty» lines, empty here means, the line contains no chars, or the line contains only empty strings (E.g. spaces). All matched lines will be removed by sed, with the d command.
I am missing the awk solution:
How does this work? Since NF stands for «number of fields», those lines being empty have 0 fields, so that awk evaluates 0 to False and no line is printed; however, if there is at least one field, the evaluation is True and makes awk perform its default action: print the current line.
sed
grep
awk
These show up correctly in your online tool, but [] should not be escaped in a bracket expression, so the code here isn’t correct for \[\[:space:\]\] or \[ \t\] — should be [[:space:]] and [ \t] .
@BenjaminW. Thanks for catching that. Those were not from the original author, but came from Edit 3 when it was changed from regular text to «code», which then «exposed» the `\` escaping. I have fixed them now.
sed ‘/^$/d’ should be fine, are you expecting to modify the file in place? If so you should use the -i flag.
Maybe those lines are not empty, so if that’s the case, look at this question Remove empty lines from txtfiles, remove spaces from start and end of line I believe that’s what you’re trying to achieve.
I believe this is the easiest and fastest one:
If you need to ignore all white-space lines as well then try this:
s="\ \ a\ b\ \ Below is TAB:\ \ Below is space:\ \ c\ \ "; echo "$s" | grep . | wc -l; echo "$s" | grep '\S' | wc -l Yes, I know, but the initial question did not mention whether the source is a file or something else, so the solution is what comes after «|», and before it just an example of a source. Simply to distinguish the solution from the source of lines.
grep ‘\S’ is definitely not portable. If you have grep -P then you can use grep -P ‘\S’ but it’s not supported on all platforms, either.
The downside of grep . compared to the other solutions is that it will highlight all the text in red. The other solutions can preserve the original colors. Compare unbuffer apt search foo | grep . to unbuffer apt search foo | grep -v ^$
@wisbucky grep does not default to color output, but often it’s enable via a shell alias or environment variable. Use grep —color=never . to override.
Another option without sed , awk , perl , etc
strings — print the strings of printable characters in files.
«For each file given, GNU strings prints the printable character sequences that are at least 4 characters long. » so very short lines might give you a surprise if you’re unaware of this. There is a —bytes=min-len option to allow shorter lines.
With help from the accepted answer here and the accepted answer above, I have used:
$ sed 's/^ *//; s/ *$//; /^$/d; /^\s*$/d' file.txt > output.txt `s/^ *//` => left trim `s/ *$//` => right trim `/^$/d` => remove empty line `/^\s*$/d` => delete lines which may contain white space This covers all the bases and works perfectly for my needs. Kudos to the original posters @Kent and @kev
The command you are trying is correct, just use -E flag with it.
-E flag makes sed catch extended regular expressions. More info here
sed -n '/ / p' filename #there is a space between '//' You are most likely seeing the unexpected behavior because your text file was created on Windows, so the end of line sequence is \r\n . You can use dos2unix to convert it to a UNIX style text file before running sed or use
to remove blank lines whether or not the carriage return is there.
Hi, what is the -r flag doing and is it possible to combine it with -i to modify the file directly and avoid printing to screen. In addition, I think that this command would also work as sed -r «/^\r$/d»
This works in awk as well.
awk '!/^$/' file xxxxxx yyyyyy zzzzzz You can do something like that using «grep», too:
My bash -specific answer is to recommend using perl substitution operator with the global pattern g flag for this, as follows:
$ perl -pe s'/^\n|^[\ ]*\n//g' $file xxxxxx yyyyyy zzzzzz This answer illustrates accounting for whether or not the empty lines have spaces in them ( [\ ]* ), as well as using | to separate multiple search terms/fields. Tested on macOS High Sierra and CentOS 6/7.
FYI, the OP’s original code sed ‘/^$/d’ $file works just fine in bash Terminal on macOS High Sierra and CentOS 6/7 Linux at a high-performance supercomputing cluster.