How to deal with spaces in a variable
As said many time on this site, leaving a variable expansion (as in $var ) or command substitution (as in `cmd` or $(cmd) ) (or arithmetic expansion (as in $((11 * 11)) ) in most shells) unquoted in Bourne/POSIX shells is the split+glob operator.
The content of $var or the output of cmd (without the trailing newline characters) is split according to the current value of the $IFS special variable (which by default contains the SPC, TAB, and NL characters (and NUL in zsh )) and each word resulting of that splitting is subject to filename generation also known as globbing.
For instance, if find ouputs ./foo bar.pdf\n./*foo*\tbar.pdf\n ( \t meaning TAB and \n NL), with the default value of $IFS , the command substitution will expand to ./foo bar.pdf\n./*foo*\tbar.pdf (trailing newline removed), and then be split into ./foo , bar.pdf , ./*foo* , and foo.pdf and ./*foo* which is a wildcard pattern will be expanded into as many arguments as there are non-hidden files in the current directory whose name contains foo .
If you want to split on newline characters only, you need to set $IFS to newline only:
If you don’t want the wildcard patterns to be expanded, you need to disable it with
However note that newline is as valid a character as any in a file name, so more generally, the find -print output cannot be post-processed reliably.
either means the a.pdf and b.pdf files in the current directory, or the file called b.pdf in the a.pdf\n. directory.
Some find implementations like GNU find (where it originated from) have a -print0 predicate to output the filename followed by a NUL character instead of a NL character. With standard find , you can use -exec printf ‘%s\0’ <> + with the same result. NUL is the only character that cannot occur in a filename.
However, zsh is the only shell that can store a NUL character in its variables (like the $IFS character), so:
IFS=$'\0' for i in `find . -print0`; do . done (no need for set -f in zsh since zsh doesn’t do globbing upon command substitution) will work in zsh but not in other shells.
Best, and portably, is to have find call the commands you want to run on those files. As @Gnouc suggests:
find . -name '*.pdf' -exec the command <> \; If you need anything more complex involving shell statements, you can still do things like:
find . -name '*.pdf' -exec sh -c ' for i do something complex with "$i" done' sh <> + With zsh or bash , you can also do:
find . -name '*.pdf' -print0 | while IFS= read -r -d '' file; do whatever with "$file" done However note that the stdin within the loop is affected.
zsh (since 1990) has most of the functionality of find included in its globbing capabilities through a syntax where you can specify any level of subdirectories ( (*/)# syntax or its simpler form **/ ) and globbing qualifiers (which are the pendant of the -type f , -mtime , -perm . in find ).
The **/ part of that was copied by ksh93 in 2003, fish in 2005, bash in 2009 and tcsh in 2010 (though tcsh also copied the ***/ part). And all of them do not enable it by default. Unfortunately, note that both bash and fish ** do follow symlinks to directories (like -L / -follow in find , or *** in zsh or tcsh ).
In those shells, you can find pdf files in any level of subdirectories without having to rely on find , but note that caveat above about fish and bash , and only zsh has the globbing qualifiers.
So, for instance, the zsh equivalent of:
find . -name '*.pdf' -type f -exec ls -ld <> + While with bash , you’d have to do something like:
shopt -s failglob shopt -s globstar files=(./**/*.pdf) && for i do [ -f "$i" ] && ! [ -L "$i" ] && set -- "$i" "$@" shift done && ls -ld "$@" Spaces and curly braces for variables in bash
Sure, I can use typical notation like «$VAR» . But that’s cumbersome when using quotes within quoted text, etc. I wonder if there’s a way of expanding $ <. >notation that would treat $ <. >as if it were «$<. >» while not using doublequotes themselves?
There is one notation that works exactly like «$<. >» , and it’s «$<. >» . There is no point in having two notations for the same thing.
Avoiding the double quotes is unlikely to be the right solution to whatever problem you are trying to solve.
2 Answers 2
Not following accepted best practice (and learning how the shell really works) is very likely to bite you. Repeatedly. With zombie virus-infected fangs. Some things which can help:
- You can use different quotes for different parts of a single parameter as long as the start and end quotes are next to each other:
$ printf '%q\n' "foo 'bar' baz"'nuu "boo" zoo' foo\ \'bar\'\ baznuu\ \"boo\"\ zoo $ a="some spaces in there" $ (IFS= && touch $) $ ls -1 some spaces in there You can set the IFS variable to disregard spaces when the shell splits variables. This is also useful when taking in input that may contain spaces in loops.
$ cat /tmp/t.sh IFS="$(printf '\n\t')" A="some spaces in there" touch $ ls -l $ /tmp/t.sh some spaces in there (If you have characters like * in your strings try a set -f to disable globbing (see help set) thanks @glenn jackman. But really, putting a * in a filename is asking for trouble!)
$ cat /tmp/t.sh #!/bin/bash A="some spaces in there" touch $ ls -1 $ /tmp/t.sh in some spaces there $ Bash variables with spaces
This doesn’t work when I use git fetch on the local repository. But if I do it like this (old DOS way), it works:
export GIT_SSH="/c/Progra~1/TortoiseGit/bin/TortoisePlink.exe" My question is: How can I make it work using spaces in the variable? For testing purpose you can simulate something like this (any example is good):
export VAR="/c/Program Files/TortoiseGit/bin/TortoisePlink.exe" # and try to execute like this $VAR What happens when you try /c/Program\ Files/TortoiseGit/bin/TortoisePlink.exe ? That is, include a backslash to escape the space.
@chrisaycock: I was deluded as well, but this would just be a way to avoid the double quotes in the argument of the export command. Afterwards this string is treated in the same way, spaces are still spaces 🙂
5 Answers 5
Execute it like this: «$VAR» . This is one of the most significant gotchas in shell scripting because strings are always substituted literally and any contained spaces are treated as token delimiters rather than as characters of the string. Think of substituting a variable as a kind of code pasting at runtime.
What really happens when you write $VAR is that the shell tries to execute the binary /c/Program with a first argument Files/TortoiseGit/bin/TortoisePlink.exe .
I learned this the hard way by getting a strange syntax error in a big shell script for a particular input. No other languages I can think of can complain for syntax errors if the runtime input contains special characters — but that is the nature of shell scripting since command interpreters like bash and sh interpret the code line by line.
Whenever you expect a string to contain spaces and you don’t want to treat it as separate tokens, enclose it in double quotes.
Setting variables with spaces within .env
I have a .env file with a bunch of variables and I just came across an error. One of the variables has spaces.
env: world"': No such file or directory @glennjackman, @ThomasReggi: This cannot work with the echo example because the substitution of $TEST is done before running echo in the current shell which does not have the TEST variable set. When echo is run, TEST is set, but echo does not know what to do with $TEST .
2 Answers 2
If your command is just a shell command, you could run your command in a subshell like this:
The source or . builtin has no problem with assignments containing spaces. It will set the variables in the .env file in the current shell’s environment.
In the more likely case of calling an external program, you’ll also have to add ‘export’ to each assignment in your env file like this:
This is necessary because source does not export assigned variables as env does, i.e. they are set inside the subshell only but not in the environment of another process started inside that subshell.
I don’t have a test.env file I don’t understand what that is. Let’s say my cmd is just to echo $TEST so from your example ( . .env ; echo $TEST ) does not work it gives me -bash: world: command not found .
«The source or . builtin has no problem with assignments containing spaces» is incorrect. Un-quoted spaces will cause bash to think the word after the space is a command, that’s why OP is getting «command not found». The values after the = need to be quoted if they contain spaces.
Thank you for your feedback. Let me share my thoughts on that: (1) The space in the OP’s example is quoted, so my statement is correct in that context. (2) As I noted in the question comments, just quoting the command substitution would probably be sufficient in most situations, but it would not work with the example provided by glennjackman. (3) The actual problem here is that the quotes read from .env in the command substitution are not interpreted by Bash, then word splitting occurs, and env interprets the second argument world» as a command ( env NAME=value COMMAND ARG . ).