How do I use the lines of a file as arguments of a command?
which I need to pass to the command my_command How do I use the lines of a file as arguments of a command?
«arguments», plural, or «argument», single? The accepted answer is correct only in the single-argument case — which is what the body text inquires about — but not in the multi-argument case, on which the topic appears to inquire.
The 4 answers below usually have identical results, yet they have slightly different semantics. Now I remember why I stopped writing bash scripts 😛
12 Answers 12
If your shell is bash (amongst others), a shortcut for $(cat afile) is $(< afile) , so you'd write:
Documented in the bash man page in the ‘Command Substitution’ section.
Alterately, have your command read from stdin, so: mycommand < file.txt
To be pedantic, it’s not a shortcut to use the < operator. It means that the shell itself will perform the redirection rather than executing the cat binary for it's redirection properties.
@ykaner because without «$(…)» , the contents of file.txt would be passed to the standard input of mycommand , not as an argument. «$(…)» means run the command and give back the output as a string; here the “command” only reads the file but it could be more complex.
It’s not working for me unless I lose the quotes. With the quotes, it takes the whole file as 1 argument. Without quotes, it interprets each line as a separate arg.
@lstyls: I’m not sure what you mean by «it’s not a shortcut.» Having the shell perform the redirection instead of spawning a child cat process sounds like it would save resources. The ref man describes this substitution as «equivalent but faster.» Maybe you mean «it’s not just a shorter way of typing the same thing»?
As already mentioned, you can use the backticks or $(cat filename) .
What was not mentioned, and I think is important to note, is that you must remember that the shell will break apart the contents of that file according to whitespace, giving each «word» it finds to your command as an argument. And while you may be able to enclose a command-line argument in quotes so that it can contain whitespace, escape sequences, etc., reading from the file will not do the same thing. For example, if your file contains:
the arguments you will get are:
If you want to pull each line as an argument, use the while/read/do construct:
while read i ; do command_name $i ; done < filename I should have mentioned, I am assuming that you are using bash. I realize that there are other shells out there, but almost all of the *nix machines I have worked on either ran bash or some equivalent. IIRC, this syntax should work the same on ksh and zsh.
Read should be read -r unless you want to expand backslash-escape sequences -- and NUL is a safer delimiter to use than the newline, particularly if the arguments you're passing are things like filenames, which can contain literal newlines. Also, without clearing IFS, you get leading and trailing whitespace implicitly cleared from i .
will pass file contents to the command on stdin, but will strip newlines, meaning you couldn't iterate over each line individually. For that you could write a script with a 'for' loop:
for line in `cat input_file`; do some_command "$line"; done for line in `cat input_file` do some_command "$line" done Or (multi-line variant with $() instead of `` ):
for line in $(cat input_file) do some_command "$line" done References:
Also, putting < file in backticks means it doesn't actually perform a redirection for command at all.
(Did the people who upvoted this actually test it? command `< file` doesn't work in either bash or POSIX sh; zsh is a different matter, but that's not the shell this question is about).
@CharlesDuffy This does work in bash 3.2 and zsh 5.3. This won't work in sh, ash and dash, but neither $(< file).
@mwfearnley, happy to provide that specificity. The goal is to iterate over lines, right? Put Hello World on one line. You'll see it first run some_command Hello , then some_command World , not some_command "Hello World" or some_command Hello World .
You do that using backticks:
echo World > file.txt echo Hello `cat file.txt` This doesn't create an argument -- because it isn't quoted, it's subject to string-splitting, so if you emitted echo "Hello * Starry * World" > file.txt in the first step, you'd get at least four separate arguments passed to the second command -- and likely more, as the * s would expand to the names of files present in the current directory.
. and because it's running glob expansion, it doesn't emit exactly what's in the file. And because it's performing a command substitution operation, it's extremely inefficient -- it's actually fork() ing off a subshell with a FIFO attached to its stdout, then invoking /bin/cat as a child of that subshell, then reading the output through the FIFO; compare to $( If you want to do this in a robust way that works for every possible command line argument (values with spaces, values with newlines, values with literal quote characters, non-printable values, values with glob characters, etc), it gets a bit more interesting. To write to a file, given an array of arguments: . replace with "argument one" , "argument two" , etc. as appropriate. To read from that file and use its contents (in bash, ksh93, or another recent shell with arrays): To read from that file and use its contents (in a shell without arrays; note that this will overwrite your local command-line argument list, and is thus best done inside of a function, such that you're overwriting the function's arguments and not the global list): Note that -d (allowing a different end-of-line delimiter to be used) is a non-POSIX extension, and a shell without arrays may also not support it. Should that be the case, you may need to use a non-shell language to transform the NUL-delimited content into an eval -safe form: If all you need to do is to turn file arguments.txt with contents into my_command arg1 arg2 argN then you can simply use xargs : You can put additional static arguments in the xargs call, like xargs -a arguments.txt my_command staticArg which will call my_command staticArg arg1 arg2 argN None of the answers seemed to work for me or were too complicated. Luckily, it's not complicated with xargs (Tested on Ubuntu 20.04). This works with each arg on a separate line in the file as the OP mentions and was what I needed as well. One thing to note is that it doesn't seem to work with aliased commands. The accepted answer works if the command accepts multiple args wrapped in a string. In my case using (Neo)Vim it does not and the args are all stuck together. xargs does it properly and actually gives you separate arguments supplied to the command. < foo.txt xargs my_command or xargs < foo.txt my_command , the redirection can be anywhere. You may want to < foo xargs -d \\n cmd or < foo tr \\n \\0 | xargs -0 cmd for BSD Here's how I pass contents of a file as an argument to a command: This is -- but for being substantially less efficient -- effectively equivalent to $( Simply trim the end-line characters and replace them with spaces, and then push the resulting string as possible separate arguments with echo. In my bash shell the following worked like a charm: As evident, this allows you to execute multiple commands with each line from input_file, a nice little trick I learned here. Your "nice little trick" is dangerous from a security perspective -- if you have an argument containing $(somecommand) , you'll get that command executed rather than passed through as text. Likewise, >/etc/passwd will be processed as a redirection and overwrite /etc/passwd (if run with appropriate permissions), etc. It's much safer to do the following instead (on a system with GNU extensions): xargs -d $'\n' sh -c 'for arg; do command1 "$arg"; command2 "arg"; command3 "arg"; done' _ -- and also more efficient, since it passes as many arguments to each shell as possible rather than starting one shell per line in your input file. Both solutions work even when lines have spaces: if readarray doesn't work, replace it with mapfile , they're synonyms. I formerly tried this one below, but had problems when my_command was a script: After editing @Wesley Rice's answer a couple times, I decided my changes were just getting too big to continue changing his answer instead of writing my own. So, I decided I need to write my own! Read each line of a file in and operate on it line-by-line like this: This comes directly from author Vivek Gite here: https://www.cyberciti.biz/faq/unix-howto-read-line-by-line-from-file/. He gets the credit! Syntax: Read file line by line on a Bash Unix & Linux shell: And now to answer this now-closed question which I also had: Is it possible to `git add` a list of files from a file? - here's my answer: Note that FILES_STAGED is a variable containing the absolute path to a file which contains a bunch of lines where each line is a relative path to a file I'd like to do git add on. This code snippet is about to become part of the "eRCaGuy_dotfiles/useful_scripts/sync_git_repo_to_build_machine.sh" file in this project, to enable easy syncing of files in development from one PC (ex: a computer I code on) to another (ex: a more powerful computer I build on): https://github.com/ElectricRCAircraftGuy/eRCaGuy_dotfiles. I have a python script which take's a file as an argument via sys.argv. What is the best way for passing this file as an argument in Linux? I have tried adding this line of code on the 1st line of my python script, thinking that this would enable me to drag and drop files onto it, but this does not seem to work. stackoverflow.com/questions/19978540/… suggests this depends on your desktop environment. (The "best" way is probably mu's suggestion, although that doesn't address drag and drop. ) Thanks mu. this is working. (face palm). Is this line of code necessary then? (#!/usr/bin/env python) and what is it for? This is a pretty basic example that takes the first argument (sys.argv[1]), which we will assume is a filename, opens it, and prints all of the lines. You would call this file by entering python If my source file was called readMyFile.py and I wanted to read myFile.txt, I would type As for accomplishing the drag and drop, you could always put the Python call into a bash script and then pass the argument from the bash script to the python script. Take a look at this stack overflow thread about this. Essentially, you would just make a bash script with this code in it. or in my specific example. It looks like you might have to accompany some of the above syntax with some code in a .desktop file. Check out this askubuntu link Try using python script.py filename The line #!/usr/bin/env python is the shebang line. If you make your script an executable one (that is, change it's file permission to executable using chmod ), the shebang helps identify the type of script. An alternate shebang would be to use the path for python binary directly like #!/usr/bin/python , but the path may differ on different machines and hence we normally get the path for the bindary from the env . The shebang is not normally required if you are ok running the script everytime with python script.py , but if the script is referenced elsewhere, it might cause issues. I add a shebang to all my scripts just to be sure what script it is from within my editor. Note that the the extension .py is entirely optional, and for sake of generality, you might want to run ./script in bigger projects to achieve a task; this modularity ensures that the executable script can be in python, bash, or any other scripting language as long as it does what its supposed to do. No, you can't drag and drop a file and pass it as a param to the python script even if it is an executable one.declare -a args=() while IFS='' read -r -d '' item; do args+=( "$item" ) done " set -- while IFS='' read -r -d '' item; do set -- "$@" "$item" done quoted_list() < ## Works with either Python 2.x or 3.x python -c ' import sys, pipes, shlex quote = pipes.quote if hasattr(pipes, "quote") else shlex.quote print(" ".join([quote(s) for s in sys.stdin.read().split("\0")][:-1])) ' >eval "set -- $(quoted_list xargs -a arguments.txt my_command cat foo.txt | xargs my_command cat input_file | xargs -I % sh -c 'command1 %; command2 %; command3 %;' xargs -d '\n' -a foo.txt my_command #!/bin/bash input="/path/to/txt/file" while IFS= read -r line do echo "$line" done < "$input"
1. The syntax is as follows for bash, ksh, zsh, and all other shells to read a file line by line
2. while read -r line; do COMMAND; done < input.file
3. The -r option passed to read command prevents backslash escapes from being interpreted.
4. Add IFS= option before read command to prevent leading/trailing whitespace from being trimmed -
5. while IFS= read -r line; do COMMAND_on $line; done < input.filewhile IFS= read -r line do echo " git add \"$line\"" git add "$line" done < "$FILES_STAGED" References:
Related:
Pass a file as an argument in Linux
2 Answers 2
import sys f = open(sys.argv[1]) lines = f.readlines() for line in lines: print line f.close() python readMyFile.py myFile.txt #!/bin/bash python readMyFile.py "$1"