Linux for file in loop

• /Data/*.txt expands to the paths of the text files in /Data (including the /Data/ part)
• $( . ) runs a shell command and inserts its output at that point in the command line
• basename somepath .txt outputs the base part of somepath, with .txt removed from the end (e.g. /Data/file.txt -> file )

If you needed to run MyProgram with Data/file.txt instead of /Data/file.txt , use "$" to remove the leading slash. On the other hand, if it's really Data not /Data you want to scan, just use for filename in Data/*.txt .

Источник

How To Linux Bash For Loop In Files?

Bash provides a lot of useful programming functionalities. for loop is one of the most useful of them. We can use for loop for iterative jobs. Linux system administrators generally use for loop to iterate over files and folder. In this tutorial, we will look at how to use for loop to iterate over files and directories in Linux. This example can be used any of Linux distribution which uses bash as shell-like Ubuntu, CentOS, RedHat, Fedora, Debian, Kali, Mint, etc. This mechanism is named as for each some programming languages where a list is iterated over.

For Loop Syntax

Bash provides different usages and syntax for the for loop but in general following syntax is used .

for F in ITEM1 ITEM2 . ; do CODE done

• for is the keyword which is used to create loops in bash
• F is the element or item which is populated in each step from ITEMS
• ITEM1 , ITEM2 , ITEM3 , … are items we want to iterate this can be also some list which contains multiple items
• CODE is the implementation part which is executed in each loop

Numeric Syntax

One of the most used Numeric syntax. We will provide numbers as a list and iterate over given list. Every number in the list will be iterated inside the for loop one by one. Every item will be assigned to the variable named $VAR like below and this variable named $VAR can be used inside the for loop book.

for VAR in 1 2 3 4 .. N do command1 $VAR command2 command3 . commandN done

In numeric syntax, it is very similar to the general syntax where we provide a collection of numbers.

Given File List Syntax

We will provide the files as a list and use them in each iteration. In this type, we will iterate over the list of files like file1 , file2 , file3 etc.

for VAR in file1 file2 file3 file3 . filen do command1 $VAR command2 command3 . commandN done

Command Output Syntax

We can use bash commands output as items for iterate. In this syntax, we expect that the $(BASH_COMMAND) will return a list where we will iterate over this list.

for VAR in $(BASH_COMMAND) do command1 $VAR command2 command3 . commandN done

Loop Over Given File Names

The simplest usage for for loop is over given file names. We will provide the file files by separating them with spaces. In this example, we will provide file names a , b and c and then print them with some string.

for f in "a" "b" "c"; do echo Processing $f ; done;

Loop Over Listed File Names

What can we do if there are thousands of files to be a loop in a directory. We need a more dynamic way to provide file names. We can use ls command in order to provide file names as a list without typing one by one.

for f in $(ls); do echo Processing $f ; done;

Loop Over Specified File Extensions

Some times we may need to work on specific file extensions. We can specify the file extension we want to loop with for loop. In this example, we will print encoding types of files with *.txt extension.

for f in *.txt; do file $f ; done;

Loop Over Files Reading From Text File

Files names can be stored in a text file line by line. We can read file names from the specified text file and use in a for loop. In this example, we will read the following text file and loop over lines. Our file name is filenames.txt

$ for f in $(cat filenames.txt); do echo Processing $f ; done;

C Like For Loop

Up to now, we have learned the basic syntax of for loop. There is also more formal for loop which is the same as C Programming language. We need to provide start condition, iterate operation and end condition.

for ((START_CONDITION;ITERATE_OPERATION;END_CONDITION)) do COMMAND1 COMMAND2 . COMMANDN done

In this example, we will use echo command to print from 1 to 5 with this for loop syntax.

Infinite Loop

In some cases, we may need infinite loops. The infinite loop will never end except its process is killed. We will use C like for loop in order to create an infinite loop.

#!/bin/bash for (( ; ; )) do echo "This will run forever" done

Conditional Break with exit

During for loop, we may need to exit for given conditions if they occur. exit keyword can be used to break the iteration. In this example, we will check and if the current value of c can be divided into 6 we will end the loop.

The terminal will be closed when the variable $c is equal to the 6 where the if will be true and exit statement will be executed.

Skip To Next Step with continue

In some cases, we may need to skip the current iteration but resume to the loop. We can use continue keyword which will step over to the next iteration. In this example, we will continue if the $c variable can be divided with 3.

Источник

How to loop over the lines of a file?

Using quotes for i in "$(cat $1)"; results in i being assigned the whole file at once. What should I change?

6 Answers 6

#!/bin/bash IFS=$'\n' # make newlines the only separator set -f # disable globbing for i in $(cat < "$1"); do echo "tester: $i" done 

Note however that it will skip empty lines as newline being an IFS-white-space character, sequences of it count as 1 and the leading and trailing ones are ignored. With zsh and ksh93 (not bash ), you can change it to IFS=$'\n\n' for newline not to be treated specially, however note that all trailing newline characters (so that includes trailing empty lines) will always be removed by the command substitution.

#!/bin/bash while IFS= read -r line; do echo "tester: $line" done < "$1" 

There, empty lines are preserved, but note that it would skip the last line if it was not properly delimited by a newline character.

I see IFS \ read -r line' in second example. Is really IFS=` needed ? IMHO it enough to say : while read -r line; do echo "tester: $line"; done < "$1"

@GrzegorzWierzowiecki IFS= turns off the stripping of leading and trailing whitespace. See In while IFS= read.. , why does IFS have no effect?

@BenMares To prevent globbing expressions possibly appearing in the text we are reading from being expanded to matching file names. Try, for instance, printf '%s\n' '*o' 'bar' >afile; touch foo; IFS=$'\n'; for i in $(cat afile); do echo "$i"; done .

A while IFS= read -r line || [ "$line" ]; do will process a trailing line not properly delimited by a newline character (but it will be added back).

(9 years later:)
Both provided answers would fail on files without a newline at the end, this will effectively skip the last line, produce no errors, would lead to disaster (learned hard way:).

The best concise solution I found so far that "Just Works" (in both bash and sh):

while IFS='' read -r LINE || [ -n "$" ]; do echo "processing line: $" done < /path/to/input/file.txt 

Beware: this approach adds an additional newline to the last line if there is none already.

files with characters after the last newline are not text files, and those characters don't constitute a line. In many cases those bogus characters are better left ignored or removed, though there are cases where you may want to treat it as a extra line, so it's good you show how.

Note that files with NUL characters in them are also not text files, and except in zsh that loop would also fail if the intention was to keep them as if they were allowed in text files.

If you can avoid it, don't especially if it's to process text.

Most text utilities are already designed to process text one line at a time, and, at least for the GNU implementations, do it efficiently, correctly and handle error conditions nicely. Piping one to another which runs them in parallel also means you can leverage more than one processor to do the job.

If it's not about text processing and you do need to run some command per line of a file, also note GNU xargs where you can do:

xargs -rd'\n' -I@ -a input.txt cp -- @ @.back 

With the bash shell, you can get each line of a file into an array with the readarray builtin:

readarray -t lines < input.txt && for line in "$"; do do-some-non-text-processing-you-cannot-easily-do-with-xargs "$line" || break done 

POSIXly, you can use IFS= read -r line to read one line off some input, but beware that if you redirect the whole while read loop with the input file on stdin, then commands inside the loop will also have their stdin redirected to the file, so best is to use a different fd which you close inside the loop:

while IFS= read -r line 3 output.txt 

read -r line removes leading and trailing whitespace characters from the line that it reads provided they are in the $IFS variable, though only the yash shell honours that POSIX requirement. With most shells, that's limited to space and tab. ksh93 and recent versions of bash do it for all single-byte characters considered as whitespace in the locale.

So to read a line and also strip leading and trailing blanks, you can do: IFS=$' \t' read -r line . With ksh93, yash¹ or recent versions of bash . IFS=$' \t\r' would also strip the trailing CR character found in text files from the Microsoft world.

¹ though yash doesn't support the $'. ' syntax yet, you'd need IFS=$(printf ' \t\r') there.

Источник