Linux script read file line by line

Read a file line by line assigning the value to a variable [duplicate]

Can those questions maybe be merged somehow? Both have some really good answers that highlight different aspects of the problem, the bad answers have in-depth explanations in the comments what’s bad about them, and as of now you cannot really get a whole overview on what to consider, from the answers of one single question from the pair. It would be helpful to have all of it in one spot, rather than splotted over 2 pages.

10 Answers 10

The following reads a file passed as an argument line by line:

while IFS= read -r line; do echo "Text read from file: $line" done < my_filename.txt 

This is the standard form for reading lines from a file in a loop. Explanation:

• IFS= (or IFS='' ) prevents leading/trailing whitespace from being trimmed.
• -r prevents backslash escapes from being interpreted.

Or you can put it in a bash file helper script, example contents:

#!/bin/bash while IFS= read -r line; do echo "Text read from file: $line" done < "$1" 

If the above is saved to a script with filename readfile , it can be run as follows:

chmod +x readfile ./readfile filename.txt 

If the file isn’t a standard POSIX text file (= not terminated by a newline character), the loop can be modified to handle trailing partial lines:

while IFS= read -r line || [[ -n "$line" ]]; do echo "Text read from file: $line" done < "$1" 

Here, || [[ -n $line ]] prevents the last line from being ignored if it doesn't end with a \n (since read returns a non-zero exit code when it encounters EOF).

If the commands inside the loop also read from standard input, the file descriptor used by read can be chanced to something else (avoid the standard file descriptors), e.g.:

while IFS= read -r -u3 line; do echo "Text read from file: $line" done 3< "$1" 

(Non-Bash shells might not know read -u3 ; use read

There is a caveat with this method. If anything inside the while loop is interactive (e.g. reads from stdin), then it will take its input from $1. You will not be given a chance to enter data manually.

Of note - some commands break (as in, they break the loop) this. For example, ssh without the -n flag will effectively cause you to escape the loop. There's probably a good reason for this, but it took my a while to nail down what was causing my code to fail before I discovered this.

@OndraŽižka, that's caused by ffmpeg consuming stdin. Add grumble re: advising a .sh extension. Executables on UNIX don't typically have extensions at all (you don't run ls.elf ), and having a bash shebang (and bash-only tooling such as [[ ]] ) and an extension implying POSIX sh compatibility is internally contradictory.

I encourage you to use the -r flag for read which stands for:

-r Do not treat a backslash character in any special way. Consider each backslash to be part of the input line. 

I am citing from man 1 read .

Another thing is to take a filename as an argument.

#!/usr/bin/bash filename="$1" while read -r line; do name="$line" echo "Name read from file - $name" done < "$filename" 

@TranslucentCloud, if this worked and the accepted answer didn't, I suspect that your shell was sh , not bash ; the extended test command used in the || [[ -n "$line" ]] syntax in the accepted answer is a bashism. That said, that syntax actually has pertinent meaning: It causes the loop to continue for the last line in the input file even if it doesn't have a newline. If you wanted to do that in a POSIX-compliant way, you'd want || [ -n "$line" ] , using [ rather than [[ .

That said, this does still need to be modified to set IFS= for the read to prevent trimming whitespace.

Using the following Bash template should allow you to read one value at a time from a file and process it.

while read name; do # Do what you want to $name done < filename 

@CalculusKnight, it only "worked" because you didn't use sufficiently interesting data to test with. Try content with backslashes, or having a line that contains only * .

@Matthias, assumptions that eventually turn out to be false are one of the largest sources of bugs, both security-impacting and otherwise. The largest data loss event I ever saw was due to a scenario someone assumed would "literally never come up" -- a buffer overflow dumping random memory into a buffer used to name files, causing a script that made assumptions about which names could possibly ever occur to have very, very unfortunate behavior.

@Matthias, . and that's especially true here, since code samples shown at StackOverflow are intended to be used as teaching tools, for folks to reuse the patterns in their own work!

@Matthias, I utterly disagree with the claim that "you should only devise your code for data you expect". Unexpected cases are where your bugs are, where your security vulnerabilities are -- handling them is the difference between slapdash code and robust code. Granted, that handling doesn't need to be fancy -- it can just be "exit with an error" -- but if you have no handling at all, then your behavior in unexpected cases is undefined.

#! /bin/bash cat filename | while read LINE; do echo $LINE done 

Nothing against the other answers, maybe they are more sofisticated, but I upvote this answer because it's simple, readable and is enough for what I need. Note that, for it to work, the text file to be read must end with a blank line (i.e. one needs to press Enter after the last line), otherwise the last line will be ignored. At least that is what happened to me.

And the quoting is broken; and you should not use uppercase variable names because those are reserved for system use.

@AntonioViniciusMenezesMedei, . moreover, I've seen folks sustain financial losses because they assumed these caveats would never matter to them; failed to learn good practices; and then followed the habits they were used to when writing scripts that managed backups of critical billing data. Learning to do things right is important.

Another problem here is that the pipe opens a new subshell, i.e. all variables set inside the loop can't be read after the loop finished.

filename=$1 IFS=$'\n' for next in `cat $filename`; do echo "$next read from $filename" done exit 0 

If you have set IFS differently you will get odd results.

This is a horrible method. Please don't use it unless you want to have problems with globbing that will take place before you realize it!

@MUYBelgium did you try with a file that contains a single * on a line? Anyway, this is an antipattern. Don't read lines with for.

@OndraŽižka, the read approach is the best-practices approach by community consensus. The caveat you mention in your comment is one that applies when your loop runs commands (such as ffmpeg ) that read from stdin, trivially solved by using a non-stdin FD for the loop or redirecting such commands' input. By contrast, working around the globbing bug in your for -loop approach means making (and then needing to reverse) shell-global settings changes.

@OndraŽižka, . moreover, the for loop approach you use here means that all content has to be read in before the loop can start executing at all, making it entirely unusable if you're looping over gigabytes of data even if you have disabled globbing; the while read loop needs to store no more than a single line's data at a time, meaning it can start executing while the subprocess generating content is still running (thus being usable for streaming purposes), and also has bounded memory consumption.

Источник

How to read file line by line in Bash script

How would you write a Bash script that can process a text file one line at a time. First you need a syntax and approach to read the file line by line. The methods for this approach are shown in this tutorial.

Suppose, you have a file named company.txt which contents the company names. This file contains the following content.

Example -1: Reading file content from command line

Suppose, you want to read the file, company.txt, line by line from the command line without ‘cat’ command. Run the following command to do the task. while loop will read each line from the file company.txt in each step and store the content of the line in $line variable which will be printed later.

Example -2: Reading file content using script

Create a bash file and add the following code to read the content of a particular file. Here, an existing filename is stored in $filename variable and $n variable is used to keep the value of the line number of that file. Like previous example, while loop is used to read this file with line number.

#!/bin/bash
filename = 'company.txt'
n = 1
while read line; do
# reading each line
echo "Line No. $n : $line "
n =$ ( ( n+ 1 ) )
done < $filename

Run the following command to execute the script.

Run ‘cat’ command with company.txt file to display the original content of company.txt file.

Example -3: Passing filename from the command line and reading the file

Create a bash file and add the following script. This script will take the filename from the command line argument. First argument value is read by the variable $1 which will contain the filename for reading. If the file exists in the current location then while loop will read the file line by line like previous example and print the file content.

Run the above script with employee.txt file as argument value. The output will show the content of employee.txt file by removing extra space. You can show the original content of employee.txt file by using ‘cat’ command.

Example – 4: Reading file by omitting backslash escape

If you want to read each line of a file by omitting backslash escape then you have to use ‘-r’ option with read command in while loop.

Create a file named company2.txt with backslash and run the following command to execute the script. The output will show the file content without any backslash.

You will need to read the file for many programming purposes. For example, you can search or match any particular content easily from any file by reading each line separately. So, it is an essential task for any programming. Some simple examples of reading file in bash script are shown in this tutorial. These will help you to get the idea of reading file content line by line using while loop in bash script and apply in your script more efficiently. For more information watch the video!

About the author

Fahmida Yesmin

I am a trainer of web programming courses. I like to write article or tutorial on various IT topics. I have a YouTube channel where many types of tutorials based on Ubuntu, Windows, Word, Excel, WordPress, Magento, Laravel etc. are published: Tutorials4u Help.

Источник

Read line by line in Bash script

The best way to do this is to redirect the file into the loop:

# Basic idea. Keep reading for improvements. FILE=test while read CMD; do echo "$CMD" done < "$FILE" 

There are some additional improvements that could be made:

• Add IFS= so that read won't trim leading and trailing whitespace from each line.
• Add -r to read to prevent backslashes from being interpreted as escape sequences.
• Lower-case CMD and FILE . The Bash convention is that only environmental and internal shell variables are uppercase.
• Use printf in place of echo which is safer if $cmd is a string like -n , which echo would interpret as a flag.

file=test while IFS= read -r cmd; do printf '%s\n' "$cmd" done < "$file" 

use

What you have is piping the text "cat test" into the loop.

cat test | \ while read CMD; do echo $CMD done 

xargs is the most flexible solution for splitting output into command arguments.

It is also very human readable and easy to use due to its simple parameterisation.

Format is xargs -n $NUMLINES mycommand .

For example, to echo each individual line in a file /tmp/tmp.txt you'd do:

cat /tmp/tmp.txt | xargs -n 1 echo 

Or to diff each successive pair of files listed as lines in a file of the above name you'd do:

cat /tmp/tmp.txt | xargs -n 2 diff 

The -n 2 instructs xargs to consume and pass as separate arguments two lines of what you've piped into it at a time.

You can tailor xargs to split on delimiters besides carriage return/newline.

Use man xargs and google to find out more about the power of this versatile utility.

Источник