Combining files in linux

Concatenating multiple text files into a single file in Bash

What is the quickest and most pragmatic way to combine all *.txt file in a directory into one large text file? Currently I’m using windows with cygwin so I have access to BASH. Windows shell command would be nice too but I doubt there is one.

12 Answers 12

This appends the output to all.txt

you may run into a problem where it cats all.txt into all.txt. I have this problem with grep sometimes, not sure if cat has the same behavior.

@rmeador yes, that is true, if all.txt already exists you will have this problem. This problem is solved by providing the output file with a different extension, or moving all.txt to a different folder.

Just remember, for all the solutions given so far, the shell decides the order in which the files are concatenated. For Bash, IIRC, that’s alphabetical order. If the order is important, you should either name the files appropriately (01file.txt, 02file.txt, etc. ) or specify each file in the order you want it concatenated.

$ cat file1 file2 file3 file4 file5 file6 > out.txt 

The Windows shell command type can do this:

Type type command also writes file names to stderr, which are not captured by the > redirect operator (but will show up on the console).

Just be aware that if you put the output file in the same directory as the original file it will cause a duplication because it will also combine the new output file twice.

You can use Windows shell copy to concatenate files.

To append files, specify a single file for destination, but multiple files for source (using wildcards or file1+file2+file3 format).

This as the IMHO cleanest solution with basically no side effects that beginners could trip over unfortunately does not get appreciated enough 🙁

Worked pretty well, except at the very end of my file I got a weird SUB special unicode character. Deleting it is pretty easy programmatically but not sure why that happened.

Be careful, because none of these methods work with a large number of files. Personally, I used this line:

for i in $(ls | grep ".txt");do cat $i >> output.txt;done 

EDIT: As someone said in the comments, you can replace $(ls | grep «.txt») with $(ls *.txt)

EDIT: thanks to @gnourf_gnourf expertise, the use of glob is the correct way to iterate over files in a directory. Consequently, blasphemous expressions like $(ls | grep «.txt») must be replaced by *.txt (see the article here).

Good Solution

for i in *.txt;do cat $i >> output.txt;done 

Mandatory ParsingLs link, together with a downvote (and you deserve more than one downvote, because ls | grep is a seriously bad antipattern).

Got an upvote from me because it allows for arbitrary testing/ operations by file name prior to output and it’s quick and easy and good for practice. (In my case I wanted: for i in *; do echo -e «\n$i:\n»; cat $1; done )

find . -type f -name '*.txt' -exec cat <> + >> output.txt 

Since OP says the files are in the same directory, you may need to add -maxdepth 1 to the find command.

This should be the correct answer. It will work properly in a shell script. Here is a similar method if you want output sorted: sort -u —output=»$OUTPUT_FILE» —files0-from=- < <(find "$DIRECTORY_NAME" -maxdepth 1 -type f -name '*.txt' -print0)

This is a very flexible approach relying on all the strengths of the find . My favourite! Surely cat *.txt > all.txt does the job within the same directory (as pointed out above). To me, however, becoming comfortably fluent in using find has been a very good habit. Today they’re all in one folder, tomorrow they have multiple file-endings across nested directory hierarchies. Don’t overcomplicate, but also, do make friends with find . 🙂

Источник

How to Join or Merge Text Files in Linux

The Linux cat command is one of the most versatile tools that can use to create files, view them, and even combine them in the Linux command line.

In this article, we take a detour and explore how you can join two text files in Linux using the cat command, (short for “concatenate”) is one of the most commonly used commands in Linux as well as other UNIX-like operating systems, used to concatenate files and print on the standard output.

It is not only used to view files but can also be used to create files together with the redirection character.

View Contents of File in Linux

Suppose you have three text files: sample1.txt, sample2.txt, and sample.3.txt.

To view the contents of these files without opening them, you can use the cat command as shown (remember to replace sample1.txt , sample2.txt and domains3.txt with the names of the files you wish to combine):

$ cat sample1.txt sample2.txt sample3.txt

This provides the following output, with each line in the output corresponding to the files in order of appearance.

Join Contents of Three Files in Linux

To join the three files into one text file, we will use the output redirection operator (>) to redirect output from all the files to a new file. In this example, we have redirected content from all three files to sample4.txt.

$ cat sample1.txt sample2.txt sample3.txt > sample4.txt

The new file now contains content from all the text files, which you can verify by running the following command.

CAUTION: The sample4.txt file is overwritten if it already exists. Therefore proceed with caution when using the redirection operator.

A better option is to append the content of the files to an already existing file. This prevents the deletion of pre-existing content. To achieve this, use the double redirection operator (>>) followed by the file name of the file you want to append the content.

The previous command can be modified as follows:

$ cat sample1.txt sample2.txt sample3.txt >> sample4.txt

This ensures that the existing file is not overwritten. Instead content from the other files is simply added or appended to it.

Append File Contents to New File in Linux

Alternatively, to append content to the file, simply type the cat command followed by the double redirection operator and then the name of the file. Upon pressing ENTER, type in the content you want to add. Then hit ENTER again and press ctrl + d to save the changes made.

Merge Contents of Files Using Sed Command

Alternatively, you can also use the popular sed (a streamer editor) to join or merge the content of two or more files on the command-line, by using its r flag, which instructs sed to read the file provided as an argument. If there are many files, it reads all of them and displays their content as a combined output.

$ sed r sample1.txt sample1.txt sample3.txt $ sed r sample1.txt sample1.txt sample3.txt > sample4.txt $ cat sample4.txt

That was a short guide on how you can join two or more text files on Linux. Any additional ideas you might have up your sleeve? Do let us know in the comment section.

Источник

Merge Files in the Linux Command Line

Learn various ways of merging multiple files into another file in the Linux command line.

Got two or more files and need to merge them in a single file? The simplest way would be to use the cat command. After all, the cat command’s original purpose is to concatenate files.

Use the cat command to merge files in Linux

Merging two files is simple. You just have to append the filename to the cat command and that’s it:

As you can see, I used the cat command to show the contents of a file and then merged them.

But it won’t save any changes. To save those changes, you have to redirect the file contents to another file.

Remember, the > will override the contents of the file so I would recommend using a new file as the cat will create the file if it doesn’t exist.

So how can you modify the editing file while keeping the previous content intact?

Append changes to the existing file

To modify the existing file, you just have to use >> instead of single > and you’d be good to go.

For example, I will make edit the existing file File_3.txt by appending Hello.txt and Sagar.txt :

As you can see, it added new text in the end line while keeping the old content intact.

Automate the merging files using the loop in Linux

Here, I will be using for loop (till three iterations as I want to merge three files) also you can use > or >> as per your needs.

for i in ; do cat "File_$i.txt" >> NewFile.txt; done

Use the sed command to merge files in Linux (temporary)

There are many times when you want to apply changes only for a specific time and in those cases, you can use sed.

Being a non-interactive way of editing files, the sed utility can be tremendously useful when used in the right way.

And when used with the h flag, it will hold the buffer temporarily:

Wrapping Up

This was my take on how you can merge files using the sed and cat command. And if you have any queries, leave us a comment.

Источник

How to merge all (text) files in a directory into one?

This is technically what cat («concatenate») is supposed to do, even though most people just use it for outputting files to stdout. If you give it multiple filenames it will output them all sequentially, and then you can redirect that into a new file; in the case of all files just use ./* (or /path/to/directory/* if you’re not in the directory already) and your shell will expand it to all the filenames (excluding hidden ones by default).

Make sure you don’t use the csh or tcsh shells for that which expand the glob after opening the merged-file for output, and that merged-file doesn’t exist before hand, or you’ll likely end up with an infinite loop that fills up the filesystem.

The list of files is sorted lexically. If using zsh , you can change the order (to numeric, or by age, size. ) with glob qualifiers.

To include files in sub-directories, use:

find . ! -path ./merged-file -type f -exec cat <> + > merged-file 

Though beware the list of files is not sorted and hidden files are included. -type f here restricts to regular files only as it’s unlikely you’ll want to include other types of files. With GNU find , you can change it to -xtype f to also include symlinks to regular files.

Would do the same ( (-.) achieving the equivalent of -xtype f ) but give you a sorted list and exclude hidden files (add the D qualifier to bring them back). zargs can be used there to work around argument list too long errors.

Источник