How can I use a file in a command and redirect output to the same file without truncating it?
Basically I want to take as input text from a file, remove a line from that file, and send the output back to the same file. Something along these lines if that makes it any clearer.
grep -v 'seg8\\.8\' file_name > file_name Several answers here are duplicates, and several deleted answers propose adding a pipe, like grep ‘moo’ file | cat >file which of course doesn’t help at all. Please review existing answers before adding a new one, and please test any new solution before proposing it.
14 Answers 14
Use sponge for this kind of tasks. Its part of moreutils.
grep -v 'seg4\\.6\' file_name | sponge file_name Thanks for the answer. As a possibly helpful addition, if you’re using homebrew on Mac, can use brew install moreutils .
Word of caution, «sponge» is destructive, so if you have an error in your command, you can wipe out your input file (as I did the first time trying sponge). Make sure your command works, and/or the input file is under version control if you are trying to iterate on making the command work.
You cannot do that because bash processes the redirections first, then executes the command. So by the time grep looks at file_name, it is already empty. You can use a temporary file though.
#!/bin/sh tmpfile=$(mktemp) grep -v 'seg7\\.1\' file_name > $ cat $ > file_name rm -f $
like that, consider using mktemp to create the tmpfile but note that it’s not POSIX.
The reason why you can’t do that: bash processes the redirections first, then executes the command. So by the time grep looks at file_name, it is already empty.
@glennjackman: by «processes redirection you mean that in the case of > it opens the file and clears it and in the case of >> it only opens it» ?
yes, but of note in this situation, the > redirection will open the file and truncate it before the shell launches grep .
It’s perfectly possible do it with redirections, you just have to remove the file before writing to it.
On *BSD (and hence also OSX) you can say -i » so the extension is not strictly mandatory, but the -i option does require some argument.
grep -v 'seg1\\.7\' file_name | tee file_name Your file will not be blank this time 🙂 and your output is also printed to your terminal.
I like this solution! And if you don’t want it to be printed in the terminal you can still redirect the output to /dev/null or similar places.
You can’t use redirection operator ( > or >> ) to the same file, because it has a higher precedence and it will create/truncate the file before the command is even invoked. To avoid that, you should use appropriate tools such as tee , sponge , sed -i or any other tool which can write results to the file (e.g. sort file -o file ).
Basically redirecting input to the same original file doesn’t make sense and you should use appropriate in-place editors for that, for example Ex editor (part of Vim):
ex '+g/seg2\\.7\/d' -scwq file_name - ‘+cmd’ / -c — run any Ex/Vim command
- g/pattern/d — remove lines matching a pattern using global ( help :g )
- -s — silent mode ( man ex )
- -c wq — execute :write and :quit commands
You may use sed to achieve the same (as already shown in other answers), however in-place ( -i ) is non-standard FreeBSD extension (may work differently between Unix/Linux) and basically it’s a stream editor, not a file editor. See: Does Ex mode have any practical use?
Since this question is the top result in search engines, here’s a one-liner based on https://serverfault.com/a/547331 that uses a subshell instead of sponge (which often isn’t part of a vanilla install like OS X):
echo "$(grep -v 'seg2\\.5\' file_name)" > file_name echo "$(cat file_name)" > file_name Edit, the above solution has some caveats:
- printf ‘%s’ should be used instead of echo so that files containing -n don’t cause undesired behavior.
- Command substitution strips trailing newlines (this is a bug/feature of shells like bash) so we should append a postfix character like x to the output and remove it on the outside via parameter expansion of a temporary variable like $ .
- Using a temporary variable $v stomps the value of any existing variable $v in the current shell environment, so we should nest the entire expression in parentheses to preserve the previous value.
- Another bug/feature of shells like bash is that command substitution strips unprintable characters like null from the output. I verified this by calling dd if=/dev/zero bs=1 count=1 >> file_name and viewing it in hex with cat file_name | xxd -p . But echo $(cat file_name) | xxd -p is stripped. So this answer should not be used on binary files or anything using unprintable characters, as Lynch pointed out.
The general solution (albiet slightly slower, more memory intensive and still stripping unprintable characters) is:
(v=$(cat file_name; printf x); printf '%s' $ > file_name) printf "hello\nworld\n" > file_uniquely_named.txt && for ((i=0; i > file_uniquely_named.txt); done; cat file_uniquely_named.txt; rm file_uniquely_named.txt Whereas calling cat file_uniquely_named.txt > file_uniquely_named.txt in the current shell:
printf "hello\nworld\n" > file_uniquely_named.txt && for ((i=0; i file_uniquely_named.txt; done; cat file_uniquely_named.txt; rm file_uniquely_named.txt I haven’t tested this on large files (probably over 2 or 4 GB).
I have borrowed this answer from Hart Simha and kos.
Of course it will not work with large file. This can’t possibly be a good solution or work all the time. What is happening is that bash execute first the command and then load the stdout of cat and put it as first argument to echo . Of course non printable variables will not output properly and corrupt the data. Don’t try to redirect a file back to itself, it just can’t be good.
Here is a newer/better command that takes the place of sponge and is cross-platform if your shell has perl installed: stackoverflow.com/a/69212059/539149 cat file_name.txt | grep -v ‘seg7\\.8\’ | perl -spe’open(STDOUT, «>», $o)’ — -o=file_name.txt
This is very much possible, you just have to make sure that by the time you write the output, you’re writing it to a different file. This can be done by removing the file after opening a file descriptor to it, but before writing to it:
Or line by line, to understand it better :
exec 3file # run command, with the removed file as input exec 3>&- # close the file descriptor It’s still a risky thing to do, because if COMMAND fails to run properly, you’ll lose the file contents. That can be mitigated by restoring the file if COMMAND returns a non-zero exit code :
exec 3file || cat file ; exec 3>&- We can also define a shell function to make it easier to use :
# Usage: replace FILE COMMAND replace() < exec 3$1 || cat $1 ; exec 3>&- > $ echo aaa > test $ replace test tr a b $ cat test bbb Also, note that this will keep a full copy of the original file (until the third file descriptor is closed). If you’re using Linux, and the file you’re processing on is too big to fit twice on the disk, you can check out this script that will pipe the file to the specified command block-by-block while unallocating the already processed blocks. As always, read the warnings in the usage page.
One liner alternative — set the content of the file as variable:
VAR=`cat file_name`; echo "$VAR"|grep -v 'seg2\\.3\' > file_name Several other similar answers have appeared, some of which have a fuller discussion of how this works, and some caveats. Modern scripts should definitely prefer the modern $(command substitution) syntax over backticks, which were deeply obsolescent already in 2013.
Just to repeat the feedback from elsewhere, you should prefer printf over echo here for robustness; and this will lose any trailing newlines.
The following will accomplish the same thing that sponge does, without requiring moreutils :
shuf --output=file --random-source=/dev/zero The —random-source=/dev/zero part tricks shuf into doing its thing without doing any shuffling at all, so it will buffer your input without altering it.
However, it is true that using a temporary file is best, for performance reasons. So, here is a function that I have written that will do that for you in a generalized way:
# Pipes a file into a command, and pipes the output of that command # back into the same file, ensuring that the file is not truncated. # Parameters: # $1: the file. # $2: the command. (With $3. being its arguments.) # See https://stackoverflow.com/a/55655338/773113 siphon() < local tmp file rc=0 [ "$#" -ge 2 ] || < echo "Usage: siphon filename [command. ]" >&2; return 1; > file="$1"; shift tmp=$(mktemp -- "$file.XXXXXX") || return "$@" "$tmp" || rc=$? mv -- "$tmp" "$file" || rc=$(( rc | $? )) return "$rc" > Read from and write to file from stdin and stdout
I need to continually read from and write to a (virtual) file in bash, without closing the file in between. I’m looking for a program which will open the file for read and write, read from stdin and write what it reads to the file AND read from the file and write it to stdout. A bit like netcat but for files. The file in question is a virtual file in /sys (a custom thing in something I’m working on) to which one writes a command and reads to receive the response — where the write and subsequent read must be done in the same session, i.e. on the same file descriptor without closing it in between. Easy to do in C — open(«file», O_RDWR) , write() , read() — note: no need to seek() . Is there a standard GNU tool or do I need to write one?
Maybe you can used tee to achieve what you need? It will not read from the file though, but will copy to stdout what it writes to the file.
If the suggestions that you’ve gotten so far aren’t satisfactory to you, try explaining what you want to do a bit more clearly. For example, are you starting with an empty file (or maybe a non-existent file), reading from stdin, and writing the data to both the file and stdout? Or is there something already in the file when you start? If so, do you want the new data to overwrite the old data? In this case, do you want data from stdin to end up on stdout (after being written to the file), or do you want only the original contents of the file to be written to stdout?
Is that file behaving like a regular file (seekable, and read() return 0 bytes (EOF) when the end is reached), or like a bi-directional pipe or socket (where what you write is not what you may read again after seeking and where read() blocks when there’s no available input)?
read and write same file bash/unix
I have a list of keys which are listed in a txt file sample.txt address contact_id created_at creator_id custom_fields address contact_id phone name email name I have the following recursive script which can remove duplicates
#!/bin/bash function keyChecker()< if grep -q $i uniqueKeys.txt; then echo "Duplicate Key: $" else echo $i >> uniqueKeys.txt echo "New key: $" fi > function recursiveDoer() < for i in $(cat keys.txt); do keyChecker $i done recursiveDoer >touch uniqueKeys.txt counter=0 recursiveDoer - What is the proper way to write this method using recursion and no infinite loop?
- Can this be simplified and written as a non-recursive method in a single loop?
You don’t need recursion here. Remove the call inside the function, the for loop checks all keys and this is the end.
Actually that is not true, if you do that, you will find that it will treat all keys as new keys. I believe this is caused by not closing the file. Further testing needed
You really do not need a recursive function here. However, you should change both instances of $i in keychecker to $1 (or better yet to «$1» ) since you apparently expect keychecker to check its argument. (Also in the echoed strings. $ is unnecessary; just use $1 .)
rici, those comments don’t actually help with my problem. All you have done is describe a different way to reference my arguement. You did not solve the recursion issue. Please keep comments to recursion issues
That is why it is a comment and not an answer 🙂 However, «the recursion is unnecessary» is, in fact, an accurate statement. The file is not kept open between iterations.
2 Answers 2
If you do not care about the order of the keys you can simply use
sort -u keys.txt > uniqueKeys.txt
great answer, though for my usecase I do need them to be in order as they are later associated with value pairs.
this awk magic will only print the unique entries in the same order
you can overwrite the input file with this idiom
awk '!a[$0]++' file > temp && mv temp file this probably replicates your code
awk '!a[$0]++ "uniqueKeys.txt" next> ' file Explanation
a[$0] creates an entry in the associative map a for the read line as the key. ++ forces the null value to be treated as 0 and increments. ! forces the value to be treated as Boolean and negates it. Taken all together, it will be only true for the first time the key is seen, therefore effectively de-duplicating the lines in the file.