Linux truncate file beginning

Truncating a File While It’s Being Used (Linux)

Take a look at the utility split(1) , part of GNU Coreutils.

How to delete older contents of file that is being continuously written to?

As Carl mentioned in the comments, you cannot really do this on an actively written log file. However, if the initial data is not relevant to you, you can do the following (though beware that you will loose all data)

For future, you can use a utility called logrotate(8)

How to empty ( truncate ) a file on linux that already exists and is protected in someway?

You have the noclobber option set. The error looks like it’s from csh, so you would do:

If I’m wrong and you are using bash, you should do:

in bash, you can also shorten that to:

Truncate file at front

Truncate files at front seems not too hard to implement at system level.

• The first one is at programming level. When opening file in random access the current paradigm is to use offset from the beginning of the file to point out different places in the file. If we truncate at beginning of file (or perform insertion or removal from the middle of the file) that is not any more a stable property. (While appendind or truncating from the end is not a problem).

In other words truncating the beginning would change the only reference point and that is bad.

• At a system level uses exist as you pointed out, but are quite rare. I believe most uses of files are of the write once read many kind, so even truncate is not a critical feature and we could probably do without it (well some things would become more difficult, but nothing would become impossible).

If we want more complex accesses (and there are indeed needs) we open files in random mode and add some internal data structure. Theses informations can also be shared between several files. This leads us to the last issue I see, probably the most important.

• In a sense when we using random access files with some internal structure. we are still using files but we are not any more using files paradigm. Typical such cases are the databases where we want to perform insertion or removal of records without caring at all about their physical place. Databases can use files as low level implementation but for optimisation purposes some database editors choose to completely bypass filesystem (think about Oracle partitions).

I see no technical reason why we couldn’t do everything that is currently done in an operating system with files using a database as data storage layer. I even heard that NTFS has many common points with databases in it’s internals. An operating system can (and probably will in some not so far future) use another paradigm than files one.

Summarily i believe that’s not a technical problem at all, just a change of paradigm and that removing the beginning is definitely not part of the current «files paradigm», but not a big and useful enough change to compell changing anything at all.

Writing to the same file with awk and truncate

This probably isn’t an answer but it’s too much to go in a comment. I don’t know the details of most of the tools you mention, nor do I really understand what it is you’re trying to do but:

A shell is a tool to manipulate files and processes and schedule calls to other tools. Awk is a tool to manipulate text. You’re trying to use awk like a shell — you have it sequencing calls to truncate and pkill and calling system to spawn a subshell each time you want to execute either of them. What you should be doing, for example, is just:

but what you’re actually doing is:

Can you take that role away from awk and give it back to your shell? It should make your overall script simpler, conceptually at least, and probably more robust.

Maybe try something like this (untested):

#!/usr/bin/env bash

while IFS= read -r str; do 
case $str in 
Return ) exit 0 ;; 
BackSpace ) truncate -s-1 timer.txt ;; 
? ) printf "%s" "$str" | tee -a timer.txt ;; 
esac 
pkill -RTMIN+3 dwmblocks
done < <(
xev -root | 
awk -F'[ )]+' '/^KeyPress/ NR in a'
)

I moved the write to timer.txt inside the loop to make sure tee s not trying to write to it while you’re truncating it — that may not be necessary.

Источник

How to effectively truncate the head of the file?

Everyone knows the truncate(file, size) function, which changes the file size to a given size, by truncating the tail of the file. But how to do the same, only with the truncation of not the tail of the file and his head?

When the file is a logfile and the program writing to it keeps the filedescriptor open, no more logging will appear after moving the file. See unix.stackexchange.com/a/122942/57293

open the file, then sendfile(fd, fd, &offset, size) , I test on ubuntu 16, works ok, but the older kernel doesn’t support

3 Answers 3

Generally, you have to rewrite the entire file. The simplest way is to skip the first few bytes, copy everything else to a temporary file, and rename the temporary file on top of the old one when done. A more elaborate way is to rewrite it in place, analogous to how memmove works, with read/seek/write/seek or pread/pwrite, and then truncate the last bit when done.

If you are on a recent version of Linux (>= 3.15), and you have a supported filesystem (currently ext4 or xfs), and the amount you wish to remove happens to be a multiple of the filesystem block size, you could use the non-portable fallocate(2) with the FALLOC_FL_COLLAPSE_RANGE flag. Note that this feature is not supported by the portable posix_fallocate .

You could just use tail —lines= to always cap the log file to the last linecount lines. This works if you’re not trying to truncate to a specific / fixed file size.

The only way is to read the whole file and overwrite it from a needed position.

Linked

Related

Hot Network Questions

Subscribe to RSS

To subscribe to this RSS feed, copy and paste this URL into your RSS reader.

Site design / logo © 2023 Stack Exchange Inc; user contributions licensed under CC BY-SA . rev 2023.7.17.43537

By clicking “Accept all cookies”, you agree Stack Exchange can store cookies on your device and disclose information in accordance with our Cookie Policy.

Источник

Delete the first n bytes of files

I’ve got an extreme problem, and all of the solutions I can imagine are complicated. According to my UNIX/Linux experience there must be an easy way. I want to delete the first 31 bytes of each file in /foo/ . Each file is long enough. Well, I’m sure somebody will deliver me a suprisingly easy solution I just can’t imagine. Maybe awk?

Any awk/sed/ed solution will be line-oriented, so if you don’t know the first line will be at least 31 characters then complications ensue.

5 Answers 5

for file in /foo/* do if [ -f "$file" ] then dd if="$file" of="$file.truncated" bs=31 skip=1 && mv "$file.truncated" "$file" fi done 

or the faster, thanks to Gilles’ suggestion:

for file in /foo/* do if [ -f $file ] then tail +32c $file > $file.truncated && mv $file.truncated $file fi done 

Note: Posix tail specify «-c +32» instead of «+32c» but Solaris default tail doesn’t like it:

 $ /usr/bin/tail -c +32 /tmp/foo > /tmp/foo1 tail: cannot open input 

/usr/xpg4/bin/tail is fine with both syntaxes.

If you want to keep the original file permissions, replace

. && cat "$file.truncated" "$file" && rm "$file.truncated" 

Suggesting dd here is overkill, tail is more appropriate (simpler, less risk of a killer typo, no spurious messages on stderr).

You are right. I usually avoid commands intended to process text files when processing possibly binary ones but «tail +32c» will work here.

mv changes the file permissions. you can use cat instead of mv like this: tail +32c $file > $file.truncated && cat $file.truncated > $file && rm $file.truncated @jlliagre

The following commands cut first 31 bytes from $file (using $file~ as a temp. copy):

dd if="$file" of="$file~" bs=1 skip=31 mv "$file~" "$file" 

You only need to list or find all files under /foo/ and execute the two above for every $file found.

tail -c +32 outputs its input minus the first 31 bytes. (Yes, the argument is off by one.) To edit a file in place, use sponge in a loop, or if you don’t have it and don’t want to bother, do its job in the shell:

for x in /foo/*; do tail -c +32 "$x" | sponge "$x"; done for x in /foo/*; do tail -c +32 "$x" >"$x.new" && mv "$x.new" "$x"; done 

If the commands are interrupted for whatever reason (e.g. a power failure), it might be hard to figure out where you left off. Writing the new files to a separate directory would make things easier.

mkdir /foo.tmp cd /foo for x in *; do tail -c +42 -- "$x" >"/foo.tmp/$x" && rm -- "$x"; done mv /foo.tmp/* /foo rmdir /foo.tmp 

If the files are really large (as in, large enough that having two copies of even a single one is a problem), you can use one of the techniques mentioned in this thread.

Источник