Shell command to monitor changes in a file
I know there was a command on Unix that I could use to monitor a file and see changes that are getting written to it. This was quite useful especially for checking log files. Do you know what it is called?
14 Answers 14
Sidenote: If your distribution provides the tailf command, use that in preference to tail -f. tailf is more efficient because it doesn’t need to access the watched file if it’s not being written to (poll accesses are annoying if you mounted the file system with atime updating.)
tail -F will follow filenames rather than file objects, which is especially useful in case of log file rotation.
Update, a few years later: tailf is now deprecated and tail -f is safe. (confirm this on your system with man tailf .) See documentation: man7.org/linux/man-pages/man1/tailf.1.html
You probably meant tail, as per Jon Skeet’s answer.
Another useful one is watch; it allows you to run a command periodically and see the output full screen. For example:
watch -n 10 -d ls -l /var/adm/messages
Will run the command ls -l /var/adm/messages every 10 seconds, and highlight the difference in the output between subsequent runs. (Useful for watching how quickly a logfile is growing, for example).
inotifywait from inotify-tools is useful if you want to run a command every time a file (or any files in a directory) change. For example:
inotifywait -r -m -e modify /var/log | while read file_path file_event file_name; do echo $$ event: $ done Setting up watches. Beware: since -r was given, this may take a while! Watches established. /var/log/messages event: MODIFY /var/log/kern event: MODIFY . Just a note that path isn’t the greatest choice for a variable name. On zsh , it seems that environment vars aren’t case-sensitive. For me, setting path causes PATH to also get set, and that basically means nothing will execute until you fix that. On bash , setting path has no effect on PATH .
@Thanatos Zsh variables are case-sensitive, but among the variables set by Zsh itself, Zsh «ties» the *PATH variables to an array of the same name, but lowercase. Tied variables always consist of a scalar and an array (e.g. PATH and path ), and modifying one modifies the other. A key feature is that the array version is automatically split on the separator in the scalar version (the : ). See for yourself with print «$PATH\n$path» . The second paragraph in the PARAMETERS USED BY THE SHELL section in the zshparam(1) man page has more detailed information.
As a further note, there’s quite a few variables used by Zsh that are tied, not just PATH and path . They are all listed in the section in my previous comment including, but not limited to: FPATH / fpath , CDPATH / cdpath , MANPATH / manpath , FIGNORE / fignore , and more.
I prefer using less +FG 1 over tail -f because I find myself needing to search a log file for a specific error or ID. If I need to search for something, I type ^C to stop following the file and ? to start searching backwards.
Key bindings are pretty much the same as in vi . Any command can be initialized on startup using the + option:
+cmd Causes the specified cmd to be executed each time a new file is examined. For example, +G causes less to initially display each file starting at the end rather than the beginning. For really long logs, I find it convenient to use the -n option which turns off line numbering. From the manpage:
-n or --line-numbers Suppresses line numbers. The default (to use line numbers) may cause less to run more slowly in some cases, especially with a very large input file. Suppressing line numbers with the -n option will avoid this problem. Using line numbers means: the line number will be displayed in the verbose prompt and in the = command, and the v command will pass the current line number to the editor (see also the discussion of LESSEDIT in PROMPTS below). 1. Hat-tip to rgmarcha for pointing this out in the comments.
Best way to monitor file system changes in linux
I’m looking at building a file system sync utility that monitors file system activity, but it appears that some of the file system monitoring features in the linux kernel are obsolete or not fully featured. What my research as found dnotify came first with notification has the features of notifying for delete,modify,access,attribs,create,move can determine file descriptor, however is now out dated by inotify and fanotify inotify came out second with notification has the features of notifying access, modify, attrib, close, move, delete, create, etc however it does not give you a file descriptor or process and will be outdated by fanotify fanotify is the latest which informs of access, modify, close, but does not inform of delete or attribs, but does provide file descriptor I need a way of determining the process (e.g. from fd) and things like delete, modify, attribs, etc in order to sync everything, any suggestions? Unfortunately dnotify seems the best but most out-dated
3 Answers 3
You should use a library instead of inotify and friends — something like FAM or Gamin (it’s the same API for both). This will make your program portable to other Unixes.
There’s a good lib providing file descriptors or process with inotify. It has his own C API and the inotifywatch util (good for scripts), all in inotify-tools package.
I strongly disagree that fanotify will outdate inotify.
FAM and gamin are very good server/client options. Both of them use inotify as first option over the outdated dnotify and polls. I prefer gamin.
Command to monitor file changes within a directory and execute command
to compile it into HTML format. I am trying to automate the process. so far I have come with this using entr:
ls *.adoc | entr asciidoctor -q *.adoc but only works with existing folder not for subfolders. I have tried this variation, but it doesn’t work:
find . -name '*.adoc' | entr asciidoctor -q *.adoc 4 Answers 4
Linux provides a nice interface for monitoring all file system events like creating, modifying, removing files. The interface is inotify family of system calls, the userspace utilities leveraging these calls are provided by the inotify-tools package in Ubuntu (available on the universe repository). If you don’t have it already, install by:
sudo apt-get install inotify-tools inotify-tools provides inotifywait and inotifywatch binaries, we need the first one.
So you want to run the command asciidoctor -q some_file when any .adoc file is modified ( some_file will replaced by that), if so assuming your .adoc files are in directory /foo/bar , you can set the watch:
inotifywait -qm --event modify --format '%w' /foo/bar/*.adoc - -q enables the quiet mode, no info from inotifywait itself
- -m enables monitor mode, otherwise it will exit after the first event
- —event modify , we are only interested in modify event i.e. when a file is modified. Other possible values include open , close etc.
- —format %w , we only want the file name that is modified rather than bunch of other info as we will use the file name as input to another command
- /foo/bar/*.adoc will be expanded to all .adoc files under /foo/bar directory
Now the above will show you the filename whenever any is modified, now to run the command on the filename (assuming the command takes arguments via STDIN):
inotifywait -qm --event modify --format '%w' /foo/bar/*.adoc | while read -r file ; do asciidoctor -q "$file" done You can also setup a recursive watch on the directory, you will then need to use grep to filter the desired files only. Here setting the watch recursively ( -r ) on directory /foo/bar and using grep to filter only .adoc files:
inotifywait -qrm --event modify --format '%w%f' /foo/bar | grep '\.adoc$' | while read -r file ; do asciidoctor -q "$file" done When watching directories the output format specifier %w resolves to the directory name, so we need %f to get the file name. While watching files, %f would resolve to empty string.
Note that, you can also run inotifywait in daemon ( -d ) mode, you can also script the whole thing, and/or run in background, and/or play with it more other options.
Also, you can replace asciidoctor with any other command of your choice, if you want.
Check man inotifywait to get more idea.
Linux: How to Monitor File System for Changes
The Filesystem notification APIs allows applications to watch certain files and be notified when they are opened, modified, deleted, or renamed. This greatly aids the applications, because before such filesystem event monitoring tools existed, such applications would have to read the disk repeatedly to detect any changes, which resulted in high disk and CPU usage.
File system event monitoring tools
There are few tools to do the job, namely:
- FAM (File Alteration Monitor) – It is one of the oldest portable event monitors. It sends events to an application when changes are made to files or directories that an application has registered to be monitored. It is also complicated.
- Gamin – newer and simpler than FAM . It tries to be compatible with FAM while not implementing many of the obscure feature. It is moderately maintained and widely available on many distros. It is portable, but the focus of development and testing is on Linux. It has BSD support and can be found in FreeBSD Ports.
- dnotify – introduced in 2.4 kernel series. It can only watch directories and requires maintaining an open file descriptor to the directory that the user wants to watch. It has been obsoleted by inotify .
- inotify – a dnotify replacement. It’s part of the Linux kernel subsystem and included in mainline Linux kernel since release 2.6.13. It’s fast and lightweight and should be available under all Linux distributions.
Installation and Examples
In this article, we will use inotify (inode notify) to detect file system changes. Let’s begin by first installing the inotify-tools package. The package includes a set of command-line programs that provide a simple interface to inotify . The libnotify-bin package is optional and is only needed for GUI desktop notifications.