Best way to monitor file system changes in linux
I’m looking at building a file system sync utility that monitors file system activity, but it appears that some of the file system monitoring features in the linux kernel are obsolete or not fully featured. What my research as found dnotify came first with notification has the features of notifying for delete,modify,access,attribs,create,move can determine file descriptor, however is now out dated by inotify and fanotify inotify came out second with notification has the features of notifying access, modify, attrib, close, move, delete, create, etc however it does not give you a file descriptor or process and will be outdated by fanotify fanotify is the latest which informs of access, modify, close, but does not inform of delete or attribs, but does provide file descriptor I need a way of determining the process (e.g. from fd) and things like delete, modify, attribs, etc in order to sync everything, any suggestions? Unfortunately dnotify seems the best but most out-dated
3 Answers 3
You should use a library instead of inotify and friends — something like FAM or Gamin (it’s the same API for both). This will make your program portable to other Unixes.
There’s a good lib providing file descriptors or process with inotify. It has his own C API and the inotifywatch util (good for scripts), all in inotify-tools package.
I strongly disagree that fanotify will outdate inotify.
FAM and gamin are very good server/client options. Both of them use inotify as first option over the outdated dnotify and polls. I prefer gamin.
Track file changes using auditd
Most of Linux distributions comes with Linux Auditing System that makes it possible to track file changes, file accesses as well as system calls. It’s pretty useful functionality for sysadmins who wish to know who and when accessed and/or changed sensitive files like /etc/passwd, /etc/sudoers or others.
Daemon auditd that usually runs in background and starts after reboot by default logs those events into /var/log/audit.log file (or into other file if different syslog facility is specified). The common usage is to list all files which should be watched and search auditd’s logs from time to time. For example, I prefer to track any file changes into /etc/passwd, reading/writing of /etc/sudoers, executing of /bin/some/binary or just everything (read, write, attributes changes, executing) for my /very/important/file.
In order to configure that you’ll need two commands: auditctl and ausearch. First one is for configuring auditd daemon (e.g. setting a watch on a file), second one is for searching auditd logs (it’s possible to use grep against /var/log/audit.log too but ausearch command makes this task easier).
Install and start Linux Auditing System
If it happened that auditd daemon isn’t installed in your system then you can fix this by one of below commands:
sudo apt-get install audit
The next step is to make sure that auditd is running, if command ps ax | grep [a]udit shows nothing then start auditd using command:
As soon as auditd daemon is started we can start configuring it for tracking file changes using auditctl command.
Make auditd to log all file changes
auditctl -w /etc/passwd -k passwd-ra -p ra
This command will add a rule for auditd daemon to monitor file /etc/passwd file (see option -w /etc/passwd) for reading or changing the atributes (see option -p ra, where r is for read, a is for attribute). Also this command specifies filter key (-k passwd-ra) that will uniquely identify auditd records in its logs files.
Now let’s test this rule: optput the last 20 lines of /etc/passwd file and then search audit log for corresponding records
[[email protected] artemn]# ausearch -k passwd-ra ---- time->Wed Jul 4 15:17:14 2012 type=CONFIG_CHANGE msg=audit(1341407834.821:207310): auid=500 ses=23783 op="add rule" key="passwd-ra" list=4 res=1 ---- time->Wed Jul 4 15:17:20 2012 type=PATH msg=audit(1341407840.181:207311): item=0 name="/etc/passwd" inode=31982841 dev=09:02 mode=0100644 ouid=0 ogid=0 rdev=00:00 type=CWD msg=audit(1341407840.181:207311): cwd="/home/artemn" type=SYSCALL msg=audit(1341407840.181:207311): arch=c000003e syscall=2 success=yes exit=3 a0=7fffecd41817 a1=0 a2=0 a3=7fffecd40b40 items=1 ppid=642502 pid=521288 auid=500 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=23783 comm="tail" exe="/usr/bin/tail" key="passwd-ra"
As you can see the output of second command shows that auditd has one record for filter key ‘passwd-ra’, it shows that root user (uid=0 gid=0) has read file /etc/passwd using command tail (comm=”tail” exe=”/usr/bin/tail”) at July 4, 2012 (time->Wed Jul 4 15:17:20 2012).
Utility ausearch is pretty powerful so I recommend to read output of man ausearch , in the meantime here are some useful examples:
ausearch -x /bin/grep ausearch -x rm
This approach allows to scan auditd records for certain executable, e.g. if you’d like to see if any of watched files was deleted (or not) using command rm then you should use second command of above two.
This one will show you all records for certain UID (username).
Shell command to monitor changes in a file
I know there was a command on Unix that I could use to monitor a file and see changes that are getting written to it. This was quite useful especially for checking log files. Do you know what it is called?
14 Answers 14
Sidenote: If your distribution provides the tailf command, use that in preference to tail -f. tailf is more efficient because it doesn’t need to access the watched file if it’s not being written to (poll accesses are annoying if you mounted the file system with atime updating.)
tail -F will follow filenames rather than file objects, which is especially useful in case of log file rotation.
Update, a few years later: tailf is now deprecated and tail -f is safe. (confirm this on your system with man tailf .) See documentation: man7.org/linux/man-pages/man1/tailf.1.html
You probably meant tail, as per Jon Skeet’s answer.
Another useful one is watch; it allows you to run a command periodically and see the output full screen. For example:
watch -n 10 -d ls -l /var/adm/messages
Will run the command ls -l /var/adm/messages every 10 seconds, and highlight the difference in the output between subsequent runs. (Useful for watching how quickly a logfile is growing, for example).
inotifywait from inotify-tools is useful if you want to run a command every time a file (or any files in a directory) change. For example:
inotifywait -r -m -e modify /var/log | while read file_path file_event file_name; do echo $$ event: $ done Setting up watches. Beware: since -r was given, this may take a while! Watches established. /var/log/messages event: MODIFY /var/log/kern event: MODIFY . Just a note that path isn’t the greatest choice for a variable name. On zsh , it seems that environment vars aren’t case-sensitive. For me, setting path causes PATH to also get set, and that basically means nothing will execute until you fix that. On bash , setting path has no effect on PATH .
@Thanatos Zsh variables are case-sensitive, but among the variables set by Zsh itself, Zsh «ties» the *PATH variables to an array of the same name, but lowercase. Tied variables always consist of a scalar and an array (e.g. PATH and path ), and modifying one modifies the other. A key feature is that the array version is automatically split on the separator in the scalar version (the : ). See for yourself with print «$PATH\n$path» . The second paragraph in the PARAMETERS USED BY THE SHELL section in the zshparam(1) man page has more detailed information.
As a further note, there’s quite a few variables used by Zsh that are tied, not just PATH and path . They are all listed in the section in my previous comment including, but not limited to: FPATH / fpath , CDPATH / cdpath , MANPATH / manpath , FIGNORE / fignore , and more.
I prefer using less +FG 1 over tail -f because I find myself needing to search a log file for a specific error or ID. If I need to search for something, I type ^C to stop following the file and ? to start searching backwards.
Key bindings are pretty much the same as in vi . Any command can be initialized on startup using the + option:
+cmd Causes the specified cmd to be executed each time a new file is examined. For example, +G causes less to initially display each file starting at the end rather than the beginning. For really long logs, I find it convenient to use the -n option which turns off line numbering. From the manpage:
-n or --line-numbers Suppresses line numbers. The default (to use line numbers) may cause less to run more slowly in some cases, especially with a very large input file. Suppressing line numbers with the -n option will avoid this problem. Using line numbers means: the line number will be displayed in the verbose prompt and in the = command, and the v command will pass the current line number to the editor (see also the discussion of LESSEDIT in PROMPTS below). 1. Hat-tip to rgmarcha for pointing this out in the comments.
How to run a shell script when a file or directory changes?
@MerlynMorgan-Graham I’d move this to superuser, because the answer might not have anything do with programming — i.e. there might be some program or configuration option that can be used, without any programming needed. I had the same question, and searched superuser first :p
12 Answers 12
You may try entr tool to run arbitrary commands when files change. Example for files:
$ ls -d * | entr sh -c 'make && make test' $ ls *.css *.html | entr reload-browser Firefox or print Changed! when file file.txt is saved:
$ echo file.txt | entr echo Changed! For directories use -d , but you’ve to use it in the loop, e.g.:
while true; do find path/ | entr -d echo Changed; done while true; do ls path/* | entr -pd echo Changed; done entr is the simplest, most composable and unix-y tool for the job. Love it. incron can be replaced with entr and a process manager like runit , s6 or even systemd .
I have a script regularly appending to a log file. When I use entr to monitor that log file and touch the log, everything works fine, but when the script appends to the file, entr fails. This may be because I have noatime set in my fstab for my ssd — but that only stops the updating of the access time not the modify time, so this confuses me. I have then tried entr -cdr on the directory of files that are updated with the log. That recognizes with the directory contents change, but the -r does not work. The entr process just ends.