Linux track changes to files

Watchman – A File and Directory Watching Tool for Changes

Watchman is an open source and cross-platform file watching service that watches files and records or performs actions when they change. It is developed by Facebook and runs on Linux, OS X, FreeBSD, and Solaris. It runs in a client-server model and employs the inotify utility of the Linux kernel to provide a more powerful notification.

Useful Concepts of Watchman

• It recursively watches watch one or more directory trees.
• Each watched directory is called a root.
• It can be configured via the command-line or a configuration file written in JSON format.
• It records changes to log files.
• Supports subscription to file changes that occur in a root.
• Allows you to query a root for file changes since you last checked, or the current state of the tree.
• It can watch an entire project.

In this article, we will explain how to install and use watchman to watch (monitor) files and record when they change in Linux. We will also briefly demonstrate how to watch a directory and invoke a script when it changes.

Installing Watchman File Watching Service in Linux

We will install watchman service from sources, so first install these required dependencies libssl-dev, autoconf, automake libtool, setuptools, python-devel and libfolly using following command on your Linux distribution.

----------- On Debian/Ubuntu ----------- $ sudo apt install autoconf automake build-essential python-setuptools python-dev libssl-dev libtool ----------- On RHEL/CentOS ----------- # yum install autoconf automake python-setuptools python-devel openssl-devel libssl-devel libtool # yum groupinstall 'Development Tools' ----------- On Fedora ----------- $ sudo dnf install autoconf automake python-setuptools openssl-devel libssl-devel libtool $ sudo dnf groupinstall 'Development Tools'

Once required dependencies installed, you can start building watchman by downloading its github repository, move into the local repository, configure, build and install it using following commands.

$ git clone https://github.com/facebook/watchman.git $ cd watchman $ git checkout v4.9.0 $ ./autogen.sh $ ./configure $ make $ sudo make install

Watching Files and Directories with Watchman in Linux

Watchman can be configured in two ways: (1) via the command-line while the daemon is running in background or (2) via a configuration file written in JSON format.

To watch a directory (e.g ~/bin ) for changes, run the following command.

The following command writes a configuration file called state under /usr/local/var/run/watchman/-state/, in JSON format as well as a log file called log in the same location.

You can view the two files using the cat command as show.

$ cat /usr/local/var/run/watchman/aaronkilik-state/state $ cat /usr/local/var/run/watchman/aaronkilik-state/log

You can also define what action to trigger when a directory being watched for changes. For example in the following command, ‘ test-trigger ‘ is the name of the trigger and ~bin/pav.sh is the script that will be invoked when changes are detected in the directory being monitored.

For test purposes, the pav.sh script simply creates a file with a timestamp (i.e file.$time.txt ) within the same directory where the script is stored.

time=`date +%Y-%m-%d.%H:%M:%S` touch file.$time.txt

Save the file and make the script executable as shown.

To launch the trigger, run the following command.

$ watchman -- trigger ~/bin 'test-trigger' -- ~/bin/pav.sh

When you execute watchman to keep an eye on a directory, its added to the watch list and to view it, run the following command.

To view the trigger list for a root, run the following command (replace ~/bin with the root name).

Based on the above configuration, each time the ~/bin directory changes, a file such as file.2019-03-13.23:14:17.txt is created inside it and you can view them using ls command.

Uninstalling Watchman Service in Linux

If you want to uninstall watchman, move into the source directory and run the following commands:

$ sudo make uninstall $ cd '/usr/local/bin' && rm -f watchman $ cd '/usr/local/share/doc/watchman-4.9.0 ' && rm -f README.markdown

For more information, visit the Watchman Github repository: https://github.com/facebook/watchman.

You might also like to read these following related articles.

Watchman is an open source file watching service that watches files and records, or triggers actions, when they change. Use the feedback form below to ask questions or share your thoughts with us.

Источник

Track file changes using auditd

Most of Linux distributions comes with Linux Auditing System that makes it possible to track file changes, file accesses as well as system calls. It’s pretty useful functionality for sysadmins who wish to know who and when accessed and/or changed sensitive files like /etc/passwd, /etc/sudoers or others.

Daemon auditd that usually runs in background and starts after reboot by default logs those events into /var/log/audit.log file (or into other file if different syslog facility is specified). The common usage is to list all files which should be watched and search auditd’s logs from time to time. For example, I prefer to track any file changes into /etc/passwd, reading/writing of /etc/sudoers, executing of /bin/some/binary or just everything (read, write, attributes changes, executing) for my /very/important/file.

In order to configure that you’ll need two commands: auditctl and ausearch. First one is for configuring auditd daemon (e.g. setting a watch on a file), second one is for searching auditd logs (it’s possible to use grep against /var/log/audit.log too but ausearch command makes this task easier).

Install and start Linux Auditing System

If it happened that auditd daemon isn’t installed in your system then you can fix this by one of below commands:

sudo apt-get install audit

The next step is to make sure that auditd is running, if command ps ax | grep [a]udit shows nothing then start auditd using command:

As soon as auditd daemon is started we can start configuring it for tracking file changes using auditctl command.

Make auditd to log all file changes

auditctl -w /etc/passwd -k passwd-ra -p ra

This command will add a rule for auditd daemon to monitor file /etc/passwd file (see option -w /etc/passwd) for reading or changing the atributes (see option -p ra, where r is for read, a is for attribute). Also this command specifies filter key (-k passwd-ra) that will uniquely identify auditd records in its logs files.

Now let’s test this rule: optput the last 20 lines of /etc/passwd file and then search audit log for corresponding records

[[email protected] artemn]# ausearch -k passwd-ra ---- time->Wed Jul 4 15:17:14 2012 type=CONFIG_CHANGE msg=audit(1341407834.821:207310): auid=500 ses=23783 op="add rule" key="passwd-ra" list=4 res=1 ---- time->Wed Jul 4 15:17:20 2012 type=PATH msg=audit(1341407840.181:207311): item=0 name="/etc/passwd" inode=31982841 dev=09:02 mode=0100644 ouid=0 ogid=0 rdev=00:00 type=CWD msg=audit(1341407840.181:207311): cwd="/home/artemn" type=SYSCALL msg=audit(1341407840.181:207311): arch=c000003e syscall=2 success=yes exit=3 a0=7fffecd41817 a1=0 a2=0 a3=7fffecd40b40 items=1 ppid=642502 pid=521288 auid=500 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=23783 comm="tail" exe="/usr/bin/tail" key="passwd-ra"

As you can see the output of second command shows that auditd has one record for filter key ‘passwd-ra’, it shows that root user (uid=0 gid=0) has read file /etc/passwd using command tail (comm=”tail” exe=”/usr/bin/tail”) at July 4, 2012 (time->Wed Jul 4 15:17:20 2012).

Utility ausearch is pretty powerful so I recommend to read output of man ausearch , in the meantime here are some useful examples:

ausearch -x /bin/grep ausearch -x rm

This approach allows to scan auditd records for certain executable, e.g. if you’d like to see if any of watched files was deleted (or not) using command rm then you should use second command of above two.

This one will show you all records for certain UID (username).

Источник

Shell command to monitor changes in a file

I know there was a command on Unix that I could use to monitor a file and see changes that are getting written to it. This was quite useful especially for checking log files. Do you know what it is called?

14 Answers 14

Sidenote: If your distribution provides the tailf command, use that in preference to tail -f. tailf is more efficient because it doesn’t need to access the watched file if it’s not being written to (poll accesses are annoying if you mounted the file system with atime updating.)

tail -F will follow filenames rather than file objects, which is especially useful in case of log file rotation.

Update, a few years later: tailf is now deprecated and tail -f is safe. (confirm this on your system with man tailf .) See documentation: man7.org/linux/man-pages/man1/tailf.1.html

You probably meant tail, as per Jon Skeet’s answer.

Another useful one is watch; it allows you to run a command periodically and see the output full screen. For example:

watch -n 10 -d ls -l /var/adm/messages

Will run the command ls -l /var/adm/messages every 10 seconds, and highlight the difference in the output between subsequent runs. (Useful for watching how quickly a logfile is growing, for example).

inotifywait from inotify-tools is useful if you want to run a command every time a file (or any files in a directory) change. For example:

inotifywait -r -m -e modify /var/log | while read file_path file_event file_name; do echo $$ event: $ done 

Setting up watches. Beware: since -r was given, this may take a while! Watches established. /var/log/messages event: MODIFY /var/log/kern event: MODIFY . 

Just a note that path isn’t the greatest choice for a variable name. On zsh , it seems that environment vars aren’t case-sensitive. For me, setting path causes PATH to also get set, and that basically means nothing will execute until you fix that. On bash , setting path has no effect on PATH .

@Thanatos Zsh variables are case-sensitive, but among the variables set by Zsh itself, Zsh «ties» the *PATH variables to an array of the same name, but lowercase. Tied variables always consist of a scalar and an array (e.g. PATH and path ), and modifying one modifies the other. A key feature is that the array version is automatically split on the separator in the scalar version (the : ). See for yourself with print «$PATH\n$path» . The second paragraph in the PARAMETERS USED BY THE SHELL section in the zshparam(1) man page has more detailed information.

As a further note, there’s quite a few variables used by Zsh that are tied, not just PATH and path . They are all listed in the section in my previous comment including, but not limited to: FPATH / fpath , CDPATH / cdpath , MANPATH / manpath , FIGNORE / fignore , and more.

I prefer using less +FG 1 over tail -f because I find myself needing to search a log file for a specific error or ID. If I need to search for something, I type ^C to stop following the file and ? to start searching backwards.

Key bindings are pretty much the same as in vi . Any command can be initialized on startup using the + option:

+cmd Causes the specified cmd to be executed each time a new file is examined. For example, +G causes less to initially display each file starting at the end rather than the beginning. 

For really long logs, I find it convenient to use the -n option which turns off line numbering. From the manpage:

-n or --line-numbers Suppresses line numbers. The default (to use line numbers) may cause less to run more slowly in some cases, especially with a very large input file. Suppressing line numbers with the -n option will avoid this problem. Using line numbers means: the line number will be displayed in the verbose prompt and in the = command, and the v command will pass the current line number to the editor (see also the discussion of LESSEDIT in PROMPTS below). 

_{1. Hat-tip to rgmarcha for pointing this out in the comments.}

Источник