Is there an easy way to log all commands executed, including command line arguments?
I am trying to find how to log a specific instantiation of rrdtool to see whether the path it is receiving is incorrect. I know I could wrap the executable in a shell script that would log the parameters, but I was wondering if there was a more kernel-specific way to monitor for that, perhaps a filesystem callback that sees when a particular /proc/pid/exe matches a given binary?
Is there a way to get auditd to record the command-line arguments as well as the program ran? serverfault.com/questions/765179/…
3 Answers 3
Yes, there is a kernel facility: the audit subsystem. The auditd daemon does the logging, and the command auditctl sets up the logging rules. You can log all calls to a specific system alls, with some filtering. If you want to log all commands executed and their arguments, log the execve system call:
auditctl -a exit,always -S execve To specifically trace the invocation of a specific program, add a filter on the program executable:
auditctl -a exit,always -S execve -F path=/usr/bin/rrdtool The logs show up in /var/log/audit.log , or wherever your distribution puts them. You need to be root to control the audit subsystem.
Once you’re done investigating, use the same command line with -d instead of -a to delete a logging rule, or run auditctl -D to delete all audit rules.
For debugging purposes, replacing the program by a wrapper script gives you more flexibility to log things like the environment, information about the parent process, etc.
@Graeme The problem described in the question was tracking an invocation of rrdtool . If you want to log invocations of all programs, drop the -F path=… part (you’ll get a lot of logs of course).
This is all great, but how do I reset the config it to initial state? Otherwise it’ll continue filling the log with new and new launched commands. or is this auditctl command only effective until reboot?
@Ruslan The effect of auditctl only survives until reboot, but that’s a good point anyway, I’ve added instructions on removing them without rebooting to my answer.
Snoopy is more lightweight solution as it does not need kernel cooperation. All that is needed is dynamic loader (dl) that preloads snoopy library, path to which is specified in /etc/ld.so.preload .
Disclosure: I am current snoopy maintainer.
Is it possible to do logging for commands directly or indirectly spawned from a particular shell only?
I am not sure I understand your question — do you mean «shell» as specific program that is used as shell (bash, dash, zsh etc.), or you mean you would like to log just specific PTY? Snoopy provides filtering framework, but currently only a couple of very basic filters are implemented, see here for the list: link. If you have a concrete usecase that might be applicable to others, please do explain in feature request, and, oh, btw, patches are welcome 🙂
There is no specific filter for PTY available ATM. However, you could use snoopy to log everything, including which PTY the event occured on, and then do the filtering in your syslog daemon. I do not know which one you are using, but syslog-ng (for example) can do regex matching, positive or negative.
Sure, thanks! The tool and the approach is in general quite useful. I can easily do some filtering to get what I need.
The Linux kernel «audit» subsystem can do what you need.
e.g. if you run these commands:
auditctl -a exit,always -F arch=b64 -S execve auditctl -a exit,always -F arch=b32 -S execve Then every execution event is logged, and a lot of information is provided around that
e.g. this is the output of me running tail /var/log/audit/audit.log
exit=0 a0=7f0e4a21e987 a1=7f0e4a21e6b0 a2=7f0e4a21e808 a3=8 items=2 ppid=906 pid=928 auid=500 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="tail" exe="/usr/bin/tail" subj=kernel key=(null) type=EXECVE msg=audit(1543671660.203:64): argc=2 a0="tail" a1="/var/log/audit/audit.log" type=CWD msg=audit(1543671660.203:64): cwd="/home/sweh" type=PATH msg=audit(1543671660.203:64): item=0 name="/usr/bin/tail" inode=266003 dev=fd:03 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=unlabeled objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0 type=PATH msg=audit(1543671660.203:64): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=273793 dev=fd:03 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=unlabeled objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0 type=PROCTITLE msg=audit(1543671660.203:64): proctitle=7461696C002F7661722F6C6F672F61756469742F61756469742E6C6F67 There’s some interesting values that can be seen; e.g. «auid» is 500, which is my login ID, even though «uid» is zero (‘cos I’m running under su ). So even though the user may have switched accounts with su or sudo we can still track back to their «audit ID»
Now those auditctl commands will be lost on a reboot. You can put them into a configuration file (eg in the /etc/audit/rules.d/ directory, on CentOS 7). The exact location will depend on your OS version. The auditctl manual page should help here.
Beware, though. this will cause a lot of log messages to be generated. Make sure you have enough space on the disk!
If necessary the rules can be limited to a specific user, or a specific command.
And also beware; if a user puts the password in the command execution (e.g. mysql —user=username —password=passwd ) then this will be logged.
Where is the log of used commands in Linux?
command provides the list of commands we used previously but its limit is less. New commands are overwritten on old commands then old commands are missing. I think somewhere the log of the commands is stored in the system. Is that log available to open and read? If possible can we change the maximum size of log that it can handle? Is the output of history command depends upon the login-user?
5 Answers 5
A simple solution is to increase the number of commands saved and returned by history . If you use bash, add this line to your ~/.profile :
export HISTSIZE=100000 export HISTFILESIZE=999999 HISTFILESIZE The maximum number of lines contained in the history file. When this variable is assigned a value, the history file is trun‐ cated, if necessary, to contain no more than that number of lines by removing the oldest entries. The history file is also truncated to this size after writing it when a shell exits. If the value is 0, the history file is truncated to zero size. Non-numeric values and numeric values less than zero inhibit truncation. The shell sets the default value to the value of HISTSIZE after reading any startup files. HISTSIZE The number of commands to remember in the command history (see HISTORY below). If the value is 0, commands are not saved in the history list. Numeric values less than zero result in every command being saved on the history list (there is no limit). The shell sets the default value to 500 after reading any startup files. @gafoor yes, the history file I mention in my answer. Nothing else by default. If you set the variables I shows, you will keep a list of thousands of commands.
The file ~/.bash_history saves the list of executed commands. At least in CentOS this file exists, I don’t know if it exists in other distros.
@AbdulGafoor yes, but that’s what you see when you run history . It will only have as many commands as those returned by history . To keep more, you need to set the variables I show in my answer.
This file also exists in Mac OSX. You can type vim ~/.bash_history to see the contents of the file under Mac.
It’s the HISTFILE variable which sets the location history file, which by default (for bash) is ~/.bash_history
There is no such log, at least not by default.
There are tools you can install, like acct («GNU Accounting utilities for process and login accounting») which can keep track of all programs executed on the system.
acct is packaged for most (probably all except tiny-distros) linux distros. The home page is at http://www.gnu.org/software/acct/
acct provides the following commands, lastcomm probably does what you’re asking for:
ac prints statistics about users’ connect time. ac can tell you how long a particular user or group of users were connected to your system, printing totals by day or for all of the entries in the wtmp file.
accton turns accounting on or off.
lastcomm lists the commands executed on the system, most recent first, showing the run state of each command. With last, you can search the acct file for a particular user, terminal, or command.
sa summarizes the information in the acct file into the savacct and usracct file. It also generates reports about commands, giving the number of invocations, cpu time used, average core usage, etc.
dump-acct dump-utmp display acct and utmp files in a human-readable format.
Logging terminal commands
Assuming you’re doing this for security purposes, take a look at process accounting. You didn’t say which UNIX variant you’re on, but for Linux, take a look at http://www.gnu.org/software/acct/ Beware that you will generate a lot of data as process accounting writes data about every command run system wide (its a kernel feature). Your distro probably has the utilities already; in Debian check the ‘acct’ package.
If you’re just trying to log what you did so you can go back and look at it later, script (as in the other answer) or screen (with -L) will provide a better solution. However, they will not work for security auditing because its trivial to bypass them: just run a shell script. Also, line editing makes a royal mess of the files. Curses apps (like, say, a text editor) are even worse.
Also, if you force your users through e.g., script, you may wind up capturing information (e.g., email messages) which it may be illegal for you to intercept.
One more option — sudosh or other equivalents. Not sure that they provide much benefit compared to script, but probably worthy of mention.
Process accounting is the way to go, despite it sucking up lots of disk space. It’s not something I’d leave running unless you have a very grunty box but it’s very useful for problem solving since it basically tracks every process, a claim the simpler «snapshot of ps» tools can’t match.
You basically turn it on with accton /var/account/pacct and the kernel then writes details of every process that exits, including:
- process name (not args, unfortunately).
- start time.
- elapsed time.
- user and system CPU times.
- exit code.
- memory, I/O, fault and swap statistics.
- process ID.
- parent process ID.
You shut it down with a naked accton so all you people who laughed at Windows for using a Start button to shut down, HAH !! 🙂
There are two variants of records that can be logged, v1 and v3 — make sure you’re using the latest process accounting software and v3 records as they hold more useful information. The /usr/include/linux/acct.h file shows what you can get from there.
The records are all fixed size (regardless of version) and easy to process.
We’ve just recently finished a Linux agent for our performance monitoring software — I thought I’d jot this down while it’s still fresh in my memory.
One thing to watch out for are the comp_t fields — these are actually a weird exponent/mantissa type for holding a large range of values in 16 bits — the algorithm is pretty simple for turning it into a long:
Another thing is that some values are in clock ticks, not seconds, so you have to divide by the value returned by sysconf (_SC_CLK_TCK) .