Linux: How to know where a process was started and how it was started?
I was checking a Linux box and found a perl process running and taking a good share of cpu usage. With top, i could only perl in process name. When i pressed c, to view the command-line, it showed /var/spool/mail. Which does not make sense, since this is directory. My questions are: 1) Why did this happen? How this perl process could mask its command-line? 2) What is the most reliable way of finding out where and how a process was started? Thanks!
9 Answers 9
In most cases just running ps is usually sufficient, along with your favorite flags to enable wide output. I lean towards ps -feww , but the other suggestions here will work. Note that if a program was started out of someone’s $PATH , you’re only going to see the executable name, not the full path. For example, try this:
$ lftp & $ ps -feww | grep ftp lars 9600 9504 0 11:30 pts/10 00:00:00 lftp lars 9620 9504 0 11:31 pts/10 00:00:00 grep ftp It’s important to note that the information visible in ps can be completely overwritten by the running program. For example, this code:
int main (int argc, char **argv)
If I compile this into a file called «myprogram» and run it:
$ gcc -o myprogram myprogram.c $ ./myprogram & [1] 10201 And then run ps , I’ll see a different process name:
$ ps -f -p 10201 UID PID PPID C STIME TTY TIME CMD lars 10201 9734 0 11:37 pts/10 00:00:00 foobar You can also look directly at /proc//exe , which may be a symlink to the appropriate executable. In the above example, this gives you much more useful information than ps :
$ls -l /proc/9600/exe lrwxrwxrwx. 1 lars lars 0 Feb 8 11:31 /proc/9600/exe -> /usr/bin/lftp How can I log all process launches in Linux?
I would like to get a log of all processes that are launched with the time that they were launched and the arguments they were launched with. Is this possible in Linux?
8 Answers 8
Your starting point should be auditd.
apt-get install auditd auditctl -a task,always ausearch -i -sc execve well one problem could be solved by chmod 0750 /sbin/audispd but it’s still not working (Debian Wheezy)
is says Unable to set audit pid, exiting but I guess that the real problem will be that the system is running in LXC container
I tried this on a live server and I effectively killed it, became almost unresponsive. I barely managed to remove this rule and make the server responsive again
I needed to do this, except (1) I didn’t need the time and (2) I was only interested in processes that are started by a given process, and its children and subsequent descendants. Also, in the environment I was using, it wasn’t possible to get auditd or accton , but there was valgrind .
Prefix the following to the process of interest on the command line:
valgrind --trace-children=yes The information you need will be in the log output displayed on STDERR.
By default, valgrind runs with the memcheck tool. To disable the tool and its related logging, and only print the creation of new commands (in addition to the usual output of your program), use the following command instead: valgrind —tool=none —trace-children=yes [command and args here] . Whenever a subprocess is spawned, Valgrind will then log the full command, including the arguments that were passed to it.
It is very simple to install, and since 2.x it can log arbitrary data (arguments, environmental variables, cwd, etc.).
Disclosure: Snoopy maintainer here.
You could run startmon and follow its standard output, Ctrl-C when done. Here’s how to compile and run startmon on recent Red Hat derived distros (RHEL, Fedora, CentOS):
sudo yum install git cmake gcc-c++ git clone https://github.com/pturmel/startmon cd startmon cmake . make sudo ./startmon -e On Debian (and Ubuntu etc), the first line of the above changes to:
sudo apt-get install git cmake g++ Alternatively you can try the execsnoop script in perf-tools, see this answer. By default only the first 8 arguments are shown (9 including the program name); you can increase this via
If you don’t have root access to the system, the best you can do is keep polling /proc and hope it catches everything (which it won’t), but for completeness here’s a script to do that (I’ve put duplicate-removal in to simplify the output)—although this isn’t as good as tracking them properly with one of the above methods, it does have the slight advantage of unambiguously displaying separators between the command-line arguments, in case you ever need to tell the difference between spaces inside an argument and space between arguments. This script is inefficient as it uses the CPU (well, one of its cores) 100% of the time.
You can also patch execsnoop to tell you more explicitly which argument is which: grep -v sub.*arg < execsnoop >n && chmod +x n && mv n execsnoop
CONFIG_FTRACE and CONFIG_KPROBES through brendangregg/perf-tools
git clone https://github.com/brendangregg/perf-tools.git cd perf-tools git checkout 98d42a2a1493d2d1c651a5c396e015d4f082eb20 sudo ./execsnoop while true; do sleep 1; date; done First shell shows data of format:
Tracing exec()s. Ctrl-C to end. Instrumenting sys_execve PID PPID ARGS 20109 4336 date 20110 4336 sleep 1 20111 4336 date 20112 4336 sleep 1 20113 4336 date 20114 4336 sleep 1 20115 4336 date 20116 4336 sleep 1 CONFIG_PROC_EVENTS
$ su # ./proc_events & # /proc_events.out & set mcast listen ok # sleep 2 & sleep 1 & fork: parent tid=48 pid=48 -> child tid=56 pid=56 fork: parent tid=48 pid=48 -> child tid=57 pid=57 exec: tid=57 pid=57 exec: tid=56 pid=56 exit: tid=57 pid=57 exit_code=0 exit: tid=56 pid=56 exit_code=0 CONFIG_PROC_EVENTS exposes the events to userland via a netlink socket.
#define _XOPEN_SOURCE 700 #include #include #include #include #include #include #include #include #include #include #include static volatile bool need_exit = false; static int nl_connect() < int rc; int nl_sock; struct sockaddr_nl sa_nl; nl_sock = socket(PF_NETLINK, SOCK_DGRAM, NETLINK_CONNECTOR); if (nl_sock == -1) < perror("socket"); return -1; >sa_nl.nl_family = AF_NETLINK; sa_nl.nl_groups = CN_IDX_PROC; sa_nl.nl_pid = getpid(); rc = bind(nl_sock, (struct sockaddr *)&sa_nl, sizeof(sa_nl)); if (rc == -1) < perror("bind"); close(nl_sock); return -1; >return nl_sock; > static int set_proc_ev_listen(int nl_sock, bool enable) < int rc; struct __attribute__ ((aligned(NLMSG_ALIGNTO))) < struct nlmsghdr nl_hdr; struct __attribute__ ((__packed__)) < struct cn_msg cn_msg; enum proc_cn_mcast_op cn_mcast; >; > nlcn_msg; memset(&nlcn_msg, 0, sizeof(nlcn_msg)); nlcn_msg.nl_hdr.nlmsg_len = sizeof(nlcn_msg); nlcn_msg.nl_hdr.nlmsg_pid = getpid(); nlcn_msg.nl_hdr.nlmsg_type = NLMSG_DONE; nlcn_msg.cn_msg.id.idx = CN_IDX_PROC; nlcn_msg.cn_msg.id.val = CN_VAL_PROC; nlcn_msg.cn_msg.len = sizeof(enum proc_cn_mcast_op); nlcn_msg.cn_mcast = enable ? PROC_CN_MCAST_LISTEN : PROC_CN_MCAST_IGNORE; rc = send(nl_sock, &nlcn_msg, sizeof(nlcn_msg), 0); if (rc == -1) < perror("netlink send"); return -1; >return 0; > static int handle_proc_ev(int nl_sock) < int rc; struct __attribute__ ((aligned(NLMSG_ALIGNTO))) < struct nlmsghdr nl_hdr; struct __attribute__ ((__packed__)) < struct cn_msg cn_msg; struct proc_event proc_ev; >; > nlcn_msg; while (!need_exit) < rc = recv(nl_sock, &nlcn_msg, sizeof(nlcn_msg), 0); if (rc == 0) < /* shutdown? */ return 0; >else if (rc == -1) < if (errno == EINTR) continue; perror("netlink recv"); return -1; >switch (nlcn_msg.proc_ev.what) < case PROC_EVENT_NONE: printf("set mcast listen ok\n"); break; case PROC_EVENT_FORK: printf("fork: parent tid=%d pid=%d ->child tid=%d pid=%d\n", nlcn_msg.proc_ev.event_data.fork.parent_pid, nlcn_msg.proc_ev.event_data.fork.parent_tgid, nlcn_msg.proc_ev.event_data.fork.child_pid, nlcn_msg.proc_ev.event_data.fork.child_tgid); break; case PROC_EVENT_EXEC: printf("exec: tid=%d pid=%d\n", nlcn_msg.proc_ev.event_data.exec.process_pid, nlcn_msg.proc_ev.event_data.exec.process_tgid); break; case PROC_EVENT_UID: printf("uid change: tid=%d pid=%d from %d to %d\n", nlcn_msg.proc_ev.event_data.id.process_pid, nlcn_msg.proc_ev.event_data.id.process_tgid, nlcn_msg.proc_ev.event_data.id.r.ruid, nlcn_msg.proc_ev.event_data.id.e.euid); break; case PROC_EVENT_GID: printf("gid change: tid=%d pid=%d from %d to %d\n", nlcn_msg.proc_ev.event_data.id.process_pid, nlcn_msg.proc_ev.event_data.id.process_tgid, nlcn_msg.proc_ev.event_data.id.r.rgid, nlcn_msg.proc_ev.event_data.id.e.egid); break; case PROC_EVENT_EXIT: printf("exit: tid=%d pid=%d exit_code=%d\n", nlcn_msg.proc_ev.event_data.exit.process_pid, nlcn_msg.proc_ev.event_data.exit.process_tgid, nlcn_msg.proc_ev.event_data.exit.exit_code); break; default: printf("unhandled proc event\n"); break; > > return 0; > static void on_sigint(__attribute__ ((unused)) int unused) < need_exit = true; >int main() < int nl_sock; int rc = EXIT_SUCCESS; signal(SIGINT, &on_sigint); siginterrupt(SIGINT, true); nl_sock = nl_connect(); if (nl_sock == -1) exit(EXIT_FAILURE); rc = set_proc_ev_listen(nl_sock, true); if (rc == -1) < rc = EXIT_FAILURE; goto out; >rc = handle_proc_ev(nl_sock); if (rc == -1) < rc = EXIT_FAILURE; goto out; >set_proc_ev_listen(nl_sock, false); out: close(nl_sock); exit(rc); > I don’t think however that you can you get process data such as UID and process arguments because exec_proc_event contains so little data: https://github.com/torvalds/linux/blob/v4.16/include/uapi/linux/cn_proc.h#L80 We could try to immediately read it from /proc , but there is a risk that the process finished and another one took its PID, so it wouldn’t be reliable.