How do I trace a system call in Linux?
How would I follow a system call from a trap to the kernel, to how arguments are passed, to how the system call in located in the kernel, to the actual processing of the system call in the kernel, to the return back to the user and how state is restored?
3 Answers 3
This is the most powerful method I’ve found so far. It can even show the call arguments: Does ftrace allow capture of system call arguments to the Linux kernel, or only function names?
sudo apt-get install systemtap sudo stap -e 'probe syscall.mkdir < printf("%s[%d] ->%s(%s)\n", execname(), pid(), name, argstr) >' sudo rm -rf /tmp/a /tmp/b mkdir /tmp/a mkdir /tmp/b mkdir[4590] -> mkdir("/tmp/a", 0777) mkdir[4593] -> mkdir("/tmp/b", 0777) Tested on Ubuntu 18.04, Linux kernel 4.15.
ltrace -S shows both system calls and library calls
This awesome tool therefore gives even further visibility into what executables are doing.
ftrace minimal runnable example
Mentioned at https://stackoverflow.com/a/29840482/895245 but here goes a minimal runnable example.
#!/bin/sh set -eux d=debug/tracing mkdir -p debug if ! mountpoint -q debug; then mount -t debugfs nodev debug fi # Stop tracing. echo 0 > "$/tracing_on" # Clear previous traces. echo > "$/trace" # Find the tracer name. cat "$/available_tracers" # Disable tracing functions, show only system call events. echo nop > "$/current_tracer" # Find the event name with. grep mkdir "$/available_events" # Enable tracing mkdir. # Both statements below seem to do the exact same thing, # just with different interfaces. # https://www.kernel.org/doc/html/v4.18/trace/events.html echo sys_enter_mkdir > "$/set_event" # echo 1 > "$/events/syscalls/sys_enter_mkdir/enable" # Start tracing. echo 1 > "$/tracing_on" # Generate two mkdir calls by two different processes. rm -rf /tmp/a /tmp/b mkdir /tmp/a mkdir /tmp/b # View the trace. cat "$/trace" # Stop tracing. echo 0 > "$/tracing_on" umount debug # tracer: nop # # _-----=> irqs-offhttps://sourceware.org/systemtap/documentation.html # / _----=> need-resched # | / _---=> hardirq/softirq # || / _--=> preempt-depth # ||| / delay # TASK-PID CPU# |||| TIMESTAMP FUNCTION # | | | |||| | | mkdir-5619 [005] . 10249.262531: sys_mkdir(pathname: 7fff93cbfcb0, mode: 1ff) mkdir-5620 [003] . 10249.264613: sys_mkdir(pathname: 7ffcdc91ecb0, mode: 1ff) One cool thing about this method is that it shows the function call for all processes on the system at once, although you can also filter PIDs of interest with set_ftrace_pid .
Tested on Ubuntu 18.04, Linux kernel 4.15.
GDB step debug the Linux kernel
Depending on the level of internals detail you need, this is an option: How to debug the Linux kernel with GDB and QEMU?
strace minimal runnable example
Here is a minimal runnable example of strace : How should strace be used? with a freestanding hello world, which makes how everything works perfectly clear.
perf top -F 49 -e raw_syscalls:sys_enter --sort comm,dso --show-nr-samples and the BPF-based traceloop: https://github.com/kinvolk/traceloop which the article claims to be a very fast method:
sudo -E ./traceloop cgroups --dump-on-exit /sys/fs/cgroup/system.slice/sshd.service