How do I trace a system call in Linux?
How would I follow a system call from a trap to the kernel, to how arguments are passed, to how the system call in located in the kernel, to the actual processing of the system call in the kernel, to the return back to the user and how state is restored?
3 Answers 3
This is the most powerful method I’ve found so far. It can even show the call arguments: Does ftrace allow capture of system call arguments to the Linux kernel, or only function names?
sudo apt-get install systemtap sudo stap -e 'probe syscall.mkdir < printf("%s[%d] ->%s(%s)\n", execname(), pid(), name, argstr) >' sudo rm -rf /tmp/a /tmp/b mkdir /tmp/a mkdir /tmp/b mkdir[4590] -> mkdir("/tmp/a", 0777) mkdir[4593] -> mkdir("/tmp/b", 0777) Tested on Ubuntu 18.04, Linux kernel 4.15.
ltrace -S shows both system calls and library calls
This awesome tool therefore gives even further visibility into what executables are doing.
ftrace minimal runnable example
Mentioned at https://stackoverflow.com/a/29840482/895245 but here goes a minimal runnable example.
#!/bin/sh set -eux d=debug/tracing mkdir -p debug if ! mountpoint -q debug; then mount -t debugfs nodev debug fi # Stop tracing. echo 0 > "$/tracing_on" # Clear previous traces. echo > "$/trace" # Find the tracer name. cat "$/available_tracers" # Disable tracing functions, show only system call events. echo nop > "$/current_tracer" # Find the event name with. grep mkdir "$/available_events" # Enable tracing mkdir. # Both statements below seem to do the exact same thing, # just with different interfaces. # https://www.kernel.org/doc/html/v4.18/trace/events.html echo sys_enter_mkdir > "$/set_event" # echo 1 > "$/events/syscalls/sys_enter_mkdir/enable" # Start tracing. echo 1 > "$/tracing_on" # Generate two mkdir calls by two different processes. rm -rf /tmp/a /tmp/b mkdir /tmp/a mkdir /tmp/b # View the trace. cat "$/trace" # Stop tracing. echo 0 > "$/tracing_on" umount debug # tracer: nop # # _-----=> irqs-offhttps://sourceware.org/systemtap/documentation.html # / _----=> need-resched # | / _---=> hardirq/softirq # || / _--=> preempt-depth # ||| / delay # TASK-PID CPU# |||| TIMESTAMP FUNCTION # | | | |||| | | mkdir-5619 [005] . 10249.262531: sys_mkdir(pathname: 7fff93cbfcb0, mode: 1ff) mkdir-5620 [003] . 10249.264613: sys_mkdir(pathname: 7ffcdc91ecb0, mode: 1ff) One cool thing about this method is that it shows the function call for all processes on the system at once, although you can also filter PIDs of interest with set_ftrace_pid .
Tested on Ubuntu 18.04, Linux kernel 4.15.
GDB step debug the Linux kernel
Depending on the level of internals detail you need, this is an option: How to debug the Linux kernel with GDB and QEMU?
strace minimal runnable example
Here is a minimal runnable example of strace : How should strace be used? with a freestanding hello world, which makes how everything works perfectly clear.
perf top -F 49 -e raw_syscalls:sys_enter --sort comm,dso --show-nr-samples and the BPF-based traceloop: https://github.com/kinvolk/traceloop which the article claims to be a very fast method:
sudo -E ./traceloop cgroups --dump-on-exit /sys/fs/cgroup/system.slice/sshd.service How to Trace Program Execution Using Linux Strace Command
More often than not, when programmers and system administrators are trying to debug or troubleshoot issues with their compiled binaries and packages, you will, at some point, encounter errors, program crashes and exit status that will undoubtedly whack your brain out if you can’t find what’s causing the problem.
In this tutorial, we learn about strace command in Linux with usage examples.
Linux strace command
Strace is a diagnostics and debugging tool in Linux systems that is used to record and intercept system call names called by a running process and the signals which are received by the said running process. It can become a handy bug-isolation tool as well.
Strace, as a powerful tool, is mainly used to debug, troubleshoot and analyze how an application, program or binary file interacts with your Linux system. So if you are having trouble with your program or your compiled binary, you can use strace to check where the problem is being detected and which specific system call is being targeted.
The strace command can be used by non-root users as well in its basic usage. This versatility gives you in the user space the option to check non-privileged executables that just need to be examined without affecting system-wide changes.
strace [-ACdffhikqqrtttTvVwxxyyzZ] [-I n] [-b execve] [-e expr]. [-O overhead] [-S sortby] [-U columns] [-a column] [-o file] [-s strsize] [-X format] [-P path]. [-p pid]. [--seccomp-bpf]
Installation
In more recent Linux distros, strace is already pre-installed. However, if you find out that it has not yet been installed, you can execute the following commands based on your Linux distros:
For Debian / Ubuntu
For Redhat / CentOS stream
For Arch Linux