Stracing to attach to a multi-threaded process
If I want to strace a multi-threaded process (of all of its threads), how should I do it? I know that one can do strace -f to follow forked process? But how about attaching to a process which is already multi-threaded when I start stracing? Is a way to tell strace to trace all of system calls of all the threads which belong to this process?
The same strace -f is sufficient (but I don’t know how to prevent tracing child processes when you trace all threads this way).
I can confirm that strace -fp
2 Answers 2
2021 update
strace -fp PID just does the right thing on my system (Ubuntu 20.04.1 LTS). The strace manual page points this out:
-f Trace child processes as they are created by currently traced processes as a result of the fork(2), vfork(2) and clone(2) system calls. Note that -p PID -f will attach all threads of process PID if it is multi-threaded, not only thread with thread_id = PID. Looks like this text was added back in 2013. If -f had this behavior on my system at the time, I didn’t realize it. It does now, though!
Original 2013 answer
I just did this in a kludgy way, by listing each tid to be traced.
You can find them through ps :
$ ps auxw -T | fgrep program_to_trace me pid tid1 . me pid tid2 . me pid tid3 . me pid tid4 . and then, according to man strace , you can attach to multiple pids at once:
-p pid Attach to the process with the process ID pid and begin tracing. The trace may be terminated at any time by a keyboard interrupt signal (CTRL-C). strace will respond by detaching itself from the traced process(es) leaving it (them) to continue running. Mul‐ tiple -p options can be used to attach to up to 32 processes in addition to command (which is optional if at least one -p option is given). It says pid , but iirc on Linux the pid and tid share the same namespace, and this appeared to work:
$ strace -f -p tid1 -p tid2 -p tid3 -p tid4 I think that might be the best you can do for now. But I suppose someone could extend strace with a flag for expanding tids. There would probably still be a race between finding the processes and attaching to them in which a freshly started one would be missed. It’d fit in with the existing caveat about strace -f :
-f Trace child processes as they are created by currently traced processes as a result of the fork(2) system call. On non-Linux platforms the new process is attached to as soon as its pid is known (through the return value of fork(2) in the par‐ ent process). This means that such children may run uncontrolled for a while (especially in the case of a vfork(2)), until the par‐ ent is scheduled again to complete its (v)fork(2) call. On Linux the child is traced from its first instruction with no delay. If the parent process decides to wait(2) for a child that is currently being traced, it is suspended until an appropriate child process either terminates or incurs a signal that would cause it to terminate (as determined from the child's current signal dispo‐ sition). On SunOS 4.x the tracing of vforks is accomplished with some dynamic linking trickery. Is it possible to read the output from any process using its PID?
Is it possible to attach a terminal to an already running process by using its PID in a similar fashion to using the fg command for jobs?
3 Answers 3
You can get that process’s standard file descriptors, e.g. stdout :
I tried this solution with top and it doesn’t seem to work. In a terminal I launched top and in another I used your command with the corred pid but got no output.
what about stderr? I assume /proc/
What if this descriptor points to a socket? Tail can’t read it. # ls -la /proc/24510/fd/1 lrwx—— 1 root root 64 Oct 31 08:34 /proc/24510/fd/1 -> socket:[444026]
I had to cat /proc/
There are a few options here. One is to redirect the output of the command to a file, and then use tail to view new lines that are added to that file in real time.
Another option is to launch your program inside of screen , which is a sort-of text-based terminal application. Screen sessions can be attached and detached, but are nominally meant only to be used by the same user, so if you want to share them between users.
Else if you wish you can trace the process and see what it is doing with strace :
strace -e trace=open -p 22254 -s 80 -o output.txt - -p PID : Attach to the process with the process ID PID and begin tracing.
- -s SIZE : Specify the maximum string size to print (the default is 32).
- -o filename : Write the trace output to the file filename rather than to screen (stderr).
How do I attach a terminal to a detached process?
That terminal is now long closed, but process is still running, and I want to send some commands to that process’s stdin. Is that possible?
Easiest way (if you are still in same terminal) is to run jobs (to see, if process is still running) and if yes, use fg to being it to foreground. After that, you can start sending commands and you will also receive stdout data. PS: «sending it to background again» can be done using CTRL+Z (suspend) and than running bg (run last job in background). See some tutorials for this topic to learn more.
5 Answers 5
Yes, it is. First, create a pipe: mkfifo /tmp/fifo . Use gdb to attach to the process: gdb -p PID
Then close stdin: call close (0) ; and open it again: call open («/tmp/fifo», 0600)
Finally, write away (from a different terminal, as gdb will probably hang):
@rustyx: Untested, but this should work: create a file rather than a pipe, touch /tmp/thefile . Stdout is 1, so call close (1) ; also, use the correct permissions for writing: call open («/tmp/thefile», 0400) . The echo… is, of course, not needed.
This is great! I’m using this to send «y» or «n» responses to certain processes that have been completely detached. The detached process has its stdout to a separate window. When I do this trick however, I can see that it does not «receive» the ‘y’ or ‘n’ as soon as I echo it, I must quit gdb and detach it and then it receives all of the echos accordingly, so is there a way to perform this without needing to quit gdb before the process receives the input from the fifo?
When original terminal is no longer accessible.
Have a look at reptyr, which does exactly that. The github page has all the information.
reptyr — A tool for «re-ptying» programs.
reptyr is a utility for taking an existing running program and attaching it to a new terminal. Started a long-running process over ssh, but have to leave and don’t want to interrupt it? Just start a screen, use reptyr to grab it, and then kill the ssh session and head on home.
USAGE
reptyr PID
«reptyr PID» will grab the process with id PID and attach it to your current terminal.
After attaching, the process will take input from and write output to the new terminal, including ^C and ^Z. (Unfortunately, if you background it, you will still have to run «bg» or «fg» in the old terminal. This is likely impossible to fix in a reasonable way without patching your shell.)
Edit claims that » reptyr cannot grab a process which has subprocesses. Or the subprocess (reptyr version 0.6.2).» Limited support does exist Issue, Issue