How Can I Create A Dump File of a Running Process in Linux?
I have a process that is spinning out of control under Linux, and I would like to create a dump file that I can take to my dev machine, and examine there. In Windows, it is possible to create a «minidump» of a running program in several different ways, including ADVPlus and Windows Task Manager, by going to the Processes tab and right-click selecting «Create Dump File.» Is there a way to accomplish this in Linux? I would need call stacks, heap and stack memory (especially stack), exceptions and all the rest.
@Valentin: From an educational perspective, yes, the replies were helpful and I upvoted them. However they did not answer the question I actually asked, which was how to construct a dump file which I can then take to a dev machine and examine. I was looking for something analogous to a Windows minidump file.
4 Answers 4
Well the way to create a dump file is:
gcore - Generate a core file for a running process SYNOPSIS gcore [-o filename] pid
might be the tools you are looking for.
pmap shows you an overview about the memory usage of the provided process. strace tracks down every action a process takes. With -f you tell strace to also consider watching over child processes and -o xxx tells strace to write the output to a file. You can also start a new process by using strace, e.g. with
If you are interested in specific information only, such as what files were opened, you can start strace accordingly:
strace -f -o xxx -e trace=open -p
Meanwhile ProcDump from the Sysinternals suite has also been made available under the very liberal MIT license from the respective GitHub page.
Usage: procdump [OPTIONS. ] TARGET OPTIONS -C CPU threshold at which to create a dump of the process from 0 to 100 * nCPU -c CPU threshold below which to create a dump of the process from 0 to 100 * nCPU -M Memory commit threshold in MB at which to create a dump -m Trigger when memory commit drops below specified MB value. -n Number of dumps to write before exiting -s Consecutive seconds before dump is written (default is 10) TARGET must be exactly one of these: -p pid of the process So as you can deduce from the command line arguments, it’s easy to take «snapshots» of a process you know misbehaves by taking up undue amounts of resources to be later analyzed with gdb or so.
This ProcDump for Linux is, however, not feature-complete in comparison with its Windows cousin.
Core dump
A core dump is a file containing a process’s address space (memory) when the process terminates unexpectedly. Core dumps may be produced on-demand (such as by a debugger), or automatically upon termination. Core dumps are triggered by the kernel in response to program crashes, and may be passed to a helper program (such as systemd-coredump(8) ) for further processing. A core dump is not typically used by an average user, but developers could use it as a post-mortem snapshot of the program’s state at the time of the crash, especially if the fault is hard to reliably reproduce.
Warning: Core dumps should be shared only with trusted parties as they may contain sensitive data (such as passwords or cryptographic keys).
Disabling automatic core dumps
Users may wish to disable automatic core dumps for a number of reasons:
- Performance: generating core dumps for memory-heavy processes can waste system resources and delay the cleanup of memory.
- Disk space: core dumps of memory-heavy processes may consume disk space equal to, if not greater, than the process’s memory footprint if not compressed.
- Security: core dumps, although typically readable only by root, may contain sensitive data (such as passwords or cryptographic keys), which are written to disk following a crash.
Using sysctl
sysctl can be used to set the kernel.core_pattern to nothing to disable core dump handling. Create this file
kernel.core_pattern=/dev/null
To apply the setting immediately, use sysctl :
# sysctl -p /etc/sysctl.d/50-coredump.conf
Using systemd
systemd’s default behavior is defined in /usr/lib/sysctl.d/50-coredump.conf , which sets kernel.core_pattern to call systemd-coredump . It generates core dumps for all processes in /var/lib/systemd/coredump . systemd-coredump behavior can be overridden by creating a configuration snippet in the /etc/systemd/coredump.conf.d/ directory with the following content (See coredump.conf(5) § DESCRIPTION , [1]):
/etc/systemd/coredump.conf.d/custom.conf
Note: Do not forget to include the [Coredump] section name, otherwise this option will be ignored: systemd-coredump[1728]: [/etc/systemd/coredump.conf.d/custom.conf:1] Assignment outside of section. Ignoring.
Then reload the systemd manager configuration with daemon-reload.
This method alone is usually sufficient to disable userspace core dumps, so long as no other programs enable automatic core dumps on the system, but the coredump is still generated in memory and systemd-coredump run.
If you want to completely disable all coredump handling except for a log entry, set the ProcessSizeMax=0 option. See systemd-coredump(8) § Disabling coredump processing .
Using PAM limits
The maximum core dump size for users logged in via PAM is enforced by limits.conf. Setting it to zero disables core dumps entirely. [2]
Using ulimit
Command-line shells such as bash or zsh provide a builtin ulimit command which can be used to report or set resource limits of the shell and the processes started by the shell. See bash(1) § SHELL BUILTIN COMMANDS or zshbuiltins(1) for details.
To disable core dumps in the current shell:
If the system is setup to pipe coredumps into a program such as systemd-coredump using kernel.core_pattern , Linux ignores the ulimit setting and only the dumpable prctl(2) can be used to disable coredump processing for selected processes.
Making a core dump
To generate a core dump of an arbitrary process, first install the gdb package. Then find the PID of the running process, for example with pgrep:
(gdb) generate-core-file Saved corefile core.2071 (gdb) quit
Now you have a coredump file called core.2071 .
Where do they go?
The kernel.core_pattern sysctl decides where automatic core dumps go. By default, core dumps are sent to systemd-coredump which can be configured in /etc/systemd/coredump.conf . By default, all core dumps are stored in /var/lib/systemd/coredump (due to Storage=external ) and they are compressed with zstd (due to Compress=yes ). Additionally, various size limits for the storage can be configured.
Note: The default value for kernel.core_pattern is set in /usr/lib/sysctl.d/50-coredump.conf . This file may be masked or overridden to use a different setting following normal sysctl.d(5) rules.
To retrieve a core dump from the journal, see coredumpctl(1) .
Examining a core dump
Use coredumpctl to find the corresponding dump:
You need to uniquely identify the relevant dump. This is possible by specifying a PID , name of the executable, path to the executable or a journalctl predicate (see coredumpctl(1) and journalctl(1) for details). To see details of the core dumps:
# coredumpctl info match
Pay attention to «Signal» row, that helps to identify crash cause. For deeper analysis you can examine the backtrace using gdb:
# coredumpctl gdb match
When gdb is started, use the bt command to print the backtrace:
See Debugging/Getting traces if debugging symbols are requested, but not found.
Cleanup of core dump files
The core dump files stored in /var/lib/systemd/coredump/ will be automatically cleaned by systemd-tmpfiles —clean , which is triggered daily with systemd-tmpfiles-clean.timer . Core dumps are configured to persist for at least 3 days, see systemd-tmpfiles —cat-config .
See also
- american fuzzy lop — A tool for automated tests of the kernel and programs
- Filesystem fuzzing — LWN article about testing filesystems for bugs