How to runtime debug shared libraries?
Can anyone tell me how to do runtime debugging on shared libraries? I need to runtime-debug a function in my shared library, but its called by another program. How can I do something like dbx with shared libraries? I m using dbx on AIX. is gdb better than dbx for what I m trying to do?.
6 Answers 6
You just need to call gdb with the executable (it does not matter if it is yours or a 3rd party one). Here is an example where I debug the ls command and set a breakpoint in the (shared) c library. This example uses gdb 6.8 which supports deferred (pending) breakpoints which makes this easy:
gdb /bin/ls GNU gdb 6.8-debian Copyright (C) 2008 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". (no debugging symbols found) (gdb) b write Function "write" not defined. Make breakpoint pending on future shared library load? (y or [n]) y Breakpoint 1 (write) pending. (gdb) r Starting program: /bin/ls (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) [Thread debugging using libthread_db enabled] (no debugging symbols found) (no debugging symbols found) [New Thread 0x7f98d2d23780 (LWP 7029)] [Switching to Thread 0x7f98d2d23780 (LWP 7029)] Breakpoint 1, 0x00007f98d2264bb0 in write () from /lib/libc.so.6 (gdb) As you can see gdb automatically manages all threads used by the executable. You don’t have to do anything special for threads there. The breakpoint will work in any thread.
Alternatively if you want to attach the debugger to an already running application (I use tail -f /tmp/ttt here as an example):
ps ux | grep tail lothar 8496 0.0 0.0 9352 804 pts/3 S+ 12:38 0:00 tail -f /tmp/ttt lothar 8510 0.0 0.0 5164 840 pts/4 S+ 12:39 0:00 grep tail gdb GNU gdb 6.8-debian Copyright (C) 2008 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". (no debugging symbols found) (gdb) attach 8496 Attaching to program: /usr/bin/tail, process 8496 Reading symbols from /lib/librt.so.1. (no debugging symbols found). done. Loaded symbols for /lib/librt.so.1 Reading symbols from /lib/libc.so.6. (no debugging symbols found). done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib/libpthread.so.0. (no debugging symbols found). done. [Thread debugging using libthread_db enabled] [New Thread 0x7f24853f56e0 (LWP 8496)] Loaded symbols for /lib/libpthread.so.0 Reading symbols from /lib/ld-linux-x86-64.so.2. (no debugging symbols found). done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 (no debugging symbols found) 0x00007f2484d2bb50 in nanosleep () from /lib/libc.so.6 (gdb) b write Breakpoint 1 at 0x7f2484d57bb0 (gdb) c Continuing. [Switching to Thread 0x7f24853f56e0 (LWP 8496)] Breakpoint 1, 0x00007f2484d57bb0 in write () from /lib/libc.so.6 (gdb) Using standard C++ library debug symbols? Ubuntu / Linux / libstdc++6-8-dbg?
There is a package called libstdc++6-8-dbg on Ubuntu Linux (latest version at time of writing). It is described as:
GNU Standard C++ Library v3 (debugging files) This package contains the shared library of libstdc++ compiled with debugging symbols.
/usr/lib/x86_64-linux-gnu/debug/libstdc++.a /usr/lib/x86_64-linux-gnu/debug/libstdc++.so.6.0.25 /usr/lib/x86_64-linux-gnu/debug/libstdc++fs.a But this doesn’t include the debug versions of the standard library. What extra options do you need to pass to g++ to tell it to use the debug versions of the standard library provided by libstdc++6-8-dbg ?
What’s your goal? Do you want access to the debug symbols when debugging, would that be sufficient? Or maybe you just need them available at runtime? I’d consider statically linking a debug version of libstdc++ a measure of last resort.
@JohnKugelman: I want backtraces to correctly symbolize libstdc++ call frames at run-time. If you know the g++ command-line for (1) dynamically linking the debug version and (2) statically linking the debug version, I’ll take both thanks.
@JohnKugelman: gdb isn’t in play. I mean at run-time of the program I want it to use the debug versions of libstdc++ so that a stacktrace will correctly symbolize any libstdc++ calls within.
What do the numbers of libstdc++6-8-dbg refer to? On my system I am drawn to libstdc++6-10-dbg , but have GCC version 9.3.0 and libstdc++ version 3.4.
2 Answers 2
The OP wants to resolve C++ standard library’s symbols in the backtrace correctly. John’s answer correctly states that this could be achieved by linking against the debug version of the standard libraries.
However, Ubuntu also provides debug symbol packages that, once installed, allow GDB to resolve symbols in standard libraries whose debug symbols have been stripped i.e. in the release version of the standard library. We provide a how-to example below (I’m using Ubuntu 20.04):
Suppose the generated binary is named a.out. We first find the version of libstdc++ it links against:
$ ldd a.out . libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007ff8dc6f7000) . We search for the package that provides the shared library file (/lib is a symbolic link to /usr/lib. The full path must be used here.):
$ dpkg -S /usr/lib/x86_64-linux-gnu/libstdc++.so.6 libstdc++6:amd64: /usr/lib/x86_64-linux-gnu/libstdc++.so.6 Follow the instructions to add the repo for debug symbol packages and then update the package index. The link also describes how one may search for debug symbol packages, but I directly searched with the package name:
$ apt list libstdc++6\* . libstdc++6-dbgsym/focal-updates 10.2.0-5ubuntu1~20.04 amd64 . There will be tons of results, but be sure to look out for dbgsym , rather than dbg ! After installing libstdc++6-dbgsym , GDB should be able to resolve the symbols, even if your binary isn’t linked against the debug library.
The texts above should answer the OP’s question. Now I point out an issue with John’s answer.
GDB automatically reads in the debug symbols once you’ve installed the package. You don’t need to compile your program any differently.
This statement is 100% correct, but the figure included is irrelevant and doesn’t prove the statement. There are three closely-releated concepts at play here:
- Debug library package: The package libstdc++6-8-dbg provides a version of the libstdc++ library with debug symbols.
- Debug symbol pagkage: The package libstdc++6-dbgsym provides the debug symbols for the libstdc++ library. That is, it doesn’t include any machine instructions for functions like printf , only the debug symbols. libstdc++6-8-dbg bundles code and debug symbols together into one library.
- Release library package: The packages libstdc++6-8 and libstdc++6 provide the release versions of the standard library, meaning that they don’t carry debug symbols, only the code. Debug symbol packages should be used together with the release libraries.
GDB will automatically read the debug symbols in the debug symbol package, but not in the debug library. The auto-load in John’s figure simply states that the Python script /usr/share/gdb/auto-load/usr/lib/x86_64-linux-gnu/debug/libstdc++.so.6.0.25-gdb.py will be run automatically when the debug library is loaded and has nothing to do with the auto-loading of debug symbols.
Mistobaan / LD_LOAD_CHEATSHEET.md
These are a set of commands that are useful to debug why a library is not loading.
Find all the libraries discoverable in your LD_LIBRARY_PATH
$/sbin/ldconfig -N -v $(sed 's/:/ /g' libnvjpeg.so.10.0.130 libnvrtc-builtins.so.10.0 -> libnvrtc-builtins.so.10.0.130 libnppisu.so.10.0 -> libnppisu.so.10.0.130 libcufft.so.10.0 -> libcufft.so.10.0.145 libaccinj64.so.10.0 -> libaccinj64.so.10.0.130 libcusolver.so.10.0 -> libcusolver.so.10.0.130 . Find all the symbols in your binary file that are not defined
!readelf -Ws ~/anaconda3/lib/python3.7/site-packages/_torch_sox.cpython-37m-x86_64-linux-gnu.so | grep "UND" 13: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND snd_pcm_close 40: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND snd_pcm_open 49: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND snd_pcm_format_description 57: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND snd_pcm_hw_params_set_buffer_size_near 68: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND snd_pcm_hw_params_set_period_size_near 88: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND snd_pcm_drain 102: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND snd_pcm_format_mask_malloc 133: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND snd_pcm_hw_params 134: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND snd_pcm_hw_params_set_format 152: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND snd_pcm_hw_params_set_rate_resample 156: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND snd_pcm_hw_params_malloc 166: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND snd_pcm_prepare 179: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND snd_pcm_resume 187: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND snd_pcm_hw_params_get_buffer_size_min 211: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND snd_pcm_hw_params_any 223: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND snd_strerror 252: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND snd_pcm_format_mask_free 263: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND snd_pcm_writei 276: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND snd_pcm_hw_params_set_channels_near 281: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND snd_pcm_hw_params_set_access if you are getting UND in the binary then probably you are missing a -L directive when linking it.
Check what library is dynamically linking it to
Sometime it happens that you have different Versions of the libraries in your path. ldd will tell you what exact library is loading.
ldd ~/anaconda3/lib/python3.7/site-packages/_torch_sox.cpython-37m-x86_64-linux-gnu.so linux-vdso.so.1 (0x00007fff8c7a6000) libstdc++.so.6 => ~/anaconda3/lib/libstdc++.so.6 (0x00007f66c27e1000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f66c216d000) libgcc_s.so.1 => ~/anaconda3/lib/libgcc_s.so.1 (0x00007f66c2792000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f66c1f4e000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f66c1b5d000) /lib64/ld-linux-x86-64.so.2 (0x00007f66c2730000) Add a new path to LD_LIBRARY_PATH
Ideally we would like to do this:
export LD_LIBRARY_PATH=//:$LD_LIBRARY_PATH but not always works. The most consistent way to do it is to add a new file in the /etc/ld.so.conf/ directory.
sudo echo "" >> /etc/ld.so.conf.d/new_path.conf and then RELOAD the path with the following command.
You must run this command or is not gonna work. Check if the library is discovered using the first tip.