How to debug the Linux kernel with GDB and QEMU?
I’m new to kernel development and I would like to know how to run/debug the linux kernel using QEMU and gdb. I’m actually reading Robert Love’s book but unfortunately it doesn’t help the reader on how to install proper tools to run or debug the kernel. So what I did was to follow this tutorial http://opensourceforu.efytimes.com/2011/02/kernel-development-debugging-using-eclipse/. I’m using eclipse as an IDE to develop on the kernel but I wanted first to get it work under QEMU/gdb. So what I did so far was: 1) To compile the kernel with:
make defconfig (then setting the CONFIG_DEBUG_INFO=y in the .config) make -j4 qemu-system-x86_64 -s -S /dev/zero -kernel /arch/x86/boot/bzImage which run it correctly but. Now I don’t know what to do. I know that I have to use remote debugging on the port 1234 (default port used by Qemu), using the vmlinux as the symbol table file for debugging. So my question is: What should I do to run the kernel on Qemu, attach my debugger to it and thus, get them work together to make my life easier with kernel development.
Could you explain that /dev/zero argument in the qemu command above? I tried that (for aarch64, with some other options) but qemu hangs and I had to kill the kernel. I know /dev/zero outputs zero when read but don’t know how it is used here.
6 Answers 6
(gdb) target remote localhost:1234 (gdb) continue Using the ‘-s’ option makes qemu listen on port tcp::1234, which you can connect to as localhost:1234 if you are on the same machine. Qemu’s ‘-S’ option makes Qemu stop execution until you give the continue command.
Best thing would probably be to have a look at a decent GDB tutorial to get along with what you are doing. This one looks quite nice.
Thanks a lot it works :). I just finished to read the book dealing with DDD, eclipse and gdb published by not starch press but there were no remote debugging in this book. My kernel is now launching but it seems to take time to load (as Qemu seems to use only 1 thread on my machine) and is now blocked at: ? kernel_thread_helper+0x0/0x10. Is it the way the kernel use to be loaded? I mean, shouldn’t we have a command prompt once it’s loaded? Thank you
it works for me. However, I don’t know how to force kernel stop on breakpoint after calling the first continue. For example, I put a breakpoint in start_kernel function but it will never stop there. Any idea ?
Isn’t it problematic that qemu and host are using the same interfaces ? for example if it is a serial gps than host might already use this serial interface.
Step-by-step procedure tested on Ubuntu 16.10 host
To get started from scratch quickly I’ve made a minimal fully automated QEMU + Buildroot example at: https://github.com/cirosantilli/linux-kernel-module-cheat/blob/c7bbc6029af7f4fab0a23a380d1607df0b2a3701/gdb-step-debugging.md Major steps are covered below.
First get a root filesystem rootfs.cpio.gz . If you need one, consider:
git checkout v4.15 make mrproper make x86_64_defconfig cat .config-fragment CONFIG_DEBUG_INFO=y CONFIG_DEBUG_KERNEL=y CONFIG_GDB_SCRIPTS=y EOF ./scripts/kconfig/merge_config.sh .config .config-fragment make -j"$(nproc)" qemu-system-x86_64 -kernel arch/x86/boot/bzImage \ -initrd rootfs.cpio.gz -S -s \ -append nokaslr On another terminal, from inside the Linux kernel tree, supposing you want to start debugging from start_kernel :
gdb \ -ex "add-auto-load-safe-path $(pwd)" \ -ex "file vmlinux" \ -ex 'set arch i386:x86-64:intel' \ -ex 'target remote localhost:1234' \ -ex 'break start_kernel' \ -ex 'continue' \ -ex 'disconnect' \ -ex 'set arch i386:x86-64' \ -ex 'target remote localhost:1234' For Ubuntu 14.04, GDB 7.7.1, hbreak was needed, break software breakpoints were ignored. Not the case anymore in 16.10. See also: https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/901944
The messy disconnect and what come after it are to work around the error:
Remote 'g' packet reply is too long: 000000000000000017d11000008ef4810120008000000000fdfb8b07000000000d352828000000004040010000000000903fe081ffffffff883fe081ffffffff00000000000e0000ffffffffffe0ffffffffffff07ffffffffffffffff9fffff17d11000008ef4810000000000800000fffffffff8ffffffffff0000ffffffff2ddbf481ffffffff4600000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000007f0300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000801f0000 - https://sourceware.org/bugzilla/show_bug.cgi?id=13984 might be a GDB bug
- Remote ‘g’ packet reply is too long
- http://wiki.osdev.org/QEMU_and_GDB_in_long_mode osdev.org is as usual an awesome source for these problems
- https://lists.nongnu.org/archive/html/qemu-discuss/2014-10/msg00069.html
- nokaslr : https://unix.stackexchange.com/questions/397939/turning-off-kaslr-to-debug-linux-kernel-using-qemu-and-gdb/421287#421287
- the Linux kernel does not support (and does not even compile without patches) with -O0 : How to de-optimize the Linux kernel to and compile it with -O0?
- GDB 7.11 will blow your memory on some types of tab completion, even after the max-completions fix: Tab completion interrupt for large binaries Likely some corner case which was not covered in that patch. So an ulimit -Sv 500000 is a wise action before debugging. Blew up specifically when I tab completed file for the filename argument of sys_execve as in: https://stackoverflow.com/a/42290593/895245
When you try to start vmlinux exe using gdb, then first thing on gdb is to issue cmds:
(gdb) target remote localhost:1234
This will break the kernel at start_kernel.
I use eclipse to debug kernel run in qemu,and setted stop at start_kernel.But it stil run after eclipse start debug. I have set qemu stopped when it start and use gdb alone is ok.
Isn’t it problematic that qemu and host are using the same interfaces ? for example if it is a serial gps than host might already use this serial interface.
BjoernID’s answer did not really work for me. After the first continuation, no breakpoint is reached and on interrupt, I would see lines such as:
0x0000000000000000 in ?? () (gdb) break rapl_pmu_init Breakpoint 1 at 0xffffffff816631e7 (gdb) c Continuing. ^CRemote 'g' packet reply is too long: 08793000000000002988d582000000002019[..] I guess this has something to do with different CPU modes (real mode in BIOS vs. long mode when Linux has booted). Anyway, the solution is to run QEMU first without waiting (i.e. without -S ):
qemu-system-x86_64 -enable-kvm -kernel arch/x86/boot/bzImage -cpu SandyBridge -s In my case, I needed to break at something during boot, so after some deciseconds, I ran the gdb command. If you have more time (e.g. you need to debug a module that is loaded manually), then the timing doesn’t really matter.
gdb allows you to specify commands that should be run when started. This makes automation a bit easier. To connect to QEMU (which should now already be started), break on a function and continue execution, use:
gdb -ex 'target remote localhost:1234' -ex 'break rapl_pmu_init' -ex c ./vmlinux Isn’t it problematic that qemu and host are using the same interfaces ? for example if it is a serial gps than host might already use this serial interface.
@ransh Sorry I don’t understand your comment. There is no serial device involved when using the qemu -s option, it uses a TCP socket instead.
@ransh The guest is not affected by the debugger connection, that is done at the host level, completely outside the guest. None of the guest’s virtual hardware is involved.
As for me the best solution for debugging the kernel — is to use gdb from Eclipse environment. You should just set appropriate port for gdb (must be the same with one you specified in qemu launch string) in remote debugging section. Here is the manual: http://www.sw-at.com/blog/2011/02/11/linux-kernel-development-and-debugging-using-eclipse-cdt/
On Linux systems, vmlinux is a statically linked executable file that contains the Linux kernel in one of the object file formats supported by Linux, which includes ELF, COFF and a.out. The vmlinux file might be required for kernel debugging, symbol table generation or other operations, but must be made bootable before being used as an operating system kernel by adding a multiboot header, bootsector and setup routines.
An image of this initial root file system must be stored somewhere accessible by the Linux bootloader to the boot firmware of the computer. This can be the root file system itself, a boot image on an optical disc, a small partition on a local disk (a boot paratition, usually using ext4 or FAT file systems), or a TFTP server (on systems that can boot from Ethernet).
- Compile linux kernel Build the kernel with this series applied, enabling CONFIG_DEBUG_INFO (but leave CONFIG_DEBUG_INFO_REDUCED off)
- https://www.kernel.org/doc/html/latest/admin-guide/README.html
- https://wiki.archlinux.org/index.php/Kernel/Traditional_compilation
- https://lwn.net/Articles/533552/
- Install GDB and Qemu
#!/bin/bash # Os : Arch Linux # Kernel : 5.0.3 INIT_DIR=$(pwd) BBOX_URL="https://busybox.net/downloads/busybox-1.30.1.tar.bz2" BBOX_FILENAME=$(basename $) BBOX_DIRNAME=$(basename $ ".tar.bz2") RAM_FILENAME="$/initramfs.cpio.gz" function download_busybox < wget -c $2>/dev/null > function compile_busybox < tar xvf $&& cd "$/$/" echo "[*] Settings > Build options > Build static binary (no shared libs)" echo "[!] Please enter to continue" read tmpvar make menuconfig && make -j2 && make install > function config_busybox < cd "$/$/" rm -rf initramfs/ && cp -rf _install/ initramfs/ rm -f initramfs/linuxrc mkdir -p initramfs/ sudo cp -a /dev/ initramfs/dev/ cat > "$/$/initramfs/init" << EOF #!/bin/busybox sh mount -t proc none /proc mount -t sysfs none /sys exec /sbin/init EOF chmod a+x initramfs/init cd "$/$/initramfs/" find . -print0 | cpio --null -ov --format=newc | gzip -9 > "$" echo "[*] output: $" > download_busybox compile_busybox config_busybox #!/bin/bash KER_FILENAME="/home/debug/Projects/kernelbuild/linux-5.0.3/arch/x86/boot/bzImage" RAM_FILENAME="/home/debug/Projects/kerneldebug/initramfs.cpio.gz" qemu-system-x86_64 -s -kernel "$" -initrd "$" -nographic -append "console=ttyS0" $ ./qemuboot_vmlinux.sh SeaBIOS (version 1.12.0-20181126_142135-anatol) iPXE (http://ipxe.org) 00:03.0 C980 PCI2.10 PnP PMM+07F92120+07EF2120 C980 Booting from ROM. Probing EDD (edd=off to disable). o [ 0.019814] Spectre V2 : Spectre mitigation: LFENCE not serializing, switching to generic retpoline can't run '/etc/init.d/rcS': No such file or directory Please press Enter to activate this console. / # uname -a Linux archlinux 5.0.3 #2 SMP PREEMPT Mon Mar 25 10:27:13 CST 2019 x86_64 GNU/Linux / # ~/Projects/kernelbuild/linux-5.0.3 ➭ gdb vmlinux . (gdb) target remote localhost:1234 Remote debugging using localhost:1234 0xffffffff89a4b852 in ?? () (gdb) break start_kernel Breakpoint 1 at 0xffffffff826ccc08 (gdb) Display all 190 possibilities? (y or n) (gdb) info functions All defined functions: Non-debugging symbols: 0xffffffff81000000 _stext 0xffffffff81000000 _text 0xffffffff81000000 startup_64 0xffffffff81000030 secondary_startup_64 0xffffffff810000e0 verify_cpu 0xffffffff810001e0 start_cpu0 0xffffffff810001f0 __startup_64 0xffffffff81000410 pvh_start_xen 0xffffffff81001000 hypercall_page 0xffffffff81001000 xen_hypercall_set_trap_table 0xffffffff81001020 xen_hypercall_mmu_update 0xffffffff81001040 xen_hypercall_set_gdt 0xffffffff81001060 xen_hypercall_stack_switch 0xffffffff81001080 xen_hypercall_set_callbacks 0xffffffff810010a0 xen_hypercall_fpu_taskswitch 0xffffffff810010c0 xen_hypercall_sched_op_compat 0xffffffff810010e0 xen_hypercall_platform_op