Jan Kiszka | bda1a92 | 2015-02-17 13:47:32 -0800 | [diff] [blame] | 1 | Debugging kernel and modules via gdb |
| 2 | ==================================== |
| 3 | |
| 4 | The kernel debugger kgdb, hypervisors like QEMU or JTAG-based hardware |
| 5 | interfaces allow to debug the Linux kernel and its modules during runtime |
| 6 | using gdb. Gdb comes with a powerful scripting interface for python. The |
| 7 | kernel provides a collection of helper scripts that can simplify typical |
| 8 | kernel debugging steps. This is a short tutorial about how to enable and use |
| 9 | them. It focuses on QEMU/KVM virtual machines as target, but the examples can |
| 10 | be transferred to the other gdb stubs as well. |
| 11 | |
| 12 | |
| 13 | Requirements |
| 14 | ------------ |
| 15 | |
| 16 | o gdb 7.2+ (recommended: 7.4+) with python support enabled (typically true |
| 17 | for distributions) |
| 18 | |
| 19 | |
| 20 | Setup |
| 21 | ----- |
| 22 | |
| 23 | o Create a virtual Linux machine for QEMU/KVM (see www.linux-kvm.org and |
| 24 | www.qemu.org for more details). For cross-development, |
| 25 | http://landley.net/aboriginal/bin keeps a pool of machine images and |
| 26 | toolchains that can be helpful to start from. |
| 27 | |
| 28 | o Build the kernel with CONFIG_GDB_SCRIPTS enabled, but leave |
| 29 | CONFIG_DEBUG_INFO_REDUCED off. If your architecture supports |
| 30 | CONFIG_FRAME_POINTER, keep it enabled. |
| 31 | |
| 32 | o Install that kernel on the guest. |
| 33 | |
| 34 | Alternatively, QEMU allows to boot the kernel directly using -kernel, |
| 35 | -append, -initrd command line switches. This is generally only useful if |
| 36 | you do not depend on modules. See QEMU documentation for more details on |
| 37 | this mode. |
| 38 | |
| 39 | o Enable the gdb stub of QEMU/KVM, either |
| 40 | - at VM startup time by appending "-s" to the QEMU command line |
| 41 | or |
| 42 | - during runtime by issuing "gdbserver" from the QEMU monitor |
| 43 | console |
| 44 | |
| 45 | o cd /path/to/linux-build |
| 46 | |
| 47 | o Start gdb: gdb vmlinux |
| 48 | |
| 49 | Note: Some distros may restrict auto-loading of gdb scripts to known safe |
| 50 | directories. In case gdb reports to refuse loading vmlinux-gdb.py, add |
| 51 | |
| 52 | add-auto-load-safe-path /path/to/linux-build |
| 53 | |
| 54 | to ~/.gdbinit. See gdb help for more details. |
| 55 | |
| 56 | o Attach to the booted guest: |
| 57 | (gdb) target remote :1234 |
| 58 | |
| 59 | |
| 60 | Examples of using the Linux-provided gdb helpers |
| 61 | ------------------------------------------------ |
| 62 | |
| 63 | o Load module (and main kernel) symbols: |
| 64 | (gdb) lx-symbols |
| 65 | loading vmlinux |
| 66 | scanning for modules in /home/user/linux/build |
| 67 | loading @0xffffffffa0020000: /home/user/linux/build/net/netfilter/xt_tcpudp.ko |
| 68 | loading @0xffffffffa0016000: /home/user/linux/build/net/netfilter/xt_pkttype.ko |
| 69 | loading @0xffffffffa0002000: /home/user/linux/build/net/netfilter/xt_limit.ko |
| 70 | loading @0xffffffffa00ca000: /home/user/linux/build/net/packet/af_packet.ko |
| 71 | loading @0xffffffffa003c000: /home/user/linux/build/fs/fuse/fuse.ko |
| 72 | ... |
| 73 | loading @0xffffffffa0000000: /home/user/linux/build/drivers/ata/ata_generic.ko |
| 74 | |
| 75 | o Set a breakpoint on some not yet loaded module function, e.g.: |
| 76 | (gdb) b btrfs_init_sysfs |
| 77 | Function "btrfs_init_sysfs" not defined. |
| 78 | Make breakpoint pending on future shared library load? (y or [n]) y |
| 79 | Breakpoint 1 (btrfs_init_sysfs) pending. |
| 80 | |
| 81 | o Continue the target |
| 82 | (gdb) c |
| 83 | |
| 84 | o Load the module on the target and watch the symbols being loaded as well as |
| 85 | the breakpoint hit: |
| 86 | loading @0xffffffffa0034000: /home/user/linux/build/lib/libcrc32c.ko |
| 87 | loading @0xffffffffa0050000: /home/user/linux/build/lib/lzo/lzo_compress.ko |
| 88 | loading @0xffffffffa006e000: /home/user/linux/build/lib/zlib_deflate/zlib_deflate.ko |
| 89 | loading @0xffffffffa01b1000: /home/user/linux/build/fs/btrfs/btrfs.ko |
| 90 | |
| 91 | Breakpoint 1, btrfs_init_sysfs () at /home/user/linux/fs/btrfs/sysfs.c:36 |
| 92 | 36 btrfs_kset = kset_create_and_add("btrfs", NULL, fs_kobj); |
| 93 | |
| 94 | o Dump the log buffer of the target kernel: |
| 95 | (gdb) lx-dmesg |
| 96 | [ 0.000000] Initializing cgroup subsys cpuset |
| 97 | [ 0.000000] Initializing cgroup subsys cpu |
| 98 | [ 0.000000] Linux version 3.8.0-rc4-dbg+ (... |
| 99 | [ 0.000000] Command line: root=/dev/sda2 resume=/dev/sda1 vga=0x314 |
| 100 | [ 0.000000] e820: BIOS-provided physical RAM map: |
| 101 | [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable |
| 102 | [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved |
| 103 | .... |
| 104 | |
| 105 | o Examine fields of the current task struct: |
| 106 | (gdb) p $lx_current().pid |
| 107 | $1 = 4998 |
| 108 | (gdb) p $lx_current().comm |
| 109 | $2 = "modprobe\000\000\000\000\000\000\000" |
| 110 | |
| 111 | o Make use of the per-cpu function for the current or a specified CPU: |
| 112 | (gdb) p $lx_per_cpu("runqueues").nr_running |
| 113 | $3 = 1 |
| 114 | (gdb) p $lx_per_cpu("runqueues", 2).nr_running |
| 115 | $4 = 0 |
| 116 | |
| 117 | o Dig into hrtimers using the container_of helper: |
| 118 | (gdb) set $next = $lx_per_cpu("hrtimer_bases").clock_base[0].active.next |
| 119 | (gdb) p *$container_of($next, "struct hrtimer", "node") |
| 120 | $5 = { |
| 121 | node = { |
| 122 | node = { |
| 123 | __rb_parent_color = 18446612133355256072, |
| 124 | rb_right = 0x0 <irq_stack_union>, |
| 125 | rb_left = 0x0 <irq_stack_union> |
| 126 | }, |
| 127 | expires = { |
| 128 | tv64 = 1835268000000 |
| 129 | } |
| 130 | }, |
| 131 | _softexpires = { |
| 132 | tv64 = 1835268000000 |
| 133 | }, |
| 134 | function = 0xffffffff81078232 <tick_sched_timer>, |
| 135 | base = 0xffff88003fd0d6f0, |
| 136 | state = 1, |
| 137 | start_pid = 0, |
| 138 | start_site = 0xffffffff81055c1f <hrtimer_start_range_ns+20>, |
| 139 | start_comm = "swapper/2\000\000\000\000\000\000" |
| 140 | } |
| 141 | |
Kieran Bingham | 9b55803 | 2016-05-23 16:25:10 -0700 | [diff] [blame^] | 142 | o Dig into a radix tree data structure, such as the IRQ descriptors: |
| 143 | (gdb) print (struct irq_desc)$lx_radix_tree_lookup(irq_desc_tree, 18) |
| 144 | $6 = { |
| 145 | irq_common_data = { |
| 146 | state_use_accessors = 67584, |
| 147 | handler_data = 0x0 <__vectors_start>, |
| 148 | msi_desc = 0x0 <__vectors_start>, |
| 149 | affinity = {{ |
| 150 | bits = {65535} |
| 151 | }} |
| 152 | }, |
| 153 | irq_data = { |
| 154 | mask = 0, |
| 155 | irq = 18, |
| 156 | hwirq = 27, |
| 157 | common = 0xee803d80, |
| 158 | chip = 0xc0eb0854 <gic_data>, |
| 159 | domain = 0xee808000, |
| 160 | parent_data = 0x0 <__vectors_start>, |
| 161 | chip_data = 0xc0eb0854 <gic_data> |
| 162 | } <... trimmed ...> |
Jan Kiszka | bda1a92 | 2015-02-17 13:47:32 -0800 | [diff] [blame] | 163 | |
| 164 | List of commands and functions |
| 165 | ------------------------------ |
| 166 | |
| 167 | The number of commands and convenience functions may evolve over the time, |
| 168 | this is just a snapshot of the initial version: |
| 169 | |
| 170 | (gdb) apropos lx |
| 171 | function lx_current -- Return current task |
| 172 | function lx_module -- Find module by name and return the module variable |
| 173 | function lx_per_cpu -- Return per-cpu variable |
| 174 | function lx_task_by_pid -- Find Linux task by PID and return the task_struct variable |
| 175 | function lx_thread_info -- Calculate Linux thread_info from task variable |
| 176 | lx-dmesg -- Print Linux kernel log buffer |
| 177 | lx-lsmod -- List currently loaded modules |
| 178 | lx-symbols -- (Re-)load symbols of Linux kernel and currently loaded modules |
| 179 | |
| 180 | Detailed help can be obtained via "help <command-name>" for commands and "help |
| 181 | function <function-name>" for convenience functions. |