blob: 4ab7d43d07544f7a22b35e49f9eb691eb3e9204d [file] [log] [blame]
Jan Kiszkabda1a922015-02-17 13:47:32 -08001Debugging kernel and modules via gdb
2====================================
3
4The kernel debugger kgdb, hypervisors like QEMU or JTAG-based hardware
5interfaces allow to debug the Linux kernel and its modules during runtime
6using gdb. Gdb comes with a powerful scripting interface for python. The
7kernel provides a collection of helper scripts that can simplify typical
8kernel debugging steps. This is a short tutorial about how to enable and use
9them. It focuses on QEMU/KVM virtual machines as target, but the examples can
10be transferred to the other gdb stubs as well.
11
12
13Requirements
14------------
15
16 o gdb 7.2+ (recommended: 7.4+) with python support enabled (typically true
17 for distributions)
18
19
20Setup
21-----
22
23 o Create a virtual Linux machine for QEMU/KVM (see www.linux-kvm.org and
24 www.qemu.org for more details). For cross-development,
25 http://landley.net/aboriginal/bin keeps a pool of machine images and
26 toolchains that can be helpful to start from.
27
28 o Build the kernel with CONFIG_GDB_SCRIPTS enabled, but leave
29 CONFIG_DEBUG_INFO_REDUCED off. If your architecture supports
30 CONFIG_FRAME_POINTER, keep it enabled.
31
32 o Install that kernel on the guest.
33
34 Alternatively, QEMU allows to boot the kernel directly using -kernel,
35 -append, -initrd command line switches. This is generally only useful if
36 you do not depend on modules. See QEMU documentation for more details on
37 this mode.
38
39 o Enable the gdb stub of QEMU/KVM, either
40 - at VM startup time by appending "-s" to the QEMU command line
41 or
42 - during runtime by issuing "gdbserver" from the QEMU monitor
43 console
44
45 o cd /path/to/linux-build
46
47 o Start gdb: gdb vmlinux
48
49 Note: Some distros may restrict auto-loading of gdb scripts to known safe
50 directories. In case gdb reports to refuse loading vmlinux-gdb.py, add
51
52 add-auto-load-safe-path /path/to/linux-build
53
54 to ~/.gdbinit. See gdb help for more details.
55
56 o Attach to the booted guest:
57 (gdb) target remote :1234
58
59
60Examples of using the Linux-provided gdb helpers
61------------------------------------------------
62
63 o Load module (and main kernel) symbols:
64 (gdb) lx-symbols
65 loading vmlinux
66 scanning for modules in /home/user/linux/build
67 loading @0xffffffffa0020000: /home/user/linux/build/net/netfilter/xt_tcpudp.ko
68 loading @0xffffffffa0016000: /home/user/linux/build/net/netfilter/xt_pkttype.ko
69 loading @0xffffffffa0002000: /home/user/linux/build/net/netfilter/xt_limit.ko
70 loading @0xffffffffa00ca000: /home/user/linux/build/net/packet/af_packet.ko
71 loading @0xffffffffa003c000: /home/user/linux/build/fs/fuse/fuse.ko
72 ...
73 loading @0xffffffffa0000000: /home/user/linux/build/drivers/ata/ata_generic.ko
74
75 o Set a breakpoint on some not yet loaded module function, e.g.:
76 (gdb) b btrfs_init_sysfs
77 Function "btrfs_init_sysfs" not defined.
78 Make breakpoint pending on future shared library load? (y or [n]) y
79 Breakpoint 1 (btrfs_init_sysfs) pending.
80
81 o Continue the target
82 (gdb) c
83
84 o Load the module on the target and watch the symbols being loaded as well as
85 the breakpoint hit:
86 loading @0xffffffffa0034000: /home/user/linux/build/lib/libcrc32c.ko
87 loading @0xffffffffa0050000: /home/user/linux/build/lib/lzo/lzo_compress.ko
88 loading @0xffffffffa006e000: /home/user/linux/build/lib/zlib_deflate/zlib_deflate.ko
89 loading @0xffffffffa01b1000: /home/user/linux/build/fs/btrfs/btrfs.ko
90
91 Breakpoint 1, btrfs_init_sysfs () at /home/user/linux/fs/btrfs/sysfs.c:36
92 36 btrfs_kset = kset_create_and_add("btrfs", NULL, fs_kobj);
93
94 o Dump the log buffer of the target kernel:
95 (gdb) lx-dmesg
96 [ 0.000000] Initializing cgroup subsys cpuset
97 [ 0.000000] Initializing cgroup subsys cpu
98 [ 0.000000] Linux version 3.8.0-rc4-dbg+ (...
99 [ 0.000000] Command line: root=/dev/sda2 resume=/dev/sda1 vga=0x314
100 [ 0.000000] e820: BIOS-provided physical RAM map:
101 [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
102 [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
103 ....
104
105 o Examine fields of the current task struct:
106 (gdb) p $lx_current().pid
107 $1 = 4998
108 (gdb) p $lx_current().comm
109 $2 = "modprobe\000\000\000\000\000\000\000"
110
111 o Make use of the per-cpu function for the current or a specified CPU:
112 (gdb) p $lx_per_cpu("runqueues").nr_running
113 $3 = 1
114 (gdb) p $lx_per_cpu("runqueues", 2).nr_running
115 $4 = 0
116
117 o Dig into hrtimers using the container_of helper:
118 (gdb) set $next = $lx_per_cpu("hrtimer_bases").clock_base[0].active.next
119 (gdb) p *$container_of($next, "struct hrtimer", "node")
120 $5 = {
121 node = {
122 node = {
123 __rb_parent_color = 18446612133355256072,
124 rb_right = 0x0 <irq_stack_union>,
125 rb_left = 0x0 <irq_stack_union>
126 },
127 expires = {
128 tv64 = 1835268000000
129 }
130 },
131 _softexpires = {
132 tv64 = 1835268000000
133 },
134 function = 0xffffffff81078232 <tick_sched_timer>,
135 base = 0xffff88003fd0d6f0,
136 state = 1,
137 start_pid = 0,
138 start_site = 0xffffffff81055c1f <hrtimer_start_range_ns+20>,
139 start_comm = "swapper/2\000\000\000\000\000\000"
140 }
141
Kieran Bingham9b558032016-05-23 16:25:10 -0700142 o Dig into a radix tree data structure, such as the IRQ descriptors:
143 (gdb) print (struct irq_desc)$lx_radix_tree_lookup(irq_desc_tree, 18)
144 $6 = {
145 irq_common_data = {
146 state_use_accessors = 67584,
147 handler_data = 0x0 <__vectors_start>,
148 msi_desc = 0x0 <__vectors_start>,
149 affinity = {{
150 bits = {65535}
151 }}
152 },
153 irq_data = {
154 mask = 0,
155 irq = 18,
156 hwirq = 27,
157 common = 0xee803d80,
158 chip = 0xc0eb0854 <gic_data>,
159 domain = 0xee808000,
160 parent_data = 0x0 <__vectors_start>,
161 chip_data = 0xc0eb0854 <gic_data>
162 } <... trimmed ...>
Jan Kiszkabda1a922015-02-17 13:47:32 -0800163
164List of commands and functions
165------------------------------
166
167The number of commands and convenience functions may evolve over the time,
168this is just a snapshot of the initial version:
169
170 (gdb) apropos lx
171 function lx_current -- Return current task
172 function lx_module -- Find module by name and return the module variable
173 function lx_per_cpu -- Return per-cpu variable
174 function lx_task_by_pid -- Find Linux task by PID and return the task_struct variable
175 function lx_thread_info -- Calculate Linux thread_info from task variable
176 lx-dmesg -- Print Linux kernel log buffer
177 lx-lsmod -- List currently loaded modules
178 lx-symbols -- (Re-)load symbols of Linux kernel and currently loaded modules
179
180Detailed help can be obtained via "help <command-name>" for commands and "help
181function <function-name>" for convenience functions.