blob: 79775a4130b5e1992d8ed8ca82d3056d0263e0bc [file] [log] [blame]
David Wilderdc851a02006-06-25 05:47:55 -07001================================================================
2Documentation for Kdump - The kexec-based Crash Dumping Solution
Vivek Goyalb089f4a2005-06-25 14:58:15 -07003================================================================
4
David Wilderdc851a02006-06-25 05:47:55 -07005This document includes overview, setup and installation, and analysis
6information.
Vivek Goyalb089f4a2005-06-25 14:58:15 -07007
David Wilderdc851a02006-06-25 05:47:55 -07008Overview
9========
Vivek Goyalb089f4a2005-06-25 14:58:15 -070010
David Wilderdc851a02006-06-25 05:47:55 -070011Kdump uses kexec to quickly boot to a dump-capture kernel whenever a
12dump of the system kernel's memory needs to be taken (for example, when
13the system panics). The system kernel's memory image is preserved across
14the reboot and is accessible to the dump-capture kernel.
Vivek Goyalb089f4a2005-06-25 14:58:15 -070015
David Wilderdc851a02006-06-25 05:47:55 -070016You can use common Linux commands, such as cp and scp, to copy the
17memory image to a dump file on the local disk, or across the network to
18a remote system.
Vivek Goyalb089f4a2005-06-25 14:58:15 -070019
Hormsee8bb9e2007-01-22 20:40:49 -080020Kdump and kexec are currently supported on the x86, x86_64, ppc64 and ia64
David Wilderdc851a02006-06-25 05:47:55 -070021architectures.
Vivek Goyalb089f4a2005-06-25 14:58:15 -070022
David Wilderdc851a02006-06-25 05:47:55 -070023When the system kernel boots, it reserves a small section of memory for
24the dump-capture kernel. This ensures that ongoing Direct Memory Access
25(DMA) from the system kernel does not corrupt the dump-capture kernel.
26The kexec -p command loads the dump-capture kernel into this reserved
27memory.
Vivek Goyalb089f4a2005-06-25 14:58:15 -070028
David Wilderdc851a02006-06-25 05:47:55 -070029On x86 machines, the first 640 KB of physical memory is needed to boot,
30regardless of where the kernel loads. Therefore, kexec backs up this
31region just before rebooting into the dump-capture kernel.
Vivek Goyalb089f4a2005-06-25 14:58:15 -070032
David Wilderdc851a02006-06-25 05:47:55 -070033All of the necessary information about the system kernel's core image is
34encoded in the ELF format, and stored in a reserved area of memory
35before a crash. The physical address of the start of the ELF header is
36passed to the dump-capture kernel through the elfcorehdr= boot
37parameter.
Vivek Goyalb089f4a2005-06-25 14:58:15 -070038
David Wilderdc851a02006-06-25 05:47:55 -070039With the dump-capture kernel, you can access the memory image, or "old
40memory," in two ways:
Vivek Goyalb089f4a2005-06-25 14:58:15 -070041
David Wilderdc851a02006-06-25 05:47:55 -070042- Through a /dev/oldmem device interface. A capture utility can read the
43 device file and write out the memory in raw format. This is a raw dump
44 of memory. Analysis and capture tools must be intelligent enough to
45 determine where to look for the right information.
Vivek Goyalb089f4a2005-06-25 14:58:15 -070046
David Wilderdc851a02006-06-25 05:47:55 -070047- Through /proc/vmcore. This exports the dump as an ELF-format file that
48 you can write out using file copy commands such as cp or scp. Further,
49 you can use analysis tools such as the GNU Debugger (GDB) and the Crash
50 tool to debug the dump file. This method ensures that the dump pages are
51 correctly ordered.
Vivek Goyalb089f4a2005-06-25 14:58:15 -070052
Vivek Goyalb089f4a2005-06-25 14:58:15 -070053
David Wilderdc851a02006-06-25 05:47:55 -070054Setup and Installation
55======================
Vivek Goyalb089f4a2005-06-25 14:58:15 -070056
Vivek Goyal9c61a442007-01-10 23:15:35 -080057Install kexec-tools
58-------------------
Vivek Goyalb089f4a2005-06-25 14:58:15 -070059
David Wilderdc851a02006-06-25 05:47:55 -0700601) Login as the root user.
Vivek Goyalb089f4a2005-06-25 14:58:15 -070061
David Wilderdc851a02006-06-25 05:47:55 -0700622) Download the kexec-tools user-space package from the following URL:
Vivek Goyalb089f4a2005-06-25 14:58:15 -070063
Hormsea112bd2007-01-22 20:40:48 -080064http://www.kernel.org/pub/linux/kernel/people/horms/kexec-tools/kexec-tools-testing.tar.gz
65
66This is a symlink to the latest version, which at the time of writing is
6720061214, the only release of kexec-tools-testing so far. As other versions
68are made released, the older onese will remain available at
69http://www.kernel.org/pub/linux/kernel/people/horms/kexec-tools/
Vivek Goyal9c61a442007-01-10 23:15:35 -080070
71Note: Latest kexec-tools-testing git tree is available at
72
73git://git.kernel.org/pub/scm/linux/kernel/git/horms/kexec-tools-testing.git
74or
75http://www.kernel.org/git/?p=linux/kernel/git/horms/kexec-tools-testing.git;a=summary
Vivek Goyalb089f4a2005-06-25 14:58:15 -070076
David Wilderdc851a02006-06-25 05:47:55 -0700773) Unpack the tarball with the tar command, as follows:
78
Hormsea112bd2007-01-22 20:40:48 -080079 tar xvpzf kexec-tools-testing.tar.gz
David Wilderdc851a02006-06-25 05:47:55 -070080
Hormsea112bd2007-01-22 20:40:48 -0800814) Change to the kexec-tools directory, as follows:
David Wilderdc851a02006-06-25 05:47:55 -070082
Hormsea112bd2007-01-22 20:40:48 -080083 cd kexec-tools-testing-VERSION
David Wilderdc851a02006-06-25 05:47:55 -070084
Vivek Goyal9c61a442007-01-10 23:15:35 -0800855) Configure the package, as follows:
David Wilderdc851a02006-06-25 05:47:55 -070086
87 ./configure
88
Vivek Goyal9c61a442007-01-10 23:15:35 -0800896) Compile the package, as follows:
David Wilderdc851a02006-06-25 05:47:55 -070090
91 make
92
Vivek Goyal9c61a442007-01-10 23:15:35 -0800937) Install the package, as follows:
David Wilderdc851a02006-06-25 05:47:55 -070094
95 make install
96
97
Vivek Goyal9c61a442007-01-10 23:15:35 -080098Build the system and dump-capture kernels
99-----------------------------------------
100There are two possible methods of using Kdump.
David Wilderdc851a02006-06-25 05:47:55 -0700101
Vivek Goyal9c61a442007-01-10 23:15:35 -08001021) Build a separate custom dump-capture kernel for capturing the
103 kernel core dump.
David Wilderdc851a02006-06-25 05:47:55 -0700104
Vivek Goyal9c61a442007-01-10 23:15:35 -08001052) Or use the system kernel binary itself as dump-capture kernel and there is
106 no need to build a separate dump-capture kernel. This is possible
107 only with the architecutres which support a relocatable kernel. As
108 of today i386 and ia64 architectures support relocatable kernel.
109
110Building a relocatable kernel is advantageous from the point of view that
111one does not have to build a second kernel for capturing the dump. But
112at the same time one might want to build a custom dump capture kernel
113suitable to his needs.
114
115Following are the configuration setting required for system and
116dump-capture kernels for enabling kdump support.
117
118System kernel config options
119----------------------------
David Wilderdc851a02006-06-25 05:47:55 -0700120
1211) Enable "kexec system call" in "Processor type and features."
122
123 CONFIG_KEXEC=y
124
1252) Enable "sysfs file system support" in "Filesystem" -> "Pseudo
126 filesystems." This is usually enabled by default.
127
128 CONFIG_SYSFS=y
129
130 Note that "sysfs file system support" might not appear in the "Pseudo
131 filesystems" menu if "Configure standard kernel features (for small
132 systems)" is not enabled in "General Setup." In this case, check the
133 .config file itself to ensure that sysfs is turned on, as follows:
134
135 grep 'CONFIG_SYSFS' .config
136
1373) Enable "Compile the kernel with debug info" in "Kernel hacking."
138
139 CONFIG_DEBUG_INFO=Y
140
141 This causes the kernel to be built with debug symbols. The dump
142 analysis tools require a vmlinux with debug symbols in order to read
143 and analyze a dump file.
144
Vivek Goyal9c61a442007-01-10 23:15:35 -0800145Dump-capture kernel config options (Arch Independent)
146-----------------------------------------------------
David Wilderdc851a02006-06-25 05:47:55 -0700147
Vivek Goyal9c61a442007-01-10 23:15:35 -08001481) Enable "kernel crash dumps" support under "Processor type and
149 features":
David Wilderdc851a02006-06-25 05:47:55 -0700150
Vivek Goyal9c61a442007-01-10 23:15:35 -0800151 CONFIG_CRASH_DUMP=y
David Wilderdc851a02006-06-25 05:47:55 -0700152
Vivek Goyal9c61a442007-01-10 23:15:35 -08001532) Enable "/proc/vmcore support" under "Filesystems" -> "Pseudo filesystems".
David Wilderdc851a02006-06-25 05:47:55 -0700154
Vivek Goyal9c61a442007-01-10 23:15:35 -0800155 CONFIG_PROC_VMCORE=y
156 (CONFIG_PROC_VMCORE is set by default when CONFIG_CRASH_DUMP is selected.)
David Wilderdc851a02006-06-25 05:47:55 -0700157
Vivek Goyal9c61a442007-01-10 23:15:35 -0800158Dump-capture kernel config options (Arch Dependent, i386)
159--------------------------------------------------------
1601) On x86, enable high memory support under "Processor type and
David Wilderdc851a02006-06-25 05:47:55 -0700161 features":
162
163 CONFIG_HIGHMEM64G=y
164 or
165 CONFIG_HIGHMEM4G
166
Vivek Goyal9c61a442007-01-10 23:15:35 -08001672) On x86 and x86_64, disable symmetric multi-processing support
David Wilderdc851a02006-06-25 05:47:55 -0700168 under "Processor type and features":
169
170 CONFIG_SMP=n
Vivek Goyal9c61a442007-01-10 23:15:35 -0800171
David Wilderdc851a02006-06-25 05:47:55 -0700172 (If CONFIG_SMP=y, then specify maxcpus=1 on the kernel command line
173 when loading the dump-capture kernel, see section "Load the Dump-capture
174 Kernel".)
175
Vivek Goyal9c61a442007-01-10 23:15:35 -08001763) If one wants to build and use a relocatable kernel,
177 Enable "Build a relocatable kernel" support under "Processor type and
178 features"
David Wilderdc851a02006-06-25 05:47:55 -0700179
Vivek Goyal9c61a442007-01-10 23:15:35 -0800180 CONFIG_RELOCATABLE=y
David Wilderdc851a02006-06-25 05:47:55 -0700181
Vivek Goyal9c61a442007-01-10 23:15:35 -08001824) Use a suitable value for "Physical address where the kernel is
183 loaded" (under "Processor type and features"). This only appears when
184 "kernel crash dumps" is enabled. A suitable value depends upon
185 whether kernel is relocatable or not.
David Wilderdc851a02006-06-25 05:47:55 -0700186
Vivek Goyal9c61a442007-01-10 23:15:35 -0800187 If you are using a relocatable kernel use CONFIG_PHYSICAL_START=0x100000
188 This will compile the kernel for physical address 1MB, but given the fact
189 kernel is relocatable, it can be run from any physical address hence
190 kexec boot loader will load it in memory region reserved for dump-capture
191 kernel.
David Wilderdc851a02006-06-25 05:47:55 -0700192
Vivek Goyal9c61a442007-01-10 23:15:35 -0800193 Otherwise it should be the start of memory region reserved for
194 second kernel using boot parameter "crashkernel=Y@X". Here X is
195 start of memory region reserved for dump-capture kernel.
196 Generally X is 16MB (0x1000000). So you can set
197 CONFIG_PHYSICAL_START=0x1000000
198
1995) Make and install the kernel and its modules. DO NOT add this kernel
200 to the boot loader configuration files.
201
202Dump-capture kernel config options (Arch Dependent, x86_64)
203----------------------------------------------------------
2041) On x86 and x86_64, disable symmetric multi-processing support
205 under "Processor type and features":
206
207 CONFIG_SMP=n
208
209 (If CONFIG_SMP=y, then specify maxcpus=1 on the kernel command line
210 when loading the dump-capture kernel, see section "Load the Dump-capture
211 Kernel".)
212
2132) Use a suitable value for "Physical address where the kernel is
David Wilderdc851a02006-06-25 05:47:55 -0700214 loaded" (under "Processor type and features"). This only appears when
215 "kernel crash dumps" is enabled. By default this value is 0x1000000
216 (16MB). It should be the same as X in the "crashkernel=Y@X" boot
Vivek Goyal9c61a442007-01-10 23:15:35 -0800217 parameter.
David Wilderdc851a02006-06-25 05:47:55 -0700218
Vivek Goyal9c61a442007-01-10 23:15:35 -0800219 For x86_64, normally "CONFIG_PHYSICAL_START=0x1000000".
David Wilderdc851a02006-06-25 05:47:55 -0700220
Vivek Goyal9c61a442007-01-10 23:15:35 -08002213) Make and install the kernel and its modules. DO NOT add this kernel
David Wilderdc851a02006-06-25 05:47:55 -0700222 to the boot loader configuration files.
223
Vivek Goyal9c61a442007-01-10 23:15:35 -0800224Dump-capture kernel config options (Arch Dependent, ppc64)
225----------------------------------------------------------
226
227- Make and install the kernel and its modules. DO NOT add this kernel
228 to the boot loader configuration files.
229
230Dump-capture kernel config options (Arch Dependent, ia64)
231----------------------------------------------------------
Hormsee8bb9e2007-01-22 20:40:49 -0800232
233- No specific options are required to create a dump-capture kernel
234 for ia64, other than those specified in the arch idependent section
235 above. This means that it is possible to use the system kernel
236 as a dump-capture kernel if desired.
237
238 The crashkernel region can be automatically placed by the system
239 kernel at run time. This is done by specifying the base address as 0,
240 or omitting it all together.
241
242 crashkernel=256M@0
243 or
244 crashkernel=256M
245
246 If the start address is specified, note that the start address of the
247 kernel will be aligned to 64Mb, so if the start address is not then
248 any space below the alignment point will be wasted.
Vivek Goyal9c61a442007-01-10 23:15:35 -0800249
250
251Boot into System Kernel
252=======================
253
2541) Make and install the kernel and its modules. Update the boot loader
255 (such as grub, yaboot, or lilo) configuration files as necessary.
256
2572) Boot the system kernel with the boot parameter "crashkernel=Y@X",
258 where Y specifies how much memory to reserve for the dump-capture kernel
259 and X specifies the beginning of this reserved memory. For example,
260 "crashkernel=64M@16M" tells the system kernel to reserve 64 MB of memory
261 starting at physical address 0x01000000 (16MB) for the dump-capture kernel.
262
263 On x86 and x86_64, use "crashkernel=64M@16M".
264
265 On ppc64, use "crashkernel=128M@32M".
David Wilderdc851a02006-06-25 05:47:55 -0700266
Hormsee8bb9e2007-01-22 20:40:49 -0800267 On ia64, 256M@256M is a generous value that typically works.
268 The region may be automatically placed on ia64, see the
269 dump-capture kernel config option notes above.
270
David Wilderdc851a02006-06-25 05:47:55 -0700271Load the Dump-capture Kernel
272============================
273
Vivek Goyal9c61a442007-01-10 23:15:35 -0800274After booting to the system kernel, dump-capture kernel needs to be
275loaded.
David Wilderdc851a02006-06-25 05:47:55 -0700276
Vivek Goyal9c61a442007-01-10 23:15:35 -0800277Based on the architecture and type of image (relocatable or not), one
278can choose to load the uncompressed vmlinux or compressed bzImage/vmlinuz
279of dump-capture kernel. Following is the summary.
280
281For i386:
282 - Use vmlinux if kernel is not relocatable.
283 - Use bzImage/vmlinuz if kernel is relocatable.
284For x86_64:
285 - Use vmlinux
286For ppc64:
287 - Use vmlinux
288For ia64:
Hormsee8bb9e2007-01-22 20:40:49 -0800289 - Use vmlinux or vmlinuz.gz
290
Vivek Goyal9c61a442007-01-10 23:15:35 -0800291
292If you are using a uncompressed vmlinux image then use following command
293to load dump-capture kernel.
294
295 kexec -p <dump-capture-kernel-vmlinux-image> \
David Wilderdc851a02006-06-25 05:47:55 -0700296 --initrd=<initrd-for-dump-capture-kernel> --args-linux \
Vivek Goyal9c61a442007-01-10 23:15:35 -0800297 --append="root=<root-dev> <arch-specific-options>"
298
299If you are using a compressed bzImage/vmlinuz, then use following command
300to load dump-capture kernel.
301
302 kexec -p <dump-capture-kernel-bzImage> \
303 --initrd=<initrd-for-dump-capture-kernel> \
304 --append="root=<root-dev> <arch-specific-options>"
305
Hormsee8bb9e2007-01-22 20:40:49 -0800306Please note, that --args-linux does not need to be specified for ia64.
307It is planned to make this a no-op on that architecture, but for now
308it should be omitted
309
Vivek Goyal9c61a442007-01-10 23:15:35 -0800310Following are the arch specific command line options to be used while
311loading dump-capture kernel.
312
Hormsee8bb9e2007-01-22 20:40:49 -0800313For i386, x86_64 and ia64:
Horms473e66f2007-02-12 00:52:18 -0800314 "1 irqpoll maxcpus=1"
Vivek Goyal9c61a442007-01-10 23:15:35 -0800315
316For ppc64:
Horms473e66f2007-02-12 00:52:18 -0800317 "1 maxcpus=1 noirqdistrib"
Vivek Goyal9c61a442007-01-10 23:15:35 -0800318
David Wilderdc851a02006-06-25 05:47:55 -0700319
320Notes on loading the dump-capture kernel:
321
David Wilderdc851a02006-06-25 05:47:55 -0700322* By default, the ELF headers are stored in ELF64 format to support
323 systems with more than 4GB memory. The --elf32-core-headers option can
324 be used to force the generation of ELF32 headers. This is necessary
325 because GDB currently cannot open vmcore files with ELF64 headers on
326 32-bit systems. ELF32 headers can be used on non-PAE systems (that is,
327 less than 4GB of memory).
328
329* The "irqpoll" boot parameter reduces driver initialization failures
330 due to shared interrupts in the dump-capture kernel.
331
332* You must specify <root-dev> in the format corresponding to the root
333 device name in the output of mount command.
334
Horms473e66f2007-02-12 00:52:18 -0800335* Boot parameter "1" boots the dump-capture kernel into single-user
336 mode without networking. If you want networking, use "3".
David Wilderdc851a02006-06-25 05:47:55 -0700337
Vivek Goyal9c61a442007-01-10 23:15:35 -0800338* We generally don' have to bring up a SMP kernel just to capture the
339 dump. Hence generally it is useful either to build a UP dump-capture
340 kernel or specify maxcpus=1 option while loading dump-capture kernel.
David Wilderdc851a02006-06-25 05:47:55 -0700341
342Kernel Panic
343============
344
345After successfully loading the dump-capture kernel as previously
346described, the system will reboot into the dump-capture kernel if a
347system crash is triggered. Trigger points are located in panic(),
348die(), die_nmi() and in the sysrq handler (ALT-SysRq-c).
349
350The following conditions will execute a crash trigger point:
351
352If a hard lockup is detected and "NMI watchdog" is configured, the system
353will boot into the dump-capture kernel ( die_nmi() ).
354
355If die() is called, and it happens to be a thread with pid 0 or 1, or die()
356is called inside interrupt context or die() is called and panic_on_oops is set,
357the system will boot into the dump-capture kernel.
358
Paolo Ornati670e9f32006-10-03 22:57:56 +0200359On powererpc systems when a soft-reset is generated, die() is called by all cpus and the system will boot into the dump-capture kernel.
David Wilderdc851a02006-06-25 05:47:55 -0700360
361For testing purposes, you can trigger a crash by using "ALT-SysRq-c",
362"echo c > /proc/sysrq-trigger or write a module to force the panic.
363
364Write Out the Dump File
365=======================
366
367After the dump-capture kernel is booted, write out the dump file with
368the following command:
Vivek Goyalb089f4a2005-06-25 14:58:15 -0700369
370 cp /proc/vmcore <dump-file>
371
David Wilderdc851a02006-06-25 05:47:55 -0700372You can also access dumped memory as a /dev/oldmem device for a linear
373and raw view. To create the device, use the following command:
Vivek Goyalb089f4a2005-06-25 14:58:15 -0700374
David Wilderdc851a02006-06-25 05:47:55 -0700375 mknod /dev/oldmem c 1 12
Vivek Goyalb089f4a2005-06-25 14:58:15 -0700376
David Wilderdc851a02006-06-25 05:47:55 -0700377Use the dd command with suitable options for count, bs, and skip to
378access specific portions of the dump.
Vivek Goyalb089f4a2005-06-25 14:58:15 -0700379
David Wilderdc851a02006-06-25 05:47:55 -0700380To see the entire memory, use the following command:
381
382 dd if=/dev/oldmem of=oldmem.001
Vivek Goyalb089f4a2005-06-25 14:58:15 -0700383
Maneesh Sonia7e670d2006-01-09 20:51:53 -0800384
David Wilderdc851a02006-06-25 05:47:55 -0700385Analysis
Vivek Goyalb089f4a2005-06-25 14:58:15 -0700386========
Vivek Goyalb089f4a2005-06-25 14:58:15 -0700387
David Wilderdc851a02006-06-25 05:47:55 -0700388Before analyzing the dump image, you should reboot into a stable kernel.
Vivek Goyalb089f4a2005-06-25 14:58:15 -0700389
David Wilderdc851a02006-06-25 05:47:55 -0700390You can do limited analysis using GDB on the dump file copied out of
391/proc/vmcore. Use the debug vmlinux built with -g and run the following
392command:
Vivek Goyalb089f4a2005-06-25 14:58:15 -0700393
David Wilderdc851a02006-06-25 05:47:55 -0700394 gdb vmlinux <dump-file>
Vivek Goyalb089f4a2005-06-25 14:58:15 -0700395
David Wilderdc851a02006-06-25 05:47:55 -0700396Stack trace for the task on processor 0, register display, and memory
397display work fine.
398
399Note: GDB cannot analyze core files generated in ELF64 format for x86.
400On systems with a maximum of 4GB of memory, you can generate
401ELF32-format headers using the --elf32-core-headers kernel option on the
402dump kernel.
403
404You can also use the Crash utility to analyze dump files in Kdump
405format. Crash is available on Dave Anderson's site at the following URL:
406
407 http://people.redhat.com/~anderson/
Maneesh Sonia7e670d2006-01-09 20:51:53 -0800408
409
David Wilderdc851a02006-06-25 05:47:55 -0700410To Do
411=====
412
4131) Provide a kernel pages filtering mechanism, so core file size is not
414 extreme on systems with huge memory banks.
415
4162) Relocatable kernel can help in maintaining multiple kernels for
417 crash_dump, and the same kernel as the system kernel can be used to
418 capture the dump.
Maneesh Sonia7e670d2006-01-09 20:51:53 -0800419
Vivek Goyalb089f4a2005-06-25 14:58:15 -0700420
David Wilderdc851a02006-06-25 05:47:55 -0700421Contact
Vivek Goyalb089f4a2005-06-25 14:58:15 -0700422=======
David Wilderdc851a02006-06-25 05:47:55 -0700423
Vivek Goyalb089f4a2005-06-25 14:58:15 -0700424Vivek Goyal (vgoyal@in.ibm.com)
Vivek Goyald58831e2005-06-25 14:58:17 -0700425Maneesh Soni (maneesh@in.ibm.com)
David Wilderdc851a02006-06-25 05:47:55 -0700426
427
428Trademark
429=========
430
431Linux is a trademark of Linus Torvalds in the United States, other
432countries, or both.