blob: 212cf3c21abf8849e39b5964607263e23da8e77f [file] [log] [blame]
Vivek Goyalb089f4a2005-06-25 14:58:15 -07001Documentation for kdump - the kexec-based crash dumping solution
2================================================================
3
4DESIGN
5======
6
Maneesh Sonia7e670d2006-01-09 20:51:53 -08007Kdump uses kexec to reboot to a second kernel whenever a dump needs to be
8taken. This second kernel is booted with very little memory. The first kernel
9reserves the section of memory that the second kernel uses. This ensures that
10on-going DMA from the first kernel does not corrupt the second kernel.
Vivek Goyalb089f4a2005-06-25 14:58:15 -070011
12All the necessary information about Core image is encoded in ELF format and
13stored in reserved area of memory before crash. Physical address of start of
14ELF header is passed to new kernel through command line parameter elfcorehdr=.
15
16On i386, the first 640 KB of physical memory is needed to boot, irrespective
17of where the kernel loads. Hence, this region is backed up by kexec just before
18rebooting into the new kernel.
19
20In the second kernel, "old memory" can be accessed in two ways.
21
22- The first one is through a /dev/oldmem device interface. A capture utility
23 can read the device file and write out the memory in raw format. This is raw
24 dump of memory and analysis/capture tool should be intelligent enough to
25 determine where to look for the right information. ELF headers (elfcorehdr=)
26 can become handy here.
27
28- The second interface is through /proc/vmcore. This exports the dump as an ELF
29 format file which can be written out using any file copy command
30 (cp, scp, etc). Further, gdb can be used to perform limited debugging on
31 the dump file. This method ensures methods ensure that there is correct
32 ordering of the dump pages (corresponding to the first 640 KB that has been
33 relocated).
34
35SETUP
36=====
37
Maneesh Sonia7e670d2006-01-09 20:51:53 -0800381) Download the upstream kexec-tools userspace package from
39 http://www.xmission.com/~ebiederm/files/kexec/kexec-tools-1.101.tar.gz.
Vivek Goyalb089f4a2005-06-25 14:58:15 -070040
Maneesh Sonia7e670d2006-01-09 20:51:53 -080041 Apply the latest consolidated kdump patch on top of kexec-tools-1.101
42 from http://lse.sourceforge.net/kdump/. This arrangment has been made
43 till all the userspace patches supporting kdump are integrated with
44 upstream kexec-tools userspace.
Vivek Goyalb089f4a2005-06-25 14:58:15 -070045
Maneesh Sonia7e670d2006-01-09 20:51:53 -0800462) Download and build the appropriate (2.6.13-rc1 onwards) vanilla kernels.
Vivek Goyalb089f4a2005-06-25 14:58:15 -070047 Two kernels need to be built in order to get this feature working.
Maneesh Sonia7e670d2006-01-09 20:51:53 -080048 Following are the steps to properly configure the two kernels specific
49 to kexec and kdump features:
Vivek Goyalb089f4a2005-06-25 14:58:15 -070050
Maneesh Sonia7e670d2006-01-09 20:51:53 -080051 A) First kernel or regular kernel:
52 ----------------------------------
Vivek Goyalb089f4a2005-06-25 14:58:15 -070053 a) Enable "kexec system call" feature (in Processor type and features).
Maneesh Sonia7e670d2006-01-09 20:51:53 -080054 CONFIG_KEXEC=y
55 b) Enable "sysfs file system support" (in Pseudo filesystems).
56 CONFIG_SYSFS=y
57 c) make
Vivek Goyalb089f4a2005-06-25 14:58:15 -070058 d) Boot into first kernel with the command line parameter "crashkernel=Y@X".
59 Use appropriate values for X and Y. Y denotes how much memory to reserve
Maneesh Sonia7e670d2006-01-09 20:51:53 -080060 for the second kernel, and X denotes at what physical address the
61 reserved memory section starts. For example: "crashkernel=64M@16M".
Vivek Goyalb089f4a2005-06-25 14:58:15 -070062
Vivek Goyalb089f4a2005-06-25 14:58:15 -070063
Maneesh Sonia7e670d2006-01-09 20:51:53 -080064 B) Second kernel or dump capture kernel:
65 ---------------------------------------
66 a) For i386 architecture enable Highmem support
67 CONFIG_HIGHMEM=y
68 b) Enable "kernel crash dumps" feature (under "Processor type and features")
69 CONFIG_CRASH_DUMP=y
70 c) Make sure a suitable value for "Physical address where the kernel is
71 loaded" (under "Processor type and features"). By default this value
72 is 0x1000000 (16MB) and it should be same as X (See option d above),
73 e.g., 16 MB or 0x1000000.
74 CONFIG_PHYSICAL_START=0x1000000
75 d) Enable "/proc/vmcore support" (Optional, under "Pseudo filesystems").
76 CONFIG_PROC_VMCORE=y
Vivek Goyalb089f4a2005-06-25 14:58:15 -070077
Maneesh Sonia7e670d2006-01-09 20:51:53 -0800783) After booting to regular kernel or first kernel, load the second kernel
79 using the following command:
Vivek Goyalb089f4a2005-06-25 14:58:15 -070080
Vivek Goyal952b6492005-09-09 13:10:19 -070081 kexec -p <second-kernel> --args-linux --elf32-core-headers
Maneesh Sonia7e670d2006-01-09 20:51:53 -080082 --append="root=<root-dev> init 1 irqpoll maxcpus=1"
Vivek Goyalb089f4a2005-06-25 14:58:15 -070083
Maneesh Sonia7e670d2006-01-09 20:51:53 -080084 Notes:
85 ======
86 i) <second-kernel> has to be a vmlinux image ie uncompressed elf image.
87 bzImage will not work, as of now.
88 ii) --args-linux has to be speicfied as if kexec it loading an elf image,
89 it needs to know that the arguments supplied are of linux type.
90 iii) By default ELF headers are stored in ELF64 format to support systems
91 with more than 4GB memory. Option --elf32-core-headers forces generation
92 of ELF32 headers. The reason for this option being, as of now gdb can
93 not open vmcore file with ELF64 headers on a 32 bit systems. So ELF32
94 headers can be used if one has non-PAE systems and hence memory less
95 than 4GB.
96 iv) Specify "irqpoll" as command line parameter. This reduces driver
97 initialization failures in second kernel due to shared interrupts.
98 v) <root-dev> needs to be specified in a format corresponding to the root
99 device name in the output of mount command.
100 vi) If you have built the drivers required to mount root file system as
101 modules in <second-kernel>, then, specify
102 --initrd=<initrd-for-second-kernel>.
103 vii) Specify maxcpus=1 as, if during first kernel run, if panic happens on
104 non-boot cpus, second kernel doesn't seem to be boot up all the cpus.
105 The other option is to always built the second kernel without SMP
106 support ie CONFIG_SMP=n
Vivek Goyalb089f4a2005-06-25 14:58:15 -0700107
Maneesh Sonia7e670d2006-01-09 20:51:53 -08001084) After successfully loading the second kernel as above, if a panic occurs
109 system reboots into the second kernel. A module can be written to force
110 the panic or "ALT-SysRq-c" can be used initiate a crash dump for testing
111 purposes.
Vivek Goyalb089f4a2005-06-25 14:58:15 -0700112
Maneesh Sonia7e670d2006-01-09 20:51:53 -08001135) Once the second kernel has booted, write out the dump file using
Vivek Goyalb089f4a2005-06-25 14:58:15 -0700114
115 cp /proc/vmcore <dump-file>
116
117 Dump memory can also be accessed as a /dev/oldmem device for a linear/raw
118 view. To create the device, type:
119
120 mknod /dev/oldmem c 1 12
121
122 Use "dd" with suitable options for count, bs and skip to access specific
123 portions of the dump.
124
125 Entire memory: dd if=/dev/oldmem of=oldmem.001
126
Maneesh Sonia7e670d2006-01-09 20:51:53 -0800127
Vivek Goyalb089f4a2005-06-25 14:58:15 -0700128ANALYSIS
129========
Vivek Goyalb089f4a2005-06-25 14:58:15 -0700130Limited analysis can be done using gdb on the dump file copied out of
131/proc/vmcore. Use vmlinux built with -g and run
132
133 gdb vmlinux <dump-file>
134
135Stack trace for the task on processor 0, register display, memory display
136work fine.
137
138Note: gdb cannot analyse core files generated in ELF64 format for i386.
139
Maneesh Sonia7e670d2006-01-09 20:51:53 -0800140Latest "crash" (crash-4.0-2.18) as available on Dave Anderson's site
141http://people.redhat.com/~anderson/ works well with kdump format.
142
143
Vivek Goyalb089f4a2005-06-25 14:58:15 -0700144TODO
145====
Vivek Goyalb089f4a2005-06-25 14:58:15 -07001461) Provide a kernel pages filtering mechanism so that core file size is not
147 insane on systems having huge memory banks.
Maneesh Sonia7e670d2006-01-09 20:51:53 -08001482) Relocatable kernel can help in maintaining multiple kernels for crashdump
149 and same kernel as the first kernel can be used to capture the dump.
150
Vivek Goyalb089f4a2005-06-25 14:58:15 -0700151
152CONTACT
153=======
Vivek Goyalb089f4a2005-06-25 14:58:15 -0700154Vivek Goyal (vgoyal@in.ibm.com)
Vivek Goyald58831e2005-06-25 14:58:17 -0700155Maneesh Soni (maneesh@in.ibm.com)