David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 1 | ================================================================ |
| 2 | Documentation for Kdump - The kexec-based Crash Dumping Solution |
Vivek Goyal | b089f4a | 2005-06-25 14:58:15 -0700 | [diff] [blame] | 3 | ================================================================ |
| 4 | |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 5 | This document includes overview, setup and installation, and analysis |
| 6 | information. |
Vivek Goyal | b089f4a | 2005-06-25 14:58:15 -0700 | [diff] [blame] | 7 | |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 8 | Overview |
| 9 | ======== |
Vivek Goyal | b089f4a | 2005-06-25 14:58:15 -0700 | [diff] [blame] | 10 | |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 11 | Kdump uses kexec to quickly boot to a dump-capture kernel whenever a |
| 12 | dump of the system kernel's memory needs to be taken (for example, when |
| 13 | the system panics). The system kernel's memory image is preserved across |
| 14 | the reboot and is accessible to the dump-capture kernel. |
Vivek Goyal | b089f4a | 2005-06-25 14:58:15 -0700 | [diff] [blame] | 15 | |
Pavel Machek | f4e8757 | 2007-10-16 23:31:28 -0700 | [diff] [blame] | 16 | You can use common commands, such as cp and scp, to copy the |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 17 | memory image to a dump file on the local disk, or across the network to |
| 18 | a remote system. |
Vivek Goyal | b089f4a | 2005-06-25 14:58:15 -0700 | [diff] [blame] | 19 | |
Michael Holzheu | 9130214 | 2011-12-27 11:27:25 +0100 | [diff] [blame] | 20 | Kdump and kexec are currently supported on the x86, x86_64, ppc64, ia64, |
| 21 | and s390x architectures. |
Vivek Goyal | b089f4a | 2005-06-25 14:58:15 -0700 | [diff] [blame] | 22 | |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 23 | When the system kernel boots, it reserves a small section of memory for |
| 24 | the dump-capture kernel. This ensures that ongoing Direct Memory Access |
| 25 | (DMA) from the system kernel does not corrupt the dump-capture kernel. |
| 26 | The kexec -p command loads the dump-capture kernel into this reserved |
| 27 | memory. |
Vivek Goyal | b089f4a | 2005-06-25 14:58:15 -0700 | [diff] [blame] | 28 | |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 29 | On x86 machines, the first 640 KB of physical memory is needed to boot, |
| 30 | regardless of where the kernel loads. Therefore, kexec backs up this |
| 31 | region just before rebooting into the dump-capture kernel. |
Vivek Goyal | b089f4a | 2005-06-25 14:58:15 -0700 | [diff] [blame] | 32 | |
Simon Horman | 3043013 | 2007-02-20 13:58:07 -0800 | [diff] [blame] | 33 | Similarly on PPC64 machines first 32KB of physical memory is needed for |
| 34 | booting regardless of where the kernel is loaded and to support 64K page |
| 35 | size kexec backs up the first 64KB memory. |
| 36 | |
Michael Holzheu | 9130214 | 2011-12-27 11:27:25 +0100 | [diff] [blame] | 37 | For s390x, when kdump is triggered, the crashkernel region is exchanged |
| 38 | with the region [0, crashkernel region size] and then the kdump kernel |
| 39 | runs in [0, crashkernel region size]. Therefore no relocatable kernel is |
| 40 | needed for s390x. |
| 41 | |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 42 | All of the necessary information about the system kernel's core image is |
| 43 | encoded in the ELF format, and stored in a reserved area of memory |
| 44 | before a crash. The physical address of the start of the ELF header is |
| 45 | passed to the dump-capture kernel through the elfcorehdr= boot |
Michael Holzheu | 9130214 | 2011-12-27 11:27:25 +0100 | [diff] [blame] | 46 | parameter. Optionally the size of the ELF header can also be passed |
| 47 | when using the elfcorehdr=[size[KMG]@]offset[KMG] syntax. |
| 48 | |
Vivek Goyal | b089f4a | 2005-06-25 14:58:15 -0700 | [diff] [blame] | 49 | |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 50 | With the dump-capture kernel, you can access the memory image, or "old |
| 51 | memory," in two ways: |
Vivek Goyal | b089f4a | 2005-06-25 14:58:15 -0700 | [diff] [blame] | 52 | |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 53 | - Through a /dev/oldmem device interface. A capture utility can read the |
| 54 | device file and write out the memory in raw format. This is a raw dump |
| 55 | of memory. Analysis and capture tools must be intelligent enough to |
| 56 | determine where to look for the right information. |
Vivek Goyal | b089f4a | 2005-06-25 14:58:15 -0700 | [diff] [blame] | 57 | |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 58 | - Through /proc/vmcore. This exports the dump as an ELF-format file that |
| 59 | you can write out using file copy commands such as cp or scp. Further, |
| 60 | you can use analysis tools such as the GNU Debugger (GDB) and the Crash |
| 61 | tool to debug the dump file. This method ensures that the dump pages are |
| 62 | correctly ordered. |
Vivek Goyal | b089f4a | 2005-06-25 14:58:15 -0700 | [diff] [blame] | 63 | |
Vivek Goyal | b089f4a | 2005-06-25 14:58:15 -0700 | [diff] [blame] | 64 | |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 65 | Setup and Installation |
| 66 | ====================== |
Vivek Goyal | b089f4a | 2005-06-25 14:58:15 -0700 | [diff] [blame] | 67 | |
Vivek Goyal | 9c61a44 | 2007-01-10 23:15:35 -0800 | [diff] [blame] | 68 | Install kexec-tools |
| 69 | ------------------- |
Vivek Goyal | b089f4a | 2005-06-25 14:58:15 -0700 | [diff] [blame] | 70 | |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 71 | 1) Login as the root user. |
Vivek Goyal | b089f4a | 2005-06-25 14:58:15 -0700 | [diff] [blame] | 72 | |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 73 | 2) Download the kexec-tools user-space package from the following URL: |
Vivek Goyal | b089f4a | 2005-06-25 14:58:15 -0700 | [diff] [blame] | 74 | |
Simon Horman | db6857c | 2010-11-25 10:30:12 +0900 | [diff] [blame] | 75 | http://kernel.org/pub/linux/utils/kernel/kexec/kexec-tools.tar.gz |
Horms | ea112bd | 2007-01-22 20:40:48 -0800 | [diff] [blame] | 76 | |
Simon Horman | d84a52f | 2008-07-28 15:46:34 -0700 | [diff] [blame] | 77 | This is a symlink to the latest version. |
Vivek Goyal | 9c61a44 | 2007-01-10 23:15:35 -0800 | [diff] [blame] | 78 | |
Simon Horman | d84a52f | 2008-07-28 15:46:34 -0700 | [diff] [blame] | 79 | The latest kexec-tools git tree is available at: |
Vivek Goyal | 9c61a44 | 2007-01-10 23:15:35 -0800 | [diff] [blame] | 80 | |
Simon Horman | db6857c | 2010-11-25 10:30:12 +0900 | [diff] [blame] | 81 | git://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git |
| 82 | and |
| 83 | http://www.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git |
| 84 | |
| 85 | There is also a gitweb interface available at |
| 86 | http://www.kernel.org/git/?p=utils/kernel/kexec/kexec-tools.git |
Simon Horman | d84a52f | 2008-07-28 15:46:34 -0700 | [diff] [blame] | 87 | |
| 88 | More information about kexec-tools can be found at |
Olaf Hering | b1bdd2e | 2012-07-18 14:12:04 -0700 | [diff] [blame] | 89 | http://horms.net/projects/kexec/ |
Vivek Goyal | b089f4a | 2005-06-25 14:58:15 -0700 | [diff] [blame] | 90 | |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 91 | 3) Unpack the tarball with the tar command, as follows: |
| 92 | |
Simon Horman | d84a52f | 2008-07-28 15:46:34 -0700 | [diff] [blame] | 93 | tar xvpzf kexec-tools.tar.gz |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 94 | |
Horms | ea112bd | 2007-01-22 20:40:48 -0800 | [diff] [blame] | 95 | 4) Change to the kexec-tools directory, as follows: |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 96 | |
Simon Horman | d84a52f | 2008-07-28 15:46:34 -0700 | [diff] [blame] | 97 | cd kexec-tools-VERSION |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 98 | |
Vivek Goyal | 9c61a44 | 2007-01-10 23:15:35 -0800 | [diff] [blame] | 99 | 5) Configure the package, as follows: |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 100 | |
| 101 | ./configure |
| 102 | |
Vivek Goyal | 9c61a44 | 2007-01-10 23:15:35 -0800 | [diff] [blame] | 103 | 6) Compile the package, as follows: |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 104 | |
| 105 | make |
| 106 | |
Vivek Goyal | 9c61a44 | 2007-01-10 23:15:35 -0800 | [diff] [blame] | 107 | 7) Install the package, as follows: |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 108 | |
| 109 | make install |
| 110 | |
| 111 | |
Vivek Goyal | 9c61a44 | 2007-01-10 23:15:35 -0800 | [diff] [blame] | 112 | Build the system and dump-capture kernels |
| 113 | ----------------------------------------- |
| 114 | There are two possible methods of using Kdump. |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 115 | |
Vivek Goyal | 9c61a44 | 2007-01-10 23:15:35 -0800 | [diff] [blame] | 116 | 1) Build a separate custom dump-capture kernel for capturing the |
| 117 | kernel core dump. |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 118 | |
Vivek Goyal | 9c61a44 | 2007-01-10 23:15:35 -0800 | [diff] [blame] | 119 | 2) Or use the system kernel binary itself as dump-capture kernel and there is |
| 120 | no need to build a separate dump-capture kernel. This is possible |
Matt LaPlante | 19f5946 | 2009-04-27 15:06:31 +0200 | [diff] [blame] | 121 | only with the architectures which support a relocatable kernel. As |
Mohan Kumar M | 54622f1 | 2008-10-21 17:38:10 +0000 | [diff] [blame] | 122 | of today, i386, x86_64, ppc64 and ia64 architectures support relocatable |
| 123 | kernel. |
Vivek Goyal | 9c61a44 | 2007-01-10 23:15:35 -0800 | [diff] [blame] | 124 | |
| 125 | Building a relocatable kernel is advantageous from the point of view that |
| 126 | one does not have to build a second kernel for capturing the dump. But |
| 127 | at the same time one might want to build a custom dump capture kernel |
| 128 | suitable to his needs. |
| 129 | |
| 130 | Following are the configuration setting required for system and |
| 131 | dump-capture kernels for enabling kdump support. |
| 132 | |
| 133 | System kernel config options |
| 134 | ---------------------------- |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 135 | |
| 136 | 1) Enable "kexec system call" in "Processor type and features." |
| 137 | |
| 138 | CONFIG_KEXEC=y |
| 139 | |
| 140 | 2) Enable "sysfs file system support" in "Filesystem" -> "Pseudo |
| 141 | filesystems." This is usually enabled by default. |
| 142 | |
| 143 | CONFIG_SYSFS=y |
| 144 | |
| 145 | Note that "sysfs file system support" might not appear in the "Pseudo |
| 146 | filesystems" menu if "Configure standard kernel features (for small |
| 147 | systems)" is not enabled in "General Setup." In this case, check the |
| 148 | .config file itself to ensure that sysfs is turned on, as follows: |
| 149 | |
| 150 | grep 'CONFIG_SYSFS' .config |
| 151 | |
| 152 | 3) Enable "Compile the kernel with debug info" in "Kernel hacking." |
| 153 | |
| 154 | CONFIG_DEBUG_INFO=Y |
| 155 | |
| 156 | This causes the kernel to be built with debug symbols. The dump |
| 157 | analysis tools require a vmlinux with debug symbols in order to read |
| 158 | and analyze a dump file. |
| 159 | |
Vivek Goyal | 9c61a44 | 2007-01-10 23:15:35 -0800 | [diff] [blame] | 160 | Dump-capture kernel config options (Arch Independent) |
| 161 | ----------------------------------------------------- |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 162 | |
Vivek Goyal | 9c61a44 | 2007-01-10 23:15:35 -0800 | [diff] [blame] | 163 | 1) Enable "kernel crash dumps" support under "Processor type and |
| 164 | features": |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 165 | |
Vivek Goyal | 9c61a44 | 2007-01-10 23:15:35 -0800 | [diff] [blame] | 166 | CONFIG_CRASH_DUMP=y |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 167 | |
Vivek Goyal | 9c61a44 | 2007-01-10 23:15:35 -0800 | [diff] [blame] | 168 | 2) Enable "/proc/vmcore support" under "Filesystems" -> "Pseudo filesystems". |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 169 | |
Vivek Goyal | 9c61a44 | 2007-01-10 23:15:35 -0800 | [diff] [blame] | 170 | CONFIG_PROC_VMCORE=y |
| 171 | (CONFIG_PROC_VMCORE is set by default when CONFIG_CRASH_DUMP is selected.) |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 172 | |
Bernhard Walle | 8bc9d42 | 2007-10-16 23:31:21 -0700 | [diff] [blame] | 173 | Dump-capture kernel config options (Arch Dependent, i386 and x86_64) |
| 174 | -------------------------------------------------------------------- |
| 175 | |
| 176 | 1) On i386, enable high memory support under "Processor type and |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 177 | features": |
| 178 | |
| 179 | CONFIG_HIGHMEM64G=y |
| 180 | or |
| 181 | CONFIG_HIGHMEM4G |
| 182 | |
Bernhard Walle | 8bc9d42 | 2007-10-16 23:31:21 -0700 | [diff] [blame] | 183 | 2) On i386 and x86_64, disable symmetric multi-processing support |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 184 | under "Processor type and features": |
| 185 | |
| 186 | CONFIG_SMP=n |
Vivek Goyal | 9c61a44 | 2007-01-10 23:15:35 -0800 | [diff] [blame] | 187 | |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 188 | (If CONFIG_SMP=y, then specify maxcpus=1 on the kernel command line |
| 189 | when loading the dump-capture kernel, see section "Load the Dump-capture |
| 190 | Kernel".) |
| 191 | |
Vivek Goyal | 9c61a44 | 2007-01-10 23:15:35 -0800 | [diff] [blame] | 192 | 3) If one wants to build and use a relocatable kernel, |
| 193 | Enable "Build a relocatable kernel" support under "Processor type and |
| 194 | features" |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 195 | |
Vivek Goyal | 9c61a44 | 2007-01-10 23:15:35 -0800 | [diff] [blame] | 196 | CONFIG_RELOCATABLE=y |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 197 | |
Vivek Goyal | 9c61a44 | 2007-01-10 23:15:35 -0800 | [diff] [blame] | 198 | 4) Use a suitable value for "Physical address where the kernel is |
| 199 | loaded" (under "Processor type and features"). This only appears when |
| 200 | "kernel crash dumps" is enabled. A suitable value depends upon |
| 201 | whether kernel is relocatable or not. |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 202 | |
Vivek Goyal | 9c61a44 | 2007-01-10 23:15:35 -0800 | [diff] [blame] | 203 | If you are using a relocatable kernel use CONFIG_PHYSICAL_START=0x100000 |
| 204 | This will compile the kernel for physical address 1MB, but given the fact |
| 205 | kernel is relocatable, it can be run from any physical address hence |
| 206 | kexec boot loader will load it in memory region reserved for dump-capture |
| 207 | kernel. |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 208 | |
Vivek Goyal | 9c61a44 | 2007-01-10 23:15:35 -0800 | [diff] [blame] | 209 | Otherwise it should be the start of memory region reserved for |
| 210 | second kernel using boot parameter "crashkernel=Y@X". Here X is |
| 211 | start of memory region reserved for dump-capture kernel. |
| 212 | Generally X is 16MB (0x1000000). So you can set |
| 213 | CONFIG_PHYSICAL_START=0x1000000 |
| 214 | |
| 215 | 5) Make and install the kernel and its modules. DO NOT add this kernel |
| 216 | to the boot loader configuration files. |
| 217 | |
Vivek Goyal | 9c61a44 | 2007-01-10 23:15:35 -0800 | [diff] [blame] | 218 | Dump-capture kernel config options (Arch Dependent, ppc64) |
| 219 | ---------------------------------------------------------- |
| 220 | |
Mohan Kumar M | 54622f1 | 2008-10-21 17:38:10 +0000 | [diff] [blame] | 221 | 1) Enable "Build a kdump crash kernel" support under "Kernel" options: |
| 222 | |
| 223 | CONFIG_CRASH_DUMP=y |
| 224 | |
| 225 | 2) Enable "Build a relocatable kernel" support |
| 226 | |
| 227 | CONFIG_RELOCATABLE=y |
| 228 | |
| 229 | Make and install the kernel and its modules. |
Vivek Goyal | 9c61a44 | 2007-01-10 23:15:35 -0800 | [diff] [blame] | 230 | |
| 231 | Dump-capture kernel config options (Arch Dependent, ia64) |
| 232 | ---------------------------------------------------------- |
Horms | ee8bb9e | 2007-01-22 20:40:49 -0800 | [diff] [blame] | 233 | |
| 234 | - No specific options are required to create a dump-capture kernel |
Matt LaPlante | 19f5946 | 2009-04-27 15:06:31 +0200 | [diff] [blame] | 235 | for ia64, other than those specified in the arch independent section |
Horms | ee8bb9e | 2007-01-22 20:40:49 -0800 | [diff] [blame] | 236 | above. This means that it is possible to use the system kernel |
| 237 | as a dump-capture kernel if desired. |
| 238 | |
| 239 | The crashkernel region can be automatically placed by the system |
| 240 | kernel at run time. This is done by specifying the base address as 0, |
| 241 | or omitting it all together. |
| 242 | |
| 243 | crashkernel=256M@0 |
| 244 | or |
| 245 | crashkernel=256M |
| 246 | |
| 247 | If the start address is specified, note that the start address of the |
| 248 | kernel will be aligned to 64Mb, so if the start address is not then |
| 249 | any space below the alignment point will be wasted. |
Vivek Goyal | 9c61a44 | 2007-01-10 23:15:35 -0800 | [diff] [blame] | 250 | |
| 251 | |
Bernhard Walle | fb39159 | 2007-10-18 23:41:02 -0700 | [diff] [blame] | 252 | Extended crashkernel syntax |
| 253 | =========================== |
| 254 | |
| 255 | While the "crashkernel=size[@offset]" syntax is sufficient for most |
| 256 | configurations, sometimes it's handy to have the reserved memory dependent |
| 257 | on the value of System RAM -- that's mostly for distributors that pre-setup |
| 258 | the kernel command line to avoid a unbootable system after some memory has |
| 259 | been removed from the machine. |
| 260 | |
| 261 | The syntax is: |
| 262 | |
| 263 | crashkernel=<range1>:<size1>[,<range2>:<size2>,...][@offset] |
| 264 | range=start-[end] |
| 265 | |
Michael Ellerman | be089d79 | 2008-05-01 04:34:49 -0700 | [diff] [blame] | 266 | 'start' is inclusive and 'end' is exclusive. |
| 267 | |
Bernhard Walle | fb39159 | 2007-10-18 23:41:02 -0700 | [diff] [blame] | 268 | For example: |
| 269 | |
| 270 | crashkernel=512M-2G:64M,2G-:128M |
| 271 | |
| 272 | This would mean: |
| 273 | |
| 274 | 1) if the RAM is smaller than 512M, then don't reserve anything |
| 275 | (this is the "rescue" case) |
Michael Ellerman | be089d79 | 2008-05-01 04:34:49 -0700 | [diff] [blame] | 276 | 2) if the RAM size is between 512M and 2G (exclusive), then reserve 64M |
Bernhard Walle | fb39159 | 2007-10-18 23:41:02 -0700 | [diff] [blame] | 277 | 3) if the RAM size is larger than 2G, then reserve 128M |
| 278 | |
| 279 | |
Michael Ellerman | be089d79 | 2008-05-01 04:34:49 -0700 | [diff] [blame] | 280 | |
Vivek Goyal | 9c61a44 | 2007-01-10 23:15:35 -0800 | [diff] [blame] | 281 | Boot into System Kernel |
| 282 | ======================= |
| 283 | |
Simon Horman | 3043013 | 2007-02-20 13:58:07 -0800 | [diff] [blame] | 284 | 1) Update the boot loader (such as grub, yaboot, or lilo) configuration |
| 285 | files as necessary. |
Vivek Goyal | 9c61a44 | 2007-01-10 23:15:35 -0800 | [diff] [blame] | 286 | |
| 287 | 2) Boot the system kernel with the boot parameter "crashkernel=Y@X", |
| 288 | where Y specifies how much memory to reserve for the dump-capture kernel |
| 289 | and X specifies the beginning of this reserved memory. For example, |
| 290 | "crashkernel=64M@16M" tells the system kernel to reserve 64 MB of memory |
| 291 | starting at physical address 0x01000000 (16MB) for the dump-capture kernel. |
| 292 | |
| 293 | On x86 and x86_64, use "crashkernel=64M@16M". |
| 294 | |
| 295 | On ppc64, use "crashkernel=128M@32M". |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 296 | |
Horms | ee8bb9e | 2007-01-22 20:40:49 -0800 | [diff] [blame] | 297 | On ia64, 256M@256M is a generous value that typically works. |
| 298 | The region may be automatically placed on ia64, see the |
| 299 | dump-capture kernel config option notes above. |
| 300 | |
Michael Holzheu | 9130214 | 2011-12-27 11:27:25 +0100 | [diff] [blame] | 301 | On s390x, typically use "crashkernel=xxM". The value of xx is dependent |
| 302 | on the memory consumption of the kdump system. In general this is not |
| 303 | dependent on the memory size of the production system. |
| 304 | |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 305 | Load the Dump-capture Kernel |
| 306 | ============================ |
| 307 | |
Vivek Goyal | 9c61a44 | 2007-01-10 23:15:35 -0800 | [diff] [blame] | 308 | After booting to the system kernel, dump-capture kernel needs to be |
| 309 | loaded. |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 310 | |
Vivek Goyal | 9c61a44 | 2007-01-10 23:15:35 -0800 | [diff] [blame] | 311 | Based on the architecture and type of image (relocatable or not), one |
| 312 | can choose to load the uncompressed vmlinux or compressed bzImage/vmlinuz |
| 313 | of dump-capture kernel. Following is the summary. |
| 314 | |
Bernhard Walle | 8bc9d42 | 2007-10-16 23:31:21 -0700 | [diff] [blame] | 315 | For i386 and x86_64: |
Vivek Goyal | 9c61a44 | 2007-01-10 23:15:35 -0800 | [diff] [blame] | 316 | - Use vmlinux if kernel is not relocatable. |
| 317 | - Use bzImage/vmlinuz if kernel is relocatable. |
Vivek Goyal | 9c61a44 | 2007-01-10 23:15:35 -0800 | [diff] [blame] | 318 | For ppc64: |
| 319 | - Use vmlinux |
| 320 | For ia64: |
Horms | ee8bb9e | 2007-01-22 20:40:49 -0800 | [diff] [blame] | 321 | - Use vmlinux or vmlinuz.gz |
Michael Holzheu | 9130214 | 2011-12-27 11:27:25 +0100 | [diff] [blame] | 322 | For s390x: |
| 323 | - Use image or bzImage |
Horms | ee8bb9e | 2007-01-22 20:40:49 -0800 | [diff] [blame] | 324 | |
Vivek Goyal | 9c61a44 | 2007-01-10 23:15:35 -0800 | [diff] [blame] | 325 | |
| 326 | If you are using a uncompressed vmlinux image then use following command |
| 327 | to load dump-capture kernel. |
| 328 | |
| 329 | kexec -p <dump-capture-kernel-vmlinux-image> \ |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 330 | --initrd=<initrd-for-dump-capture-kernel> --args-linux \ |
Vivek Goyal | 9c61a44 | 2007-01-10 23:15:35 -0800 | [diff] [blame] | 331 | --append="root=<root-dev> <arch-specific-options>" |
| 332 | |
| 333 | If you are using a compressed bzImage/vmlinuz, then use following command |
| 334 | to load dump-capture kernel. |
| 335 | |
| 336 | kexec -p <dump-capture-kernel-bzImage> \ |
| 337 | --initrd=<initrd-for-dump-capture-kernel> \ |
| 338 | --append="root=<root-dev> <arch-specific-options>" |
| 339 | |
Horms | ee8bb9e | 2007-01-22 20:40:49 -0800 | [diff] [blame] | 340 | Please note, that --args-linux does not need to be specified for ia64. |
| 341 | It is planned to make this a no-op on that architecture, but for now |
| 342 | it should be omitted |
| 343 | |
Vivek Goyal | 9c61a44 | 2007-01-10 23:15:35 -0800 | [diff] [blame] | 344 | Following are the arch specific command line options to be used while |
| 345 | loading dump-capture kernel. |
| 346 | |
Horms | ee8bb9e | 2007-01-22 20:40:49 -0800 | [diff] [blame] | 347 | For i386, x86_64 and ia64: |
Bernhard Walle | ac984ab | 2007-10-16 23:31:23 -0700 | [diff] [blame] | 348 | "1 irqpoll maxcpus=1 reset_devices" |
Vivek Goyal | 9c61a44 | 2007-01-10 23:15:35 -0800 | [diff] [blame] | 349 | |
| 350 | For ppc64: |
Bernhard Walle | ac984ab | 2007-10-16 23:31:23 -0700 | [diff] [blame] | 351 | "1 maxcpus=1 noirqdistrib reset_devices" |
Vivek Goyal | 9c61a44 | 2007-01-10 23:15:35 -0800 | [diff] [blame] | 352 | |
Michael Holzheu | 9130214 | 2011-12-27 11:27:25 +0100 | [diff] [blame] | 353 | For s390x: |
| 354 | "1 maxcpus=1 cgroup_disable=memory" |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 355 | |
| 356 | Notes on loading the dump-capture kernel: |
| 357 | |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 358 | * By default, the ELF headers are stored in ELF64 format to support |
Bernhard Walle | 4fd4509 | 2007-10-16 23:31:22 -0700 | [diff] [blame] | 359 | systems with more than 4GB memory. On i386, kexec automatically checks if |
| 360 | the physical RAM size exceeds the 4 GB limit and if not, uses ELF32. |
| 361 | So, on non-PAE systems, ELF32 is always used. |
| 362 | |
| 363 | The --elf32-core-headers option can be used to force the generation of ELF32 |
| 364 | headers. This is necessary because GDB currently cannot open vmcore files |
| 365 | with ELF64 headers on 32-bit systems. |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 366 | |
| 367 | * The "irqpoll" boot parameter reduces driver initialization failures |
| 368 | due to shared interrupts in the dump-capture kernel. |
| 369 | |
| 370 | * You must specify <root-dev> in the format corresponding to the root |
| 371 | device name in the output of mount command. |
| 372 | |
Horms | 473e66f | 2007-02-12 00:52:18 -0800 | [diff] [blame] | 373 | * Boot parameter "1" boots the dump-capture kernel into single-user |
| 374 | mode without networking. If you want networking, use "3". |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 375 | |
Vivek Goyal | 9c61a44 | 2007-01-10 23:15:35 -0800 | [diff] [blame] | 376 | * We generally don' have to bring up a SMP kernel just to capture the |
| 377 | dump. Hence generally it is useful either to build a UP dump-capture |
| 378 | kernel or specify maxcpus=1 option while loading dump-capture kernel. |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 379 | |
Michael Holzheu | 9130214 | 2011-12-27 11:27:25 +0100 | [diff] [blame] | 380 | * For s390x there are two kdump modes: If a ELF header is specified with |
| 381 | the elfcorehdr= kernel parameter, it is used by the kdump kernel as it |
| 382 | is done on all other architectures. If no elfcorehdr= kernel parameter is |
| 383 | specified, the s390x kdump kernel dynamically creates the header. The |
| 384 | second mode has the advantage that for CPU and memory hotplug, kdump has |
| 385 | not to be reloaded with kexec_load(). |
| 386 | |
| 387 | * For s390x systems with many attached devices the "cio_ignore" kernel |
| 388 | parameter should be used for the kdump kernel in order to prevent allocation |
| 389 | of kernel memory for devices that are not relevant for kdump. The same |
| 390 | applies to systems that use SCSI/FCP devices. In that case the |
| 391 | "allow_lun_scan" zfcp module parameter should be set to zero before |
| 392 | setting FCP devices online. |
| 393 | |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 394 | Kernel Panic |
| 395 | ============ |
| 396 | |
| 397 | After successfully loading the dump-capture kernel as previously |
| 398 | described, the system will reboot into the dump-capture kernel if a |
| 399 | system crash is triggered. Trigger points are located in panic(), |
| 400 | die(), die_nmi() and in the sysrq handler (ALT-SysRq-c). |
| 401 | |
| 402 | The following conditions will execute a crash trigger point: |
| 403 | |
| 404 | If a hard lockup is detected and "NMI watchdog" is configured, the system |
| 405 | will boot into the dump-capture kernel ( die_nmi() ). |
| 406 | |
| 407 | If die() is called, and it happens to be a thread with pid 0 or 1, or die() |
| 408 | is called inside interrupt context or die() is called and panic_on_oops is set, |
| 409 | the system will boot into the dump-capture kernel. |
| 410 | |
Pavel Machek | f4e8757 | 2007-10-16 23:31:28 -0700 | [diff] [blame] | 411 | On powerpc systems when a soft-reset is generated, die() is called by all cpus |
Simon Horman | 3043013 | 2007-02-20 13:58:07 -0800 | [diff] [blame] | 412 | and the system will boot into the dump-capture kernel. |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 413 | |
| 414 | For testing purposes, you can trigger a crash by using "ALT-SysRq-c", |
Simon Horman | 3043013 | 2007-02-20 13:58:07 -0800 | [diff] [blame] | 415 | "echo c > /proc/sysrq-trigger" or write a module to force the panic. |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 416 | |
| 417 | Write Out the Dump File |
| 418 | ======================= |
| 419 | |
| 420 | After the dump-capture kernel is booted, write out the dump file with |
| 421 | the following command: |
Vivek Goyal | b089f4a | 2005-06-25 14:58:15 -0700 | [diff] [blame] | 422 | |
| 423 | cp /proc/vmcore <dump-file> |
| 424 | |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 425 | You can also access dumped memory as a /dev/oldmem device for a linear |
| 426 | and raw view. To create the device, use the following command: |
Vivek Goyal | b089f4a | 2005-06-25 14:58:15 -0700 | [diff] [blame] | 427 | |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 428 | mknod /dev/oldmem c 1 12 |
Vivek Goyal | b089f4a | 2005-06-25 14:58:15 -0700 | [diff] [blame] | 429 | |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 430 | Use the dd command with suitable options for count, bs, and skip to |
| 431 | access specific portions of the dump. |
Vivek Goyal | b089f4a | 2005-06-25 14:58:15 -0700 | [diff] [blame] | 432 | |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 433 | To see the entire memory, use the following command: |
| 434 | |
| 435 | dd if=/dev/oldmem of=oldmem.001 |
Vivek Goyal | b089f4a | 2005-06-25 14:58:15 -0700 | [diff] [blame] | 436 | |
Maneesh Soni | a7e670d | 2006-01-09 20:51:53 -0800 | [diff] [blame] | 437 | |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 438 | Analysis |
Vivek Goyal | b089f4a | 2005-06-25 14:58:15 -0700 | [diff] [blame] | 439 | ======== |
Vivek Goyal | b089f4a | 2005-06-25 14:58:15 -0700 | [diff] [blame] | 440 | |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 441 | Before analyzing the dump image, you should reboot into a stable kernel. |
Vivek Goyal | b089f4a | 2005-06-25 14:58:15 -0700 | [diff] [blame] | 442 | |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 443 | You can do limited analysis using GDB on the dump file copied out of |
| 444 | /proc/vmcore. Use the debug vmlinux built with -g and run the following |
| 445 | command: |
Vivek Goyal | b089f4a | 2005-06-25 14:58:15 -0700 | [diff] [blame] | 446 | |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 447 | gdb vmlinux <dump-file> |
Vivek Goyal | b089f4a | 2005-06-25 14:58:15 -0700 | [diff] [blame] | 448 | |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 449 | Stack trace for the task on processor 0, register display, and memory |
| 450 | display work fine. |
| 451 | |
| 452 | Note: GDB cannot analyze core files generated in ELF64 format for x86. |
| 453 | On systems with a maximum of 4GB of memory, you can generate |
| 454 | ELF32-format headers using the --elf32-core-headers kernel option on the |
| 455 | dump kernel. |
| 456 | |
| 457 | You can also use the Crash utility to analyze dump files in Kdump |
| 458 | format. Crash is available on Dave Anderson's site at the following URL: |
| 459 | |
| 460 | http://people.redhat.com/~anderson/ |
Maneesh Soni | a7e670d | 2006-01-09 20:51:53 -0800 | [diff] [blame] | 461 | |
| 462 | |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 463 | To Do |
| 464 | ===== |
| 465 | |
Simon Horman | 3043013 | 2007-02-20 13:58:07 -0800 | [diff] [blame] | 466 | 1) Provide relocatable kernels for all architectures to help in maintaining |
| 467 | multiple kernels for crash_dump, and the same kernel as the system kernel |
| 468 | can be used to capture the dump. |
Maneesh Soni | a7e670d | 2006-01-09 20:51:53 -0800 | [diff] [blame] | 469 | |
Vivek Goyal | b089f4a | 2005-06-25 14:58:15 -0700 | [diff] [blame] | 470 | |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 471 | Contact |
Vivek Goyal | b089f4a | 2005-06-25 14:58:15 -0700 | [diff] [blame] | 472 | ======= |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 473 | |
Simon Horman | db6857c | 2010-11-25 10:30:12 +0900 | [diff] [blame] | 474 | Vivek Goyal (vgoyal@redhat.com) |
Vivek Goyal | d58831e | 2005-06-25 14:58:17 -0700 | [diff] [blame] | 475 | Maneesh Soni (maneesh@in.ibm.com) |
David Wilder | dc851a0 | 2006-06-25 05:47:55 -0700 | [diff] [blame] | 476 | |