Vivek Goyal | b089f4a | 2005-06-25 14:58:15 -0700 | [diff] [blame] | 1 | Documentation for kdump - the kexec-based crash dumping solution |
| 2 | ================================================================ |
| 3 | |
| 4 | DESIGN |
| 5 | ====== |
| 6 | |
| 7 | Kdump uses kexec to reboot to a second kernel whenever a dump needs to be taken. |
| 8 | This second kernel is booted with very little memory. The first kernel reserves |
| 9 | the section of memory that the second kernel uses. This ensures that on-going |
| 10 | DMA from the first kernel does not corrupt the second kernel. |
| 11 | |
| 12 | All the necessary information about Core image is encoded in ELF format and |
| 13 | stored in reserved area of memory before crash. Physical address of start of |
| 14 | ELF header is passed to new kernel through command line parameter elfcorehdr=. |
| 15 | |
| 16 | On i386, the first 640 KB of physical memory is needed to boot, irrespective |
| 17 | of where the kernel loads. Hence, this region is backed up by kexec just before |
| 18 | rebooting into the new kernel. |
| 19 | |
| 20 | In the second kernel, "old memory" can be accessed in two ways. |
| 21 | |
| 22 | - The first one is through a /dev/oldmem device interface. A capture utility |
| 23 | can read the device file and write out the memory in raw format. This is raw |
| 24 | dump of memory and analysis/capture tool should be intelligent enough to |
| 25 | determine where to look for the right information. ELF headers (elfcorehdr=) |
| 26 | can become handy here. |
| 27 | |
| 28 | - The second interface is through /proc/vmcore. This exports the dump as an ELF |
| 29 | format file which can be written out using any file copy command |
| 30 | (cp, scp, etc). Further, gdb can be used to perform limited debugging on |
| 31 | the dump file. This method ensures methods ensure that there is correct |
| 32 | ordering of the dump pages (corresponding to the first 640 KB that has been |
| 33 | relocated). |
| 34 | |
| 35 | SETUP |
| 36 | ===== |
| 37 | |
| 38 | 1) Download http://www.xmission.com/~ebiederm/files/kexec/kexec-tools-1.101.tar.gz |
| 39 | and apply http://lse.sourceforge.net/kdump/patches/kexec-tools-1.101-kdump.patch |
| 40 | and after that build the source. |
| 41 | |
| 42 | 2) Download and build the appropriate (latest) kexec/kdump (-mm) kernel |
| 43 | patchset and apply it to the vanilla kernel tree. |
| 44 | |
| 45 | Two kernels need to be built in order to get this feature working. |
| 46 | |
| 47 | A) First kernel: |
| 48 | a) Enable "kexec system call" feature (in Processor type and features). |
| 49 | CONFIG_KEXEC=y |
| 50 | b) This kernel's physical load address should be the default value of |
| 51 | 0x100000 (0x100000, 1 MB) (in Processor type and features). |
| 52 | CONFIG_PHYSICAL_START=0x100000 |
| 53 | c) Enable "sysfs file system support" (in Pseudo filesystems). |
| 54 | CONFIG_SYSFS=y |
| 55 | d) Boot into first kernel with the command line parameter "crashkernel=Y@X". |
| 56 | Use appropriate values for X and Y. Y denotes how much memory to reserve |
| 57 | for the second kernel, and X denotes at what physical address the reserved |
| 58 | memory section starts. For example: "crashkernel=64M@16M". |
| 59 | |
| 60 | B) Second kernel: |
| 61 | a) Enable "kernel crash dumps" feature (in Processor type and features). |
| 62 | CONFIG_CRASH_DUMP=y |
| 63 | b) Specify a suitable value for "Physical address where the kernel is |
| 64 | loaded" (in Processor type and features). Typically this value |
| 65 | should be same as X (See option d) above, e.g., 16 MB or 0x1000000. |
| 66 | CONFIG_PHYSICAL_START=0x1000000 |
| 67 | c) Enable "/proc/vmcore support" (Optional, in Pseudo filesystems). |
| 68 | CONFIG_PROC_VMCORE=y |
Vivek Goyal | d58831e | 2005-06-25 14:58:17 -0700 | [diff] [blame] | 69 | d) Disable SMP support and build a UP kernel (Until it is fixed). |
| 70 | CONFIG_SMP=n |
| 71 | e) Enable "Local APIC support on uniprocessors". |
| 72 | CONFIG_X86_UP_APIC=y |
| 73 | f) Enable "IO-APIC support on uniprocessors" |
| 74 | CONFIG_X86_UP_IOAPIC=y |
Vivek Goyal | b089f4a | 2005-06-25 14:58:15 -0700 | [diff] [blame] | 75 | |
Vivek Goyal | d58831e | 2005-06-25 14:58:17 -0700 | [diff] [blame] | 76 | Note: i) Options a) and b) depend upon "Configure standard kernel features |
| 77 | (for small systems)" (under General setup). |
| 78 | ii) Option a) also depends on CONFIG_HIGHMEM (under Processor |
Vivek Goyal | b089f4a | 2005-06-25 14:58:15 -0700 | [diff] [blame] | 79 | type and features). |
Vivek Goyal | d58831e | 2005-06-25 14:58:17 -0700 | [diff] [blame] | 80 | iii) Both option a) and b) are under "Processor type and features". |
Vivek Goyal | b089f4a | 2005-06-25 14:58:15 -0700 | [diff] [blame] | 81 | |
| 82 | 3) Boot into the first kernel. You are now ready to try out kexec-based crash |
| 83 | dumps. |
| 84 | |
| 85 | 4) Load the second kernel to be booted using: |
| 86 | |
| 87 | kexec -p <second-kernel> --crash-dump --args-linux --append="root=<root-dev> |
Vivek Goyal | d58831e | 2005-06-25 14:58:17 -0700 | [diff] [blame] | 88 | init 1 irqpoll" |
Vivek Goyal | b089f4a | 2005-06-25 14:58:15 -0700 | [diff] [blame] | 89 | |
| 90 | Note: i) <second-kernel> has to be a vmlinux image. bzImage will not work, |
| 91 | as of now. |
| 92 | ii) By default ELF headers are stored in ELF32 format (for i386). This |
| 93 | is sufficient to represent the physical memory up to 4GB. To store |
| 94 | headers in ELF64 format, specifiy "--elf64-core-headers" on the |
| 95 | kexec command line additionally. |
Vivek Goyal | d58831e | 2005-06-25 14:58:17 -0700 | [diff] [blame] | 96 | iii) Specify "irqpoll" as command line parameter. This reduces driver |
| 97 | initialization failures in second kernel due to shared interrupts. |
Vivek Goyal | b089f4a | 2005-06-25 14:58:15 -0700 | [diff] [blame] | 98 | |
| 99 | 5) System reboots into the second kernel when a panic occurs. A module can be |
Vivek Goyal | d58831e | 2005-06-25 14:58:17 -0700 | [diff] [blame] | 100 | written to force the panic or "ALT-SysRq-c" can be used initiate a crash |
| 101 | dump for testing purposes. |
Vivek Goyal | b089f4a | 2005-06-25 14:58:15 -0700 | [diff] [blame] | 102 | |
| 103 | 6) Write out the dump file using |
| 104 | |
| 105 | cp /proc/vmcore <dump-file> |
| 106 | |
| 107 | Dump memory can also be accessed as a /dev/oldmem device for a linear/raw |
| 108 | view. To create the device, type: |
| 109 | |
| 110 | mknod /dev/oldmem c 1 12 |
| 111 | |
| 112 | Use "dd" with suitable options for count, bs and skip to access specific |
| 113 | portions of the dump. |
| 114 | |
| 115 | Entire memory: dd if=/dev/oldmem of=oldmem.001 |
| 116 | |
| 117 | ANALYSIS |
| 118 | ======== |
| 119 | |
| 120 | Limited analysis can be done using gdb on the dump file copied out of |
| 121 | /proc/vmcore. Use vmlinux built with -g and run |
| 122 | |
| 123 | gdb vmlinux <dump-file> |
| 124 | |
| 125 | Stack trace for the task on processor 0, register display, memory display |
| 126 | work fine. |
| 127 | |
| 128 | Note: gdb cannot analyse core files generated in ELF64 format for i386. |
| 129 | |
| 130 | TODO |
| 131 | ==== |
| 132 | |
| 133 | 1) Provide a kernel pages filtering mechanism so that core file size is not |
| 134 | insane on systems having huge memory banks. |
| 135 | 2) Modify "crash" tool to make it recognize this dump. |
| 136 | |
| 137 | CONTACT |
| 138 | ======= |
| 139 | |
Vivek Goyal | b089f4a | 2005-06-25 14:58:15 -0700 | [diff] [blame] | 140 | Vivek Goyal (vgoyal@in.ibm.com) |
Vivek Goyal | d58831e | 2005-06-25 14:58:17 -0700 | [diff] [blame] | 141 | Maneesh Soni (maneesh@in.ibm.com) |