Jason Wessel | e3e2aaf | 2008-03-20 13:43:45 -0500 | [diff] [blame] | 1 | <?xml version="1.0" encoding="UTF-8"?> |
| 2 | <!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN" |
| 3 | "http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd" []> |
| 4 | |
| 5 | <book id="kgdbOnLinux"> |
| 6 | <bookinfo> |
| 7 | <title>Using kgdb and the kgdb Internals</title> |
| 8 | |
| 9 | <authorgroup> |
| 10 | <author> |
| 11 | <firstname>Jason</firstname> |
| 12 | <surname>Wessel</surname> |
| 13 | <affiliation> |
| 14 | <address> |
| 15 | <email>jason.wessel@windriver.com</email> |
| 16 | </address> |
| 17 | </affiliation> |
| 18 | </author> |
| 19 | </authorgroup> |
| 20 | |
| 21 | <authorgroup> |
| 22 | <author> |
| 23 | <firstname>Tom</firstname> |
| 24 | <surname>Rini</surname> |
| 25 | <affiliation> |
| 26 | <address> |
| 27 | <email>trini@kernel.crashing.org</email> |
| 28 | </address> |
| 29 | </affiliation> |
| 30 | </author> |
| 31 | </authorgroup> |
| 32 | |
| 33 | <authorgroup> |
| 34 | <author> |
| 35 | <firstname>Amit S.</firstname> |
| 36 | <surname>Kale</surname> |
| 37 | <affiliation> |
| 38 | <address> |
| 39 | <email>amitkale@linsyssoft.com</email> |
| 40 | </address> |
| 41 | </affiliation> |
| 42 | </author> |
| 43 | </authorgroup> |
| 44 | |
| 45 | <copyright> |
| 46 | <year>2008</year> |
| 47 | <holder>Wind River Systems, Inc.</holder> |
| 48 | </copyright> |
| 49 | <copyright> |
| 50 | <year>2004-2005</year> |
| 51 | <holder>MontaVista Software, Inc.</holder> |
| 52 | </copyright> |
| 53 | <copyright> |
| 54 | <year>2004</year> |
| 55 | <holder>Amit S. Kale</holder> |
| 56 | </copyright> |
| 57 | |
| 58 | <legalnotice> |
| 59 | <para> |
| 60 | This file is licensed under the terms of the GNU General Public License |
| 61 | version 2. This program is licensed "as is" without any warranty of any |
| 62 | kind, whether express or implied. |
| 63 | </para> |
| 64 | |
| 65 | </legalnotice> |
| 66 | </bookinfo> |
| 67 | |
| 68 | <toc></toc> |
| 69 | <chapter id="Introduction"> |
| 70 | <title>Introduction</title> |
| 71 | <para> |
| 72 | kgdb is a source level debugger for linux kernel. It is used along |
| 73 | with gdb to debug a linux kernel. The expectation is that gdb can |
| 74 | be used to "break in" to the kernel to inspect memory, variables |
grzegorz.chwesewicz@chilan.com | f925093 | 2008-04-24 16:57:22 -0500 | [diff] [blame] | 75 | and look through call stack information similar to what an |
Jason Wessel | e3e2aaf | 2008-03-20 13:43:45 -0500 | [diff] [blame] | 76 | application developer would use gdb for. It is possible to place |
| 77 | breakpoints in kernel code and perform some limited execution |
| 78 | stepping. |
| 79 | </para> |
| 80 | <para> |
| 81 | Two machines are required for using kgdb. One of these machines is a |
| 82 | development machine and the other is a test machine. The kernel |
| 83 | to be debugged runs on the test machine. The development machine |
| 84 | runs an instance of gdb against the vmlinux file which contains |
| 85 | the symbols (not boot image such as bzImage, zImage, uImage...). |
| 86 | In gdb the developer specifies the connection parameters and |
Jason Wessel | a606b5e | 2008-06-24 10:52:55 -0500 | [diff] [blame] | 87 | connects to kgdb. The type of connection a developer makes with |
| 88 | gdb depends on the availability of kgdb I/O modules compiled as |
| 89 | builtin's or kernel modules in the test machine's kernel. |
Jason Wessel | e3e2aaf | 2008-03-20 13:43:45 -0500 | [diff] [blame] | 90 | </para> |
| 91 | </chapter> |
| 92 | <chapter id="CompilingAKernel"> |
| 93 | <title>Compiling a kernel</title> |
| 94 | <para> |
grzegorz.chwesewicz@chilan.com | f925093 | 2008-04-24 16:57:22 -0500 | [diff] [blame] | 95 | To enable <symbol>CONFIG_KGDB</symbol> you should first turn on |
| 96 | "Prompt for development and/or incomplete code/drivers" |
| 97 | (CONFIG_EXPERIMENTAL) in "General setup", then under the |
| 98 | "Kernel debugging" select "KGDB: kernel debugging with remote gdb". |
Jason Wessel | e3e2aaf | 2008-03-20 13:43:45 -0500 | [diff] [blame] | 99 | </para> |
| 100 | <para> |
Jason Wessel | 5f5ddfb | 2008-08-01 08:39:34 -0500 | [diff] [blame^] | 101 | It is advised, but not required that you turn on the |
| 102 | CONFIG_FRAME_POINTER kernel option. This option inserts code to |
| 103 | into the compiled executable which saves the frame information in |
| 104 | registers or on the stack at different points which will allow a |
| 105 | debugger such as gdb to more accurately construct stack back traces |
| 106 | while debugging the kernel. |
| 107 | </para> |
| 108 | <para> |
Jason Wessel | e3e2aaf | 2008-03-20 13:43:45 -0500 | [diff] [blame] | 109 | Next you should choose one of more I/O drivers to interconnect debugging |
| 110 | host and debugged target. Early boot debugging requires a KGDB |
| 111 | I/O driver that supports early debugging and the driver must be |
| 112 | built into the kernel directly. Kgdb I/O driver configuration |
| 113 | takes place via kernel or module parameters, see following |
| 114 | chapter. |
| 115 | </para> |
| 116 | <para> |
| 117 | The kgdb test compile options are described in the kgdb test suite chapter. |
| 118 | </para> |
| 119 | |
| 120 | </chapter> |
| 121 | <chapter id="EnableKGDB"> |
| 122 | <title>Enable kgdb for debugging</title> |
| 123 | <para> |
| 124 | In order to use kgdb you must activate it by passing configuration |
| 125 | information to one of the kgdb I/O drivers. If you do not pass any |
| 126 | configuration information kgdb will not do anything at all. Kgdb |
| 127 | will only actively hook up to the kernel trap hooks if a kgdb I/O |
| 128 | driver is loaded and configured. If you unconfigure a kgdb I/O |
| 129 | driver, kgdb will unregister all the kernel hook points. |
| 130 | </para> |
| 131 | <para> |
| 132 | All drivers can be reconfigured at run time, if |
| 133 | <symbol>CONFIG_SYSFS</symbol> and <symbol>CONFIG_MODULES</symbol> |
| 134 | are enabled, by echo'ing a new config string to |
| 135 | <constant>/sys/module/<driver>/parameter/<option></constant>. |
| 136 | The driver can be unconfigured by passing an empty string. You cannot |
| 137 | change the configuration while the debugger is attached. Make sure |
| 138 | to detach the debugger with the <constant>detach</constant> command |
| 139 | prior to trying unconfigure a kgdb I/O driver. |
| 140 | </para> |
| 141 | <sect1 id="kgdbwait"> |
| 142 | <title>Kernel parameter: kgdbwait</title> |
| 143 | <para> |
| 144 | The Kernel command line option <constant>kgdbwait</constant> makes |
| 145 | kgdb wait for a debugger connection during booting of a kernel. You |
| 146 | can only use this option you compiled a kgdb I/O driver into the |
| 147 | kernel and you specified the I/O driver configuration as a kernel |
| 148 | command line option. The kgdbwait parameter should always follow the |
| 149 | configuration parameter for the kgdb I/O driver in the kernel |
| 150 | command line else the I/O driver will not be configured prior to |
| 151 | asking the kernel to use it to wait. |
| 152 | </para> |
| 153 | <para> |
| 154 | The kernel will stop and wait as early as the I/O driver and |
| 155 | architecture will allow when you use this option. If you build the |
| 156 | kgdb I/O driver as a kernel module kgdbwait will not do anything. |
| 157 | </para> |
| 158 | </sect1> |
| 159 | <sect1 id="kgdboc"> |
| 160 | <title>Kernel parameter: kgdboc</title> |
| 161 | <para> |
| 162 | The kgdboc driver was originally an abbreviation meant to stand for |
| 163 | "kgdb over console". Kgdboc is designed to work with a single |
Jason Wessel | 225a442 | 2008-04-01 16:55:26 -0500 | [diff] [blame] | 164 | serial port. It was meant to cover the circumstance |
Jason Wessel | e3e2aaf | 2008-03-20 13:43:45 -0500 | [diff] [blame] | 165 | where you wanted to use a serial console as your primary console as |
Jason Wessel | 225a442 | 2008-04-01 16:55:26 -0500 | [diff] [blame] | 166 | well as using it to perform kernel debugging. Of course you can |
| 167 | also use kgdboc without assigning a console to the same port. |
Jason Wessel | e3e2aaf | 2008-03-20 13:43:45 -0500 | [diff] [blame] | 168 | </para> |
| 169 | <sect2 id="UsingKgdboc"> |
| 170 | <title>Using kgdboc</title> |
| 171 | <para> |
| 172 | You can configure kgdboc via sysfs or a module or kernel boot line |
| 173 | parameter depending on if you build with CONFIG_KGDBOC as a module |
| 174 | or built-in. |
| 175 | <orderedlist> |
| 176 | <listitem><para>From the module load or build-in</para> |
| 177 | <para><constant>kgdboc=<tty-device>,[baud]</constant></para> |
| 178 | <para> |
| 179 | The example here would be if your console port was typically ttyS0, you would use something like <constant>kgdboc=ttyS0,115200</constant> or on the ARM Versatile AB you would likely use <constant>kgdboc=ttyAMA0,115200</constant> |
| 180 | </para> |
| 181 | </listitem> |
| 182 | <listitem><para>From sysfs</para> |
| 183 | <para><constant>echo ttyS0 > /sys/module/kgdboc/parameters/kgdboc</constant></para> |
| 184 | </listitem> |
| 185 | </orderedlist> |
| 186 | </para> |
| 187 | <para> |
| 188 | NOTE: Kgdboc does not support interrupting the target via the |
| 189 | gdb remote protocol. You must manually send a sysrq-g unless you |
| 190 | have a proxy that splits console output to a terminal problem and |
| 191 | has a separate port for the debugger to connect to that sends the |
| 192 | sysrq-g for you. |
| 193 | </para> |
| 194 | <para>When using kgdboc with no debugger proxy, you can end up |
| 195 | connecting the debugger for one of two entry points. If an |
| 196 | exception occurs after you have loaded kgdboc a message should print |
| 197 | on the console stating it is waiting for the debugger. In case you |
| 198 | disconnect your terminal program and then connect the debugger in |
| 199 | its place. If you want to interrupt the target system and forcibly |
| 200 | enter a debug session you have to issue a Sysrq sequence and then |
| 201 | type the letter <constant>g</constant>. Then you disconnect the |
| 202 | terminal session and connect gdb. Your options if you don't like |
| 203 | this are to hack gdb to send the sysrq-g for you as well as on the |
| 204 | initial connect, or to use a debugger proxy that allows an |
| 205 | unmodified gdb to do the debugging. |
| 206 | </para> |
| 207 | </sect2> |
Jason Wessel | e3e2aaf | 2008-03-20 13:43:45 -0500 | [diff] [blame] | 208 | </sect1> |
| 209 | <sect1 id="kgdbcon"> |
| 210 | <title>Kernel parameter: kgdbcon</title> |
| 211 | <para> |
| 212 | Kgdb supports using the gdb serial protocol to send console messages |
| 213 | to the debugger when the debugger is connected and running. There |
| 214 | are two ways to activate this feature. |
| 215 | <orderedlist> |
| 216 | <listitem><para>Activate with the kernel command line option:</para> |
| 217 | <para><constant>kgdbcon</constant></para> |
| 218 | </listitem> |
| 219 | <listitem><para>Use sysfs before configuring an io driver</para> |
| 220 | <para> |
| 221 | <constant>echo 1 > /sys/module/kgdb/parameters/kgdb_use_con</constant> |
| 222 | </para> |
| 223 | <para> |
| 224 | NOTE: If you do this after you configure the kgdb I/O driver, the |
| 225 | setting will not take effect until the next point the I/O is |
| 226 | reconfigured. |
| 227 | </para> |
| 228 | </listitem> |
| 229 | </orderedlist> |
| 230 | </para> |
| 231 | <para> |
| 232 | IMPORTANT NOTE: Using this option with kgdb over the console |
Jason Wessel | a606b5e | 2008-06-24 10:52:55 -0500 | [diff] [blame] | 233 | (kgdboc) is not supported. |
Jason Wessel | e3e2aaf | 2008-03-20 13:43:45 -0500 | [diff] [blame] | 234 | </para> |
| 235 | </sect1> |
| 236 | </chapter> |
| 237 | <chapter id="ConnectingGDB"> |
| 238 | <title>Connecting gdb</title> |
| 239 | <para> |
| 240 | If you are using kgdboc, you need to have used kgdbwait as a boot |
| 241 | argument, issued a sysrq-g, or the system you are going to debug |
| 242 | has already taken an exception and is waiting for the debugger to |
| 243 | attach before you can connect gdb. |
| 244 | </para> |
| 245 | <para> |
| 246 | If you are not using different kgdb I/O driver other than kgdboc, |
| 247 | you should be able to connect and the target will automatically |
| 248 | respond. |
| 249 | </para> |
| 250 | <para> |
| 251 | Example (using a serial port): |
| 252 | </para> |
| 253 | <programlisting> |
| 254 | % gdb ./vmlinux |
| 255 | (gdb) set remotebaud 115200 |
| 256 | (gdb) target remote /dev/ttyS0 |
| 257 | </programlisting> |
| 258 | <para> |
Jason Wessel | a606b5e | 2008-06-24 10:52:55 -0500 | [diff] [blame] | 259 | Example (kgdb to a terminal server on tcp port 2012): |
Jason Wessel | e3e2aaf | 2008-03-20 13:43:45 -0500 | [diff] [blame] | 260 | </para> |
| 261 | <programlisting> |
| 262 | % gdb ./vmlinux |
Jason Wessel | a606b5e | 2008-06-24 10:52:55 -0500 | [diff] [blame] | 263 | (gdb) target remote 192.168.2.2:2012 |
Jason Wessel | e3e2aaf | 2008-03-20 13:43:45 -0500 | [diff] [blame] | 264 | </programlisting> |
| 265 | <para> |
| 266 | Once connected, you can debug a kernel the way you would debug an |
| 267 | application program. |
| 268 | </para> |
| 269 | <para> |
| 270 | If you are having problems connecting or something is going |
| 271 | seriously wrong while debugging, it will most often be the case |
| 272 | that you want to enable gdb to be verbose about its target |
| 273 | communications. You do this prior to issuing the <constant>target |
| 274 | remote</constant> command by typing in: <constant>set remote debug 1</constant> |
| 275 | </para> |
| 276 | </chapter> |
| 277 | <chapter id="KGDBTestSuite"> |
| 278 | <title>kgdb Test Suite</title> |
| 279 | <para> |
| 280 | When kgdb is enabled in the kernel config you can also elect to |
| 281 | enable the config parameter KGDB_TESTS. Turning this on will |
| 282 | enable a special kgdb I/O module which is designed to test the |
| 283 | kgdb internal functions. |
| 284 | </para> |
| 285 | <para> |
| 286 | The kgdb tests are mainly intended for developers to test the kgdb |
| 287 | internals as well as a tool for developing a new kgdb architecture |
| 288 | specific implementation. These tests are not really for end users |
| 289 | of the Linux kernel. The primary source of documentation would be |
| 290 | to look in the drivers/misc/kgdbts.c file. |
| 291 | </para> |
| 292 | <para> |
| 293 | The kgdb test suite can also be configured at compile time to run |
| 294 | the core set of tests by setting the kernel config parameter |
| 295 | KGDB_TESTS_ON_BOOT. This particular option is aimed at automated |
| 296 | regression testing and does not require modifying the kernel boot |
| 297 | config arguments. If this is turned on, the kgdb test suite can |
| 298 | be disabled by specifying "kgdbts=" as a kernel boot argument. |
| 299 | </para> |
| 300 | </chapter> |
| 301 | <chapter id="CommonBackEndReq"> |
Jason Wessel | 225a442 | 2008-04-01 16:55:26 -0500 | [diff] [blame] | 302 | <title>KGDB Internals</title> |
| 303 | <sect1 id="kgdbArchitecture"> |
Jason Wessel | e3e2aaf | 2008-03-20 13:43:45 -0500 | [diff] [blame] | 304 | <title>Architecture Specifics</title> |
| 305 | <para> |
| 306 | Kgdb is organized into three basic components: |
| 307 | <orderedlist> |
| 308 | <listitem><para>kgdb core</para> |
| 309 | <para> |
| 310 | The kgdb core is found in kernel/kgdb.c. It contains: |
| 311 | <itemizedlist> |
| 312 | <listitem><para>All the logic to implement the gdb serial protocol</para></listitem> |
| 313 | <listitem><para>A generic OS exception handler which includes sync'ing the processors into a stopped state on an multi cpu system.</para></listitem> |
| 314 | <listitem><para>The API to talk to the kgdb I/O drivers</para></listitem> |
| 315 | <listitem><para>The API to make calls to the arch specific kgdb implementation</para></listitem> |
| 316 | <listitem><para>The logic to perform safe memory reads and writes to memory while using the debugger</para></listitem> |
| 317 | <listitem><para>A full implementation for software breakpoints unless overridden by the arch</para></listitem> |
| 318 | </itemizedlist> |
| 319 | </para> |
| 320 | </listitem> |
| 321 | <listitem><para>kgdb arch specific implementation</para> |
| 322 | <para> |
| 323 | This implementation is generally found in arch/*/kernel/kgdb.c. |
| 324 | As an example, arch/x86/kernel/kgdb.c contains the specifics to |
| 325 | implement HW breakpoint as well as the initialization to |
| 326 | dynamically register and unregister for the trap handlers on |
| 327 | this architecture. The arch specific portion implements: |
| 328 | <itemizedlist> |
| 329 | <listitem><para>contains an arch specific trap catcher which |
| 330 | invokes kgdb_handle_exception() to start kgdb about doing its |
| 331 | work</para></listitem> |
| 332 | <listitem><para>translation to and from gdb specific packet format to pt_regs</para></listitem> |
| 333 | <listitem><para>Registration and unregistration of architecture specific trap hooks</para></listitem> |
| 334 | <listitem><para>Any special exception handling and cleanup</para></listitem> |
| 335 | <listitem><para>NMI exception handling and cleanup</para></listitem> |
| 336 | <listitem><para>(optional)HW breakpoints</para></listitem> |
| 337 | </itemizedlist> |
| 338 | </para> |
| 339 | </listitem> |
| 340 | <listitem><para>kgdb I/O driver</para> |
| 341 | <para> |
Jason Wessel | 225a442 | 2008-04-01 16:55:26 -0500 | [diff] [blame] | 342 | Each kgdb I/O driver has to provide an implemenation for the following: |
| 343 | <itemizedlist> |
| 344 | <listitem><para>configuration via builtin or module</para></listitem> |
| 345 | <listitem><para>dynamic configuration and kgdb hook registration calls</para></listitem> |
| 346 | <listitem><para>read and write character interface</para></listitem> |
| 347 | <listitem><para>A cleanup handler for unconfiguring from the kgdb core</para></listitem> |
| 348 | <listitem><para>(optional) Early debug methodology</para></listitem> |
| 349 | </itemizedlist> |
| 350 | Any given kgdb I/O driver has to operate very closely with the |
| 351 | hardware and must do it in such a way that does not enable |
| 352 | interrupts or change other parts of the system context without |
| 353 | completely restoring them. The kgdb core will repeatedly "poll" |
| 354 | a kgdb I/O driver for characters when it needs input. The I/O |
| 355 | driver is expected to return immediately if there is no data |
| 356 | available. Doing so allows for the future possibility to touch |
| 357 | watch dog hardware in such a way as to have a target system not |
| 358 | reset when these are enabled. |
Jason Wessel | e3e2aaf | 2008-03-20 13:43:45 -0500 | [diff] [blame] | 359 | </para> |
| 360 | </listitem> |
| 361 | </orderedlist> |
| 362 | </para> |
| 363 | <para> |
| 364 | If you are intent on adding kgdb architecture specific support |
| 365 | for a new architecture, the architecture should define |
| 366 | <constant>HAVE_ARCH_KGDB</constant> in the architecture specific |
| 367 | Kconfig file. This will enable kgdb for the architecture, and |
| 368 | at that point you must create an architecture specific kgdb |
| 369 | implementation. |
| 370 | </para> |
| 371 | <para> |
| 372 | There are a few flags which must be set on every architecture in |
| 373 | their <asm/kgdb.h> file. These are: |
| 374 | <itemizedlist> |
| 375 | <listitem> |
| 376 | <para> |
| 377 | NUMREGBYTES: The size in bytes of all of the registers, so |
| 378 | that we can ensure they will all fit into a packet. |
| 379 | </para> |
| 380 | <para> |
| 381 | BUFMAX: The size in bytes of the buffer GDB will read into. |
| 382 | This must be larger than NUMREGBYTES. |
| 383 | </para> |
| 384 | <para> |
| 385 | CACHE_FLUSH_IS_SAFE: Set to 1 if it is always safe to call |
| 386 | flush_cache_range or flush_icache_range. On some architectures, |
| 387 | these functions may not be safe to call on SMP since we keep other |
| 388 | CPUs in a holding pattern. |
| 389 | </para> |
| 390 | </listitem> |
| 391 | </itemizedlist> |
| 392 | </para> |
| 393 | <para> |
| 394 | There are also the following functions for the common backend, |
| 395 | found in kernel/kgdb.c, that must be supplied by the |
| 396 | architecture-specific backend unless marked as (optional), in |
| 397 | which case a default function maybe used if the architecture |
| 398 | does not need to provide a specific implementation. |
| 399 | </para> |
| 400 | !Iinclude/linux/kgdb.h |
Jason Wessel | 225a442 | 2008-04-01 16:55:26 -0500 | [diff] [blame] | 401 | </sect1> |
| 402 | <sect1 id="kgdbocDesign"> |
| 403 | <title>kgdboc internals</title> |
| 404 | <para> |
| 405 | The kgdboc driver is actually a very thin driver that relies on the |
| 406 | underlying low level to the hardware driver having "polling hooks" |
| 407 | which the to which the tty driver is attached. In the initial |
| 408 | implementation of kgdboc it the serial_core was changed to expose a |
| 409 | low level uart hook for doing polled mode reading and writing of a |
| 410 | single character while in an atomic context. When kgdb makes an I/O |
| 411 | request to the debugger, kgdboc invokes a call back in the serial |
| 412 | core which in turn uses the call back in the uart driver. It is |
| 413 | certainly possible to extend kgdboc to work with non-uart based |
| 414 | consoles in the future. |
| 415 | </para> |
| 416 | <para> |
| 417 | When using kgdboc with a uart, the uart driver must implement two callbacks in the <constant>struct uart_ops</constant>. Example from drivers/8250.c:<programlisting> |
| 418 | #ifdef CONFIG_CONSOLE_POLL |
| 419 | .poll_get_char = serial8250_get_poll_char, |
| 420 | .poll_put_char = serial8250_put_poll_char, |
| 421 | #endif |
| 422 | </programlisting> |
| 423 | Any implementation specifics around creating a polling driver use the |
| 424 | <constant>#ifdef CONFIG_CONSOLE_POLL</constant>, as shown above. |
| 425 | Keep in mind that polling hooks have to be implemented in such a way |
| 426 | that they can be called from an atomic context and have to restore |
| 427 | the state of the uart chip on return such that the system can return |
| 428 | to normal when the debugger detaches. You need to be very careful |
| 429 | with any kind of lock you consider, because failing here is most |
| 430 | going to mean pressing the reset button. |
| 431 | </para> |
| 432 | </sect1> |
Jason Wessel | e3e2aaf | 2008-03-20 13:43:45 -0500 | [diff] [blame] | 433 | </chapter> |
| 434 | <chapter id="credits"> |
| 435 | <title>Credits</title> |
| 436 | <para> |
| 437 | The following people have contributed to this document: |
| 438 | <orderedlist> |
| 439 | <listitem><para>Amit Kale<email>amitkale@linsyssoft.com</email></para></listitem> |
| 440 | <listitem><para>Tom Rini<email>trini@kernel.crashing.org</email></para></listitem> |
Jason Wessel | e3e2aaf | 2008-03-20 13:43:45 -0500 | [diff] [blame] | 441 | </orderedlist> |
Jason Wessel | 225a442 | 2008-04-01 16:55:26 -0500 | [diff] [blame] | 442 | In March 2008 this document was completely rewritten by: |
| 443 | <itemizedlist> |
| 444 | <listitem><para>Jason Wessel<email>jason.wessel@windriver.com</email></para></listitem> |
| 445 | </itemizedlist> |
Jason Wessel | e3e2aaf | 2008-03-20 13:43:45 -0500 | [diff] [blame] | 446 | </para> |
| 447 | </chapter> |
| 448 | </book> |
| 449 | |