Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 1 | <?xml version="1.0" encoding="UTF-8"?> |
| 2 | <!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN" |
| 3 | "http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd" []> |
| 4 | |
| 5 | <book id="drmDevelopersGuide"> |
| 6 | <bookinfo> |
| 7 | <title>Linux DRM Developer's Guide</title> |
| 8 | |
| 9 | <copyright> |
| 10 | <year>2008-2009</year> |
| 11 | <holder> |
| 12 | Intel Corporation (Jesse Barnes <jesse.barnes@intel.com>) |
| 13 | </holder> |
| 14 | </copyright> |
| 15 | |
| 16 | <legalnotice> |
| 17 | <para> |
| 18 | The contents of this file may be used under the terms of the GNU |
| 19 | General Public License version 2 (the "GPL") as distributed in |
| 20 | the kernel source COPYING file. |
| 21 | </para> |
| 22 | </legalnotice> |
| 23 | </bookinfo> |
| 24 | |
| 25 | <toc></toc> |
| 26 | |
| 27 | <!-- Introduction --> |
| 28 | |
| 29 | <chapter id="drmIntroduction"> |
| 30 | <title>Introduction</title> |
| 31 | <para> |
| 32 | The Linux DRM layer contains code intended to support the needs |
| 33 | of complex graphics devices, usually containing programmable |
| 34 | pipelines well suited to 3D graphics acceleration. Graphics |
Michael Witten | f11aca0 | 2011-08-25 17:21:31 +0000 | [diff] [blame] | 35 | drivers in the kernel may make use of DRM functions to make |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 36 | tasks like memory management, interrupt handling and DMA easier, |
| 37 | and provide a uniform interface to applications. |
| 38 | </para> |
| 39 | <para> |
| 40 | A note on versions: this guide covers features found in the DRM |
| 41 | tree, including the TTM memory manager, output configuration and |
| 42 | mode setting, and the new vblank internals, in addition to all |
| 43 | the regular features found in current kernels. |
| 44 | </para> |
| 45 | <para> |
| 46 | [Insert diagram of typical DRM stack here] |
| 47 | </para> |
| 48 | </chapter> |
| 49 | |
| 50 | <!-- Internals --> |
| 51 | |
| 52 | <chapter id="drmInternals"> |
| 53 | <title>DRM Internals</title> |
| 54 | <para> |
| 55 | This chapter documents DRM internals relevant to driver authors |
| 56 | and developers working to add support for the latest features to |
| 57 | existing drivers. |
| 58 | </para> |
| 59 | <para> |
Michael Witten | a78f678 | 2011-08-25 17:18:08 +0000 | [diff] [blame] | 60 | First, we go over some typical driver initialization |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 61 | requirements, like setting up command buffers, creating an |
| 62 | initial output configuration, and initializing core services. |
Michael Witten | a78f678 | 2011-08-25 17:18:08 +0000 | [diff] [blame] | 63 | Subsequent sections cover core internals in more detail, |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 64 | providing implementation notes and examples. |
| 65 | </para> |
| 66 | <para> |
| 67 | The DRM layer provides several services to graphics drivers, |
| 68 | many of them driven by the application interfaces it provides |
| 69 | through libdrm, the library that wraps most of the DRM ioctls. |
| 70 | These include vblank event handling, memory |
| 71 | management, output management, framebuffer management, command |
| 72 | submission & fencing, suspend/resume support, and DMA |
| 73 | services. |
| 74 | </para> |
| 75 | <para> |
Xiao Jiang | bd91572 | 2011-02-14 17:58:23 +0800 | [diff] [blame] | 76 | The core of every DRM driver is struct drm_driver. Drivers |
Michael Witten | a78f678 | 2011-08-25 17:18:08 +0000 | [diff] [blame] | 77 | typically statically initialize a drm_driver structure, |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 78 | then pass it to drm_init() at load time. |
| 79 | </para> |
| 80 | |
| 81 | <!-- Internals: driver init --> |
| 82 | |
| 83 | <sect1> |
| 84 | <title>Driver initialization</title> |
| 85 | <para> |
| 86 | Before calling the DRM initialization routines, the driver must |
Xiao Jiang | bd91572 | 2011-02-14 17:58:23 +0800 | [diff] [blame] | 87 | first create and fill out a struct drm_driver structure. |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 88 | </para> |
| 89 | <programlisting> |
| 90 | static struct drm_driver driver = { |
Michael Witten | 0c54781 | 2011-08-25 17:55:54 +0000 | [diff] [blame] | 91 | /* Don't use MTRRs here; the Xserver or userspace app should |
| 92 | * deal with them for Intel hardware. |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 93 | */ |
| 94 | .driver_features = |
| 95 | DRIVER_USE_AGP | DRIVER_REQUIRE_AGP | |
| 96 | DRIVER_HAVE_IRQ | DRIVER_IRQ_SHARED | DRIVER_MODESET, |
| 97 | .load = i915_driver_load, |
| 98 | .unload = i915_driver_unload, |
| 99 | .firstopen = i915_driver_firstopen, |
| 100 | .lastclose = i915_driver_lastclose, |
| 101 | .preclose = i915_driver_preclose, |
| 102 | .save = i915_save, |
| 103 | .restore = i915_restore, |
| 104 | .device_is_agp = i915_driver_device_is_agp, |
| 105 | .get_vblank_counter = i915_get_vblank_counter, |
| 106 | .enable_vblank = i915_enable_vblank, |
| 107 | .disable_vblank = i915_disable_vblank, |
| 108 | .irq_preinstall = i915_driver_irq_preinstall, |
| 109 | .irq_postinstall = i915_driver_irq_postinstall, |
| 110 | .irq_uninstall = i915_driver_irq_uninstall, |
| 111 | .irq_handler = i915_driver_irq_handler, |
| 112 | .reclaim_buffers = drm_core_reclaim_buffers, |
| 113 | .get_map_ofs = drm_core_get_map_ofs, |
| 114 | .get_reg_ofs = drm_core_get_reg_ofs, |
| 115 | .fb_probe = intelfb_probe, |
| 116 | .fb_remove = intelfb_remove, |
| 117 | .fb_resize = intelfb_resize, |
| 118 | .master_create = i915_master_create, |
| 119 | .master_destroy = i915_master_destroy, |
| 120 | #if defined(CONFIG_DEBUG_FS) |
| 121 | .debugfs_init = i915_debugfs_init, |
| 122 | .debugfs_cleanup = i915_debugfs_cleanup, |
| 123 | #endif |
| 124 | .gem_init_object = i915_gem_init_object, |
| 125 | .gem_free_object = i915_gem_free_object, |
| 126 | .gem_vm_ops = &i915_gem_vm_ops, |
| 127 | .ioctls = i915_ioctls, |
| 128 | .fops = { |
| 129 | .owner = THIS_MODULE, |
| 130 | .open = drm_open, |
| 131 | .release = drm_release, |
| 132 | .ioctl = drm_ioctl, |
| 133 | .mmap = drm_mmap, |
| 134 | .poll = drm_poll, |
| 135 | .fasync = drm_fasync, |
| 136 | #ifdef CONFIG_COMPAT |
| 137 | .compat_ioctl = i915_compat_ioctl, |
| 138 | #endif |
Arnd Bergmann | dc880ab | 2010-07-06 18:54:47 +0200 | [diff] [blame] | 139 | .llseek = noop_llseek, |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 140 | }, |
| 141 | .pci_driver = { |
| 142 | .name = DRIVER_NAME, |
| 143 | .id_table = pciidlist, |
| 144 | .probe = probe, |
| 145 | .remove = __devexit_p(drm_cleanup_pci), |
| 146 | }, |
| 147 | .name = DRIVER_NAME, |
| 148 | .desc = DRIVER_DESC, |
| 149 | .date = DRIVER_DATE, |
| 150 | .major = DRIVER_MAJOR, |
| 151 | .minor = DRIVER_MINOR, |
| 152 | .patchlevel = DRIVER_PATCHLEVEL, |
| 153 | }; |
| 154 | </programlisting> |
| 155 | <para> |
| 156 | In the example above, taken from the i915 DRM driver, the driver |
Michael Witten | 2c267e9 | 2011-08-25 18:10:12 +0000 | [diff] [blame] | 157 | sets several flags indicating what core features it supports; |
| 158 | we go over the individual callbacks in later sections. Since |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 159 | flags indicate which features your driver supports to the DRM |
| 160 | core, you need to set most of them prior to calling drm_init(). Some, |
| 161 | like DRIVER_MODESET can be set later based on user supplied parameters, |
| 162 | but that's the exception rather than the rule. |
| 163 | </para> |
| 164 | <variablelist> |
| 165 | <title>Driver flags</title> |
| 166 | <varlistentry> |
| 167 | <term>DRIVER_USE_AGP</term> |
| 168 | <listitem><para> |
| 169 | Driver uses AGP interface |
| 170 | </para></listitem> |
| 171 | </varlistentry> |
| 172 | <varlistentry> |
| 173 | <term>DRIVER_REQUIRE_AGP</term> |
| 174 | <listitem><para> |
| 175 | Driver needs AGP interface to function. |
| 176 | </para></listitem> |
| 177 | </varlistentry> |
| 178 | <varlistentry> |
| 179 | <term>DRIVER_USE_MTRR</term> |
| 180 | <listitem> |
| 181 | <para> |
| 182 | Driver uses MTRR interface for mapping memory. Deprecated. |
| 183 | </para> |
| 184 | </listitem> |
| 185 | </varlistentry> |
| 186 | <varlistentry> |
| 187 | <term>DRIVER_PCI_DMA</term> |
| 188 | <listitem><para> |
| 189 | Driver is capable of PCI DMA. Deprecated. |
| 190 | </para></listitem> |
| 191 | </varlistentry> |
| 192 | <varlistentry> |
| 193 | <term>DRIVER_SG</term> |
| 194 | <listitem><para> |
| 195 | Driver can perform scatter/gather DMA. Deprecated. |
| 196 | </para></listitem> |
| 197 | </varlistentry> |
| 198 | <varlistentry> |
| 199 | <term>DRIVER_HAVE_DMA</term> |
| 200 | <listitem><para>Driver supports DMA. Deprecated.</para></listitem> |
| 201 | </varlistentry> |
| 202 | <varlistentry> |
| 203 | <term>DRIVER_HAVE_IRQ</term><term>DRIVER_IRQ_SHARED</term> |
| 204 | <listitem> |
| 205 | <para> |
Michael Witten | 02391f1 | 2011-08-25 18:20:54 +0000 | [diff] [blame] | 206 | DRIVER_HAVE_IRQ indicates whether the driver has an IRQ |
Michael Witten | b1f95bd | 2011-08-25 18:21:29 +0000 | [diff] [blame] | 207 | handler. DRIVER_IRQ_SHARED indicates whether the device & |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 208 | handler support shared IRQs (note that this is required of |
| 209 | PCI drivers). |
| 210 | </para> |
| 211 | </listitem> |
| 212 | </varlistentry> |
| 213 | <varlistentry> |
| 214 | <term>DRIVER_DMA_QUEUE</term> |
| 215 | <listitem> |
| 216 | <para> |
Michael Witten | 80c84e6 | 2011-08-25 18:23:21 +0000 | [diff] [blame] | 217 | Should be set if the driver queues DMA requests and completes them |
| 218 | asynchronously. Deprecated. |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 219 | </para> |
| 220 | </listitem> |
| 221 | </varlistentry> |
| 222 | <varlistentry> |
| 223 | <term>DRIVER_FB_DMA</term> |
| 224 | <listitem> |
| 225 | <para> |
| 226 | Driver supports DMA to/from the framebuffer. Deprecated. |
| 227 | </para> |
| 228 | </listitem> |
| 229 | </varlistentry> |
| 230 | <varlistentry> |
| 231 | <term>DRIVER_MODESET</term> |
| 232 | <listitem> |
| 233 | <para> |
| 234 | Driver supports mode setting interfaces. |
| 235 | </para> |
| 236 | </listitem> |
| 237 | </varlistentry> |
| 238 | </variablelist> |
| 239 | <para> |
| 240 | In this specific case, the driver requires AGP and supports |
Michael Witten | a78f678 | 2011-08-25 17:18:08 +0000 | [diff] [blame] | 241 | IRQs. DMA, as discussed later, is handled by device specific ioctls |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 242 | in this case. It also supports the kernel mode setting APIs, though |
| 243 | unlike in the actual i915 driver source, this example unconditionally |
| 244 | exports KMS capability. |
| 245 | </para> |
| 246 | </sect1> |
| 247 | |
| 248 | <!-- Internals: driver load --> |
| 249 | |
| 250 | <sect1> |
| 251 | <title>Driver load</title> |
| 252 | <para> |
| 253 | In the previous section, we saw what a typical drm_driver |
| 254 | structure might look like. One of the more important fields in |
| 255 | the structure is the hook for the load function. |
| 256 | </para> |
| 257 | <programlisting> |
| 258 | static struct drm_driver driver = { |
| 259 | ... |
| 260 | .load = i915_driver_load, |
| 261 | ... |
| 262 | }; |
| 263 | </programlisting> |
| 264 | <para> |
| 265 | The load function has many responsibilities: allocating a driver |
| 266 | private structure, specifying supported performance counters, |
| 267 | configuring the device (e.g. mapping registers & command |
| 268 | buffers), initializing the memory manager, and setting up the |
| 269 | initial output configuration. |
| 270 | </para> |
| 271 | <para> |
Michael Witten | 6e375f4 | 2011-08-25 18:37:05 +0000 | [diff] [blame] | 272 | If compatibility is a concern (e.g. with drivers converted over |
| 273 | to the new interfaces from the old ones), care must be taken to |
| 274 | prevent device initialization and control that is incompatible with |
| 275 | currently active userspace drivers. For instance, if user |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 276 | level mode setting drivers are in use, it would be problematic |
| 277 | to perform output discovery & configuration at load time. |
Michael Witten | 58f1d65 | 2011-08-25 18:31:42 +0000 | [diff] [blame] | 278 | Likewise, if user-level drivers unaware of memory management are |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 279 | in use, memory management and command buffer setup may need to |
| 280 | be omitted. These requirements are driver specific, and care |
| 281 | needs to be taken to keep both old and new applications and |
| 282 | libraries working. The i915 driver supports the "modeset" |
| 283 | module parameter to control whether advanced features are |
Michael Witten | 6e375f4 | 2011-08-25 18:37:05 +0000 | [diff] [blame] | 284 | enabled at load time or in legacy fashion. |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 285 | </para> |
| 286 | |
| 287 | <sect2> |
| 288 | <title>Driver private & performance counters</title> |
| 289 | <para> |
| 290 | The driver private hangs off the main drm_device structure and |
| 291 | can be used for tracking various device specific bits of |
| 292 | information, like register offsets, command buffer status, |
| 293 | register state for suspend/resume, etc. At load time, a |
Michael Witten | f11aca0 | 2011-08-25 17:21:31 +0000 | [diff] [blame] | 294 | driver may simply allocate one and set drm_device.dev_priv |
Michael Witten | 06fa7b8 | 2011-08-25 18:40:55 +0000 | [diff] [blame^] | 295 | appropriately; it should be freed and drm_device.dev_priv set |
| 296 | to NULL when the driver is unloaded. |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 297 | </para> |
| 298 | <para> |
Michael Witten | f11aca0 | 2011-08-25 17:21:31 +0000 | [diff] [blame] | 299 | The DRM supports several counters which may be used for rough |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 300 | performance characterization. Note that the DRM stat counter |
| 301 | system is not often used by applications, and supporting |
| 302 | additional counters is completely optional. |
| 303 | </para> |
| 304 | <para> |
| 305 | These interfaces are deprecated and should not be used. If performance |
| 306 | monitoring is desired, the developer should investigate and |
| 307 | potentially enhance the kernel perf and tracing infrastructure to export |
| 308 | GPU related performance information to performance monitoring |
| 309 | tools and applications. |
| 310 | </para> |
| 311 | </sect2> |
| 312 | |
| 313 | <sect2> |
| 314 | <title>Configuring the device</title> |
| 315 | <para> |
Michael Witten | a78f678 | 2011-08-25 17:18:08 +0000 | [diff] [blame] | 316 | Obviously, device configuration is device specific. |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 317 | However, there are several common operations: finding a |
| 318 | device's PCI resources, mapping them, and potentially setting |
| 319 | up an IRQ handler. |
| 320 | </para> |
| 321 | <para> |
| 322 | Finding & mapping resources is fairly straightforward. The |
| 323 | DRM wrapper functions, drm_get_resource_start() and |
Michael Witten | f11aca0 | 2011-08-25 17:21:31 +0000 | [diff] [blame] | 324 | drm_get_resource_len() may be used to find BARs on the given |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 325 | drm_device struct. Once those values have been retrieved, the |
| 326 | driver load function can call drm_addmap() to create a new |
Michael Witten | a78f678 | 2011-08-25 17:18:08 +0000 | [diff] [blame] | 327 | mapping for the BAR in question. Note you probably want a |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 328 | drm_local_map_t in your driver private structure to track any |
| 329 | mappings you create. |
| 330 | <!-- !Fdrivers/gpu/drm/drm_bufs.c drm_get_resource_* --> |
| 331 | <!-- !Finclude/drm/drmP.h drm_local_map_t --> |
| 332 | </para> |
| 333 | <para> |
| 334 | if compatibility with other operating systems isn't a concern |
| 335 | (DRM drivers can run under various BSD variants and OpenSolaris), |
Michael Witten | f11aca0 | 2011-08-25 17:21:31 +0000 | [diff] [blame] | 336 | native Linux calls may be used for the above, e.g. pci_resource_* |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 337 | and iomap*/iounmap. See the Linux device driver book for more |
| 338 | info. |
| 339 | </para> |
| 340 | <para> |
Michael Witten | f11aca0 | 2011-08-25 17:21:31 +0000 | [diff] [blame] | 341 | Once you have a register map, you may use the DRM_READn() and |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 342 | DRM_WRITEn() macros to access the registers on your device, or |
| 343 | use driver specific versions to offset into your MMIO space |
| 344 | relative to a driver specific base pointer (see I915_READ for |
| 345 | example). |
| 346 | </para> |
| 347 | <para> |
| 348 | If your device supports interrupt generation, you may want to |
| 349 | setup an interrupt handler at driver load time as well. This |
| 350 | is done using the drm_irq_install() function. If your device |
| 351 | supports vertical blank interrupts, it should call |
| 352 | drm_vblank_init() to initialize the core vblank handling code before |
| 353 | enabling interrupts on your device. This ensures the vblank related |
| 354 | structures are allocated and allows the core to handle vblank events. |
| 355 | </para> |
| 356 | <!--!Fdrivers/char/drm/drm_irq.c drm_irq_install--> |
| 357 | <para> |
Michael Witten | a78f678 | 2011-08-25 17:18:08 +0000 | [diff] [blame] | 358 | Once your interrupt handler is registered (it uses your |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 359 | drm_driver.irq_handler as the actual interrupt handling |
| 360 | function), you can safely enable interrupts on your device, |
| 361 | assuming any other state your interrupt handler uses is also |
| 362 | initialized. |
| 363 | </para> |
| 364 | <para> |
| 365 | Another task that may be necessary during configuration is |
| 366 | mapping the video BIOS. On many devices, the VBIOS describes |
| 367 | device configuration, LCD panel timings (if any), and contains |
| 368 | flags indicating device state. Mapping the BIOS can be done |
| 369 | using the pci_map_rom() call, a convenience function that |
| 370 | takes care of mapping the actual ROM, whether it has been |
| 371 | shadowed into memory (typically at address 0xc0000) or exists |
| 372 | on the PCI device in the ROM BAR. Note that once you've |
| 373 | mapped the ROM and extracted any necessary information, be |
| 374 | sure to unmap it; on many devices the ROM address decoder is |
| 375 | shared with other BARs, so leaving it mapped can cause |
| 376 | undesired behavior like hangs or memory corruption. |
| 377 | <!--!Fdrivers/pci/rom.c pci_map_rom--> |
| 378 | </para> |
| 379 | </sect2> |
| 380 | |
| 381 | <sect2> |
| 382 | <title>Memory manager initialization</title> |
| 383 | <para> |
| 384 | In order to allocate command buffers, cursor memory, scanout |
| 385 | buffers, etc., as well as support the latest features provided |
| 386 | by packages like Mesa and the X.Org X server, your driver |
| 387 | should support a memory manager. |
| 388 | </para> |
| 389 | <para> |
Michael Witten | a78f678 | 2011-08-25 17:18:08 +0000 | [diff] [blame] | 390 | If your driver supports memory management (it should!), you |
Nicolas Kaiser | ce04cc0 | 2010-05-28 07:33:49 +0200 | [diff] [blame] | 391 | need to set that up at load time as well. How you initialize |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 392 | it depends on which memory manager you're using, TTM or GEM. |
| 393 | </para> |
| 394 | <sect3> |
| 395 | <title>TTM initialization</title> |
| 396 | <para> |
| 397 | TTM (for Translation Table Manager) manages video memory and |
| 398 | aperture space for graphics devices. TTM supports both UMA devices |
| 399 | and devices with dedicated video RAM (VRAM), i.e. most discrete |
| 400 | graphics devices. If your device has dedicated RAM, supporting |
Nicolas Kaiser | ce04cc0 | 2010-05-28 07:33:49 +0200 | [diff] [blame] | 401 | TTM is desirable. TTM also integrates tightly with your |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 402 | driver specific buffer execution function. See the radeon |
| 403 | driver for examples. |
| 404 | </para> |
| 405 | <para> |
| 406 | The core TTM structure is the ttm_bo_driver struct. It contains |
| 407 | several fields with function pointers for initializing the TTM, |
| 408 | allocating and freeing memory, waiting for command completion |
| 409 | and fence synchronization, and memory migration. See the |
| 410 | radeon_ttm.c file for an example of usage. |
| 411 | </para> |
| 412 | <para> |
| 413 | The ttm_global_reference structure is made up of several fields: |
| 414 | </para> |
| 415 | <programlisting> |
| 416 | struct ttm_global_reference { |
| 417 | enum ttm_global_types global_type; |
| 418 | size_t size; |
| 419 | void *object; |
| 420 | int (*init) (struct ttm_global_reference *); |
| 421 | void (*release) (struct ttm_global_reference *); |
| 422 | }; |
| 423 | </programlisting> |
| 424 | <para> |
| 425 | There should be one global reference structure for your memory |
| 426 | manager as a whole, and there will be others for each object |
| 427 | created by the memory manager at runtime. Your global TTM should |
| 428 | have a type of TTM_GLOBAL_TTM_MEM. The size field for the global |
| 429 | object should be sizeof(struct ttm_mem_global), and the init and |
| 430 | release hooks should point at your driver specific init and |
Michael Witten | a78f678 | 2011-08-25 17:18:08 +0000 | [diff] [blame] | 431 | release routines, which probably eventually call |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 432 | ttm_mem_global_init and ttm_mem_global_release respectively. |
| 433 | </para> |
| 434 | <para> |
| 435 | Once your global TTM accounting structure is set up and initialized |
| 436 | (done by calling ttm_global_item_ref on the global object you |
Michael Witten | a78f678 | 2011-08-25 17:18:08 +0000 | [diff] [blame] | 437 | just created), you need to create a buffer object TTM to |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 438 | provide a pool for buffer object allocation by clients and the |
| 439 | kernel itself. The type of this object should be TTM_GLOBAL_TTM_BO, |
| 440 | and its size should be sizeof(struct ttm_bo_global). Again, |
Michael Witten | f11aca0 | 2011-08-25 17:21:31 +0000 | [diff] [blame] | 441 | driver specific init and release functions may be provided, |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 442 | likely eventually calling ttm_bo_global_init and |
| 443 | ttm_bo_global_release, respectively. Also like the previous |
| 444 | object, ttm_global_item_ref is used to create an initial reference |
Nicolas Kaiser | ce04cc0 | 2010-05-28 07:33:49 +0200 | [diff] [blame] | 445 | count for the TTM, which will call your initialization function. |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 446 | </para> |
| 447 | </sect3> |
| 448 | <sect3> |
| 449 | <title>GEM initialization</title> |
| 450 | <para> |
| 451 | GEM is an alternative to TTM, designed specifically for UMA |
| 452 | devices. It has simpler initialization and execution requirements |
| 453 | than TTM, but has no VRAM management capability. Core GEM |
| 454 | initialization is comprised of a basic drm_mm_init call to create |
| 455 | a GTT DRM MM object, which provides an address space pool for |
Michael Witten | a78f678 | 2011-08-25 17:18:08 +0000 | [diff] [blame] | 456 | object allocation. In a KMS configuration, the driver |
| 457 | needs to allocate and initialize a command ring buffer following |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 458 | basic GEM initialization. Most UMA devices have a so-called |
| 459 | "stolen" memory region, which provides space for the initial |
| 460 | framebuffer and large, contiguous memory regions required by the |
| 461 | device. This space is not typically managed by GEM, and must |
| 462 | be initialized separately into its own DRM MM object. |
| 463 | </para> |
| 464 | <para> |
Michael Witten | a78f678 | 2011-08-25 17:18:08 +0000 | [diff] [blame] | 465 | Initialization is driver specific, and depends on |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 466 | the architecture of the device. In the case of Intel |
| 467 | integrated graphics chips like 965GM, GEM initialization can |
| 468 | be done by calling the internal GEM init function, |
| 469 | i915_gem_do_init(). Since the 965GM is a UMA device |
Michael Witten | a78f678 | 2011-08-25 17:18:08 +0000 | [diff] [blame] | 470 | (i.e. it doesn't have dedicated VRAM), GEM manages |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 471 | making regular RAM available for GPU operations. Memory set |
| 472 | aside by the BIOS (called "stolen" memory by the i915 |
Michael Witten | a78f678 | 2011-08-25 17:18:08 +0000 | [diff] [blame] | 473 | driver) is managed by the DRM memrange allocator; the |
| 474 | rest of the aperture is managed by GEM. |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 475 | <programlisting> |
| 476 | /* Basic memrange allocator for stolen space (aka vram) */ |
| 477 | drm_memrange_init(&dev_priv->vram, 0, prealloc_size); |
| 478 | /* Let GEM Manage from end of prealloc space to end of aperture */ |
| 479 | i915_gem_do_init(dev, prealloc_size, agp_size); |
| 480 | </programlisting> |
| 481 | <!--!Edrivers/char/drm/drm_memrange.c--> |
| 482 | </para> |
| 483 | <para> |
Michael Witten | f11aca0 | 2011-08-25 17:21:31 +0000 | [diff] [blame] | 484 | Once the memory manager has been set up, we may allocate the |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 485 | command buffer. In the i915 case, this is also done with a |
| 486 | GEM function, i915_gem_init_ringbuffer(). |
| 487 | </para> |
| 488 | </sect3> |
| 489 | </sect2> |
| 490 | |
| 491 | <sect2> |
| 492 | <title>Output configuration</title> |
| 493 | <para> |
| 494 | The final initialization task is output configuration. This involves |
| 495 | finding and initializing the CRTCs, encoders and connectors |
| 496 | for your device, creating an initial configuration and |
| 497 | registering a framebuffer console driver. |
| 498 | </para> |
| 499 | <sect3> |
| 500 | <title>Output discovery and initialization</title> |
| 501 | <para> |
| 502 | Several core functions exist to create CRTCs, encoders and |
| 503 | connectors, namely drm_crtc_init(), drm_connector_init() and |
| 504 | drm_encoder_init(), along with several "helper" functions to |
| 505 | perform common tasks. |
| 506 | </para> |
| 507 | <para> |
| 508 | Connectors should be registered with sysfs once they've been |
| 509 | detected and initialized, using the |
| 510 | drm_sysfs_connector_add() function. Likewise, when they're |
| 511 | removed from the system, they should be destroyed with |
| 512 | drm_sysfs_connector_remove(). |
| 513 | </para> |
| 514 | <programlisting> |
| 515 | <![CDATA[ |
| 516 | void intel_crt_init(struct drm_device *dev) |
| 517 | { |
| 518 | struct drm_connector *connector; |
| 519 | struct intel_output *intel_output; |
| 520 | |
| 521 | intel_output = kzalloc(sizeof(struct intel_output), GFP_KERNEL); |
| 522 | if (!intel_output) |
| 523 | return; |
| 524 | |
| 525 | connector = &intel_output->base; |
| 526 | drm_connector_init(dev, &intel_output->base, |
| 527 | &intel_crt_connector_funcs, DRM_MODE_CONNECTOR_VGA); |
| 528 | |
| 529 | drm_encoder_init(dev, &intel_output->enc, &intel_crt_enc_funcs, |
| 530 | DRM_MODE_ENCODER_DAC); |
| 531 | |
| 532 | drm_mode_connector_attach_encoder(&intel_output->base, |
| 533 | &intel_output->enc); |
| 534 | |
| 535 | /* Set up the DDC bus. */ |
| 536 | intel_output->ddc_bus = intel_i2c_create(dev, GPIOA, "CRTDDC_A"); |
| 537 | if (!intel_output->ddc_bus) { |
| 538 | dev_printk(KERN_ERR, &dev->pdev->dev, "DDC bus registration " |
| 539 | "failed.\n"); |
| 540 | return; |
| 541 | } |
| 542 | |
| 543 | intel_output->type = INTEL_OUTPUT_ANALOG; |
| 544 | connector->interlace_allowed = 0; |
| 545 | connector->doublescan_allowed = 0; |
| 546 | |
| 547 | drm_encoder_helper_add(&intel_output->enc, &intel_crt_helper_funcs); |
| 548 | drm_connector_helper_add(connector, &intel_crt_connector_helper_funcs); |
| 549 | |
| 550 | drm_sysfs_connector_add(connector); |
| 551 | } |
| 552 | ]]> |
| 553 | </programlisting> |
| 554 | <para> |
| 555 | In the example above (again, taken from the i915 driver), a |
| 556 | CRT connector and encoder combination is created. A device |
| 557 | specific i2c bus is also created, for fetching EDID data and |
| 558 | performing monitor detection. Once the process is complete, |
Nicolas Kaiser | ce04cc0 | 2010-05-28 07:33:49 +0200 | [diff] [blame] | 559 | the new connector is registered with sysfs, to make its |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 560 | properties available to applications. |
| 561 | </para> |
| 562 | <sect4> |
| 563 | <title>Helper functions and core functions</title> |
| 564 | <para> |
| 565 | Since many PC-class graphics devices have similar display output |
| 566 | designs, the DRM provides a set of helper functions to make |
| 567 | output management easier. The core helper routines handle |
| 568 | encoder re-routing and disabling of unused functions following |
| 569 | mode set. Using the helpers is optional, but recommended for |
| 570 | devices with PC-style architectures (i.e. a set of display planes |
| 571 | for feeding pixels to encoders which are in turn routed to |
| 572 | connectors). Devices with more complex requirements needing |
Michael Witten | f11aca0 | 2011-08-25 17:21:31 +0000 | [diff] [blame] | 573 | finer grained management may opt to use the core callbacks |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 574 | directly. |
| 575 | </para> |
| 576 | <para> |
| 577 | [Insert typical diagram here.] [Insert OMAP style config here.] |
| 578 | </para> |
| 579 | </sect4> |
| 580 | <para> |
| 581 | For each encoder, CRTC and connector, several functions must |
| 582 | be provided, depending on the object type. Encoder objects |
Nicolas Kaiser | ce04cc0 | 2010-05-28 07:33:49 +0200 | [diff] [blame] | 583 | need to provide a DPMS (basically on/off) function, mode fixup |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 584 | (for converting requested modes into native hardware timings), |
| 585 | and prepare, set and commit functions for use by the core DRM |
| 586 | helper functions. Connector helpers need to provide mode fetch and |
| 587 | validity functions as well as an encoder matching function for |
Nicolas Kaiser | ce04cc0 | 2010-05-28 07:33:49 +0200 | [diff] [blame] | 588 | returning an ideal encoder for a given connector. The core |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 589 | connector functions include a DPMS callback, (deprecated) |
| 590 | save/restore routines, detection, mode probing, property handling, |
| 591 | and cleanup functions. |
| 592 | </para> |
| 593 | <!--!Edrivers/char/drm/drm_crtc.h--> |
| 594 | <!--!Edrivers/char/drm/drm_crtc.c--> |
| 595 | <!--!Edrivers/char/drm/drm_crtc_helper.c--> |
| 596 | </sect3> |
| 597 | </sect2> |
| 598 | </sect1> |
| 599 | |
| 600 | <!-- Internals: vblank handling --> |
| 601 | |
| 602 | <sect1> |
| 603 | <title>VBlank event handling</title> |
| 604 | <para> |
| 605 | The DRM core exposes two vertical blank related ioctls: |
| 606 | DRM_IOCTL_WAIT_VBLANK and DRM_IOCTL_MODESET_CTL. |
| 607 | <!--!Edrivers/char/drm/drm_irq.c--> |
| 608 | </para> |
| 609 | <para> |
| 610 | DRM_IOCTL_WAIT_VBLANK takes a struct drm_wait_vblank structure |
| 611 | as its argument, and is used to block or request a signal when a |
| 612 | specified vblank event occurs. |
| 613 | </para> |
| 614 | <para> |
| 615 | DRM_IOCTL_MODESET_CTL should be called by application level |
| 616 | drivers before and after mode setting, since on many devices the |
Michael Witten | a78f678 | 2011-08-25 17:18:08 +0000 | [diff] [blame] | 617 | vertical blank counter is reset at that time. Internally, |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 618 | the DRM snapshots the last vblank count when the ioctl is called |
| 619 | with the _DRM_PRE_MODESET command so that the counter won't go |
| 620 | backwards (which is dealt with when _DRM_POST_MODESET is used). |
| 621 | </para> |
| 622 | <para> |
| 623 | To support the functions above, the DRM core provides several |
| 624 | helper functions for tracking vertical blank counters, and |
| 625 | requires drivers to provide several callbacks: |
| 626 | get_vblank_counter(), enable_vblank() and disable_vblank(). The |
| 627 | core uses get_vblank_counter() to keep the counter accurate |
| 628 | across interrupt disable periods. It should return the current |
| 629 | vertical blank event count, which is often tracked in a device |
| 630 | register. The enable and disable vblank callbacks should enable |
| 631 | and disable vertical blank interrupts, respectively. In the |
| 632 | absence of DRM clients waiting on vblank events, the core DRM |
Michael Witten | a78f678 | 2011-08-25 17:18:08 +0000 | [diff] [blame] | 633 | code uses the disable_vblank() function to disable |
| 634 | interrupts, which saves power. They are re-enabled again when |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 635 | a client calls the vblank wait ioctl above. |
| 636 | </para> |
| 637 | <para> |
Michael Witten | f11aca0 | 2011-08-25 17:21:31 +0000 | [diff] [blame] | 638 | Devices that don't provide a count register may simply use an |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 639 | internal atomic counter incremented on every vertical blank |
| 640 | interrupt, and can make their enable and disable vblank |
| 641 | functions into no-ops. |
| 642 | </para> |
| 643 | </sect1> |
| 644 | |
| 645 | <sect1> |
| 646 | <title>Memory management</title> |
| 647 | <para> |
Michael Witten | 2c267e9 | 2011-08-25 18:10:12 +0000 | [diff] [blame] | 648 | The memory manager lies at the heart of many DRM operations; it |
| 649 | is required to support advanced client features like OpenGL |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 650 | pbuffers. The DRM currently contains two memory managers, TTM |
| 651 | and GEM. |
| 652 | </para> |
| 653 | |
| 654 | <sect2> |
| 655 | <title>The Translation Table Manager (TTM)</title> |
| 656 | <para> |
| 657 | TTM was developed by Tungsten Graphics, primarily by Thomas |
| 658 | Hellström, and is intended to be a flexible, high performance |
| 659 | graphics memory manager. |
| 660 | </para> |
| 661 | <para> |
| 662 | Drivers wishing to support TTM must fill out a drm_bo_driver |
| 663 | structure. |
| 664 | </para> |
| 665 | <para> |
| 666 | TTM design background and information belongs here. |
| 667 | </para> |
| 668 | </sect2> |
| 669 | |
| 670 | <sect2> |
| 671 | <title>The Graphics Execution Manager (GEM)</title> |
| 672 | <para> |
| 673 | GEM is an Intel project, authored by Eric Anholt and Keith |
| 674 | Packard. It provides simpler interfaces than TTM, and is well |
| 675 | suited for UMA devices. |
| 676 | </para> |
| 677 | <para> |
| 678 | GEM-enabled drivers must provide gem_init_object() and |
| 679 | gem_free_object() callbacks to support the core memory |
| 680 | allocation routines. They should also provide several driver |
| 681 | specific ioctls to support command execution, pinning, buffer |
| 682 | read & write, mapping, and domain ownership transfers. |
| 683 | </para> |
| 684 | <para> |
| 685 | On a fundamental level, GEM involves several operations: memory |
| 686 | allocation and freeing, command execution, and aperture management |
| 687 | at command execution time. Buffer object allocation is relatively |
| 688 | straightforward and largely provided by Linux's shmem layer, which |
| 689 | provides memory to back each object. When mapped into the GTT |
| 690 | or used in a command buffer, the backing pages for an object are |
| 691 | flushed to memory and marked write combined so as to be coherent |
| 692 | with the GPU. Likewise, when the GPU finishes rendering to an object, |
| 693 | if the CPU accesses it, it must be made coherent with the CPU's view |
| 694 | of memory, usually involving GPU cache flushing of various kinds. |
| 695 | This core CPU<->GPU coherency management is provided by the GEM |
| 696 | set domain function, which evaluates an object's current domain and |
| 697 | performs any necessary flushing or synchronization to put the object |
| 698 | into the desired coherency domain (note that the object may be busy, |
| 699 | i.e. an active render target; in that case the set domain function |
Michael Witten | a78f678 | 2011-08-25 17:18:08 +0000 | [diff] [blame] | 700 | blocks the client and waits for rendering to complete before |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 701 | performing any necessary flushing operations). |
| 702 | </para> |
| 703 | <para> |
| 704 | Perhaps the most important GEM function is providing a command |
| 705 | execution interface to clients. Client programs construct command |
| 706 | buffers containing references to previously allocated memory objects |
Michael Witten | a78f678 | 2011-08-25 17:18:08 +0000 | [diff] [blame] | 707 | and submit them to GEM. At that point, GEM takes care to bind |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 708 | all the objects into the GTT, execute the buffer, and provide |
| 709 | necessary synchronization between clients accessing the same buffers. |
| 710 | This often involves evicting some objects from the GTT and re-binding |
| 711 | others (a fairly expensive operation), and providing relocation |
| 712 | support which hides fixed GTT offsets from clients. Clients must |
| 713 | take care not to submit command buffers that reference more objects |
| 714 | than can fit in the GTT or GEM will reject them and no rendering |
| 715 | will occur. Similarly, if several objects in the buffer require |
| 716 | fence registers to be allocated for correct rendering (e.g. 2D blits |
| 717 | on pre-965 chips), care must be taken not to require more fence |
| 718 | registers than are available to the client. Such resource management |
| 719 | should be abstracted from the client in libdrm. |
| 720 | </para> |
| 721 | </sect2> |
| 722 | |
| 723 | </sect1> |
| 724 | |
| 725 | <!-- Output management --> |
| 726 | <sect1> |
| 727 | <title>Output management</title> |
| 728 | <para> |
| 729 | At the core of the DRM output management code is a set of |
| 730 | structures representing CRTCs, encoders and connectors. |
| 731 | </para> |
| 732 | <para> |
| 733 | A CRTC is an abstraction representing a part of the chip that |
| 734 | contains a pointer to a scanout buffer. Therefore, the number |
| 735 | of CRTCs available determines how many independent scanout |
| 736 | buffers can be active at any given time. The CRTC structure |
| 737 | contains several fields to support this: a pointer to some video |
| 738 | memory, a display mode, and an (x, y) offset into the video |
| 739 | memory to support panning or configurations where one piece of |
| 740 | video memory spans multiple CRTCs. |
| 741 | </para> |
| 742 | <para> |
| 743 | An encoder takes pixel data from a CRTC and converts it to a |
| 744 | format suitable for any attached connectors. On some devices, |
| 745 | it may be possible to have a CRTC send data to more than one |
| 746 | encoder. In that case, both encoders would receive data from |
| 747 | the same scanout buffer, resulting in a "cloned" display |
| 748 | configuration across the connectors attached to each encoder. |
| 749 | </para> |
| 750 | <para> |
| 751 | A connector is the final destination for pixel data on a device, |
| 752 | and usually connects directly to an external display device like |
| 753 | a monitor or laptop panel. A connector can only be attached to |
| 754 | one encoder at a time. The connector is also the structure |
| 755 | where information about the attached display is kept, so it |
| 756 | contains fields for display data, EDID data, DPMS & |
| 757 | connection status, and information about modes supported on the |
| 758 | attached displays. |
| 759 | </para> |
| 760 | <!--!Edrivers/char/drm/drm_crtc.c--> |
| 761 | </sect1> |
| 762 | |
| 763 | <sect1> |
| 764 | <title>Framebuffer management</title> |
| 765 | <para> |
| 766 | In order to set a mode on a given CRTC, encoder and connector |
| 767 | configuration, clients need to provide a framebuffer object which |
Michael Witten | a78f678 | 2011-08-25 17:18:08 +0000 | [diff] [blame] | 768 | provides a source of pixels for the CRTC to deliver to the encoder(s) |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 769 | and ultimately the connector(s) in the configuration. A framebuffer |
| 770 | is fundamentally a driver specific memory object, made into an opaque |
| 771 | handle by the DRM addfb function. Once an fb has been created this |
| 772 | way it can be passed to the KMS mode setting routines for use in |
| 773 | a configuration. |
| 774 | </para> |
| 775 | </sect1> |
| 776 | |
| 777 | <sect1> |
| 778 | <title>Command submission & fencing</title> |
| 779 | <para> |
| 780 | This should cover a few device specific command submission |
| 781 | implementations. |
| 782 | </para> |
| 783 | </sect1> |
| 784 | |
| 785 | <sect1> |
| 786 | <title>Suspend/resume</title> |
| 787 | <para> |
| 788 | The DRM core provides some suspend/resume code, but drivers |
| 789 | wanting full suspend/resume support should provide save() and |
Michael Witten | a78f678 | 2011-08-25 17:18:08 +0000 | [diff] [blame] | 790 | restore() functions. These are called at suspend, |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 791 | hibernate, or resume time, and should perform any state save or |
| 792 | restore required by your device across suspend or hibernate |
| 793 | states. |
| 794 | </para> |
| 795 | </sect1> |
| 796 | |
| 797 | <sect1> |
| 798 | <title>DMA services</title> |
| 799 | <para> |
| 800 | This should cover how DMA mapping etc. is supported by the core. |
| 801 | These functions are deprecated and should not be used. |
| 802 | </para> |
| 803 | </sect1> |
| 804 | </chapter> |
| 805 | |
| 806 | <!-- External interfaces --> |
| 807 | |
| 808 | <chapter id="drmExternals"> |
| 809 | <title>Userland interfaces</title> |
| 810 | <para> |
| 811 | The DRM core exports several interfaces to applications, |
| 812 | generally intended to be used through corresponding libdrm |
| 813 | wrapper functions. In addition, drivers export device specific |
| 814 | interfaces for use by userspace drivers & device aware |
| 815 | applications through ioctls and sysfs files. |
| 816 | </para> |
| 817 | <para> |
| 818 | External interfaces include: memory mapping, context management, |
| 819 | DMA operations, AGP management, vblank control, fence |
| 820 | management, memory management, and output management. |
| 821 | </para> |
| 822 | <para> |
| 823 | Cover generic ioctls and sysfs layout here. Only need high |
Michael Witten | a78f678 | 2011-08-25 17:18:08 +0000 | [diff] [blame] | 824 | level info, since man pages should cover the rest. |
Jesse Barnes | 2d2ef82 | 2009-10-26 13:06:31 -0700 | [diff] [blame] | 825 | </para> |
| 826 | </chapter> |
| 827 | |
| 828 | <!-- API reference --> |
| 829 | |
| 830 | <appendix id="drmDriverApi"> |
| 831 | <title>DRM Driver API</title> |
| 832 | <para> |
| 833 | Include auto-generated API reference here (need to reference it |
| 834 | from paragraphs above too). |
| 835 | </para> |
| 836 | </appendix> |
| 837 | |
| 838 | </book> |