Markus Heiser | 5377d91 | 2016-06-30 15:18:56 +0200 | [diff] [blame] | 1 | .. -*- coding: utf-8; mode: rst -*- |
| 2 | |
| 3 | .. _subdev: |
| 4 | |
| 5 | ******************** |
| 6 | Sub-device Interface |
| 7 | ******************** |
| 8 | |
| 9 | The complex nature of V4L2 devices, where hardware is often made of |
| 10 | several integrated circuits that need to interact with each other in a |
| 11 | controlled way, leads to complex V4L2 drivers. The drivers usually |
| 12 | reflect the hardware model in software, and model the different hardware |
| 13 | components as software blocks called sub-devices. |
| 14 | |
| 15 | V4L2 sub-devices are usually kernel-only objects. If the V4L2 driver |
| 16 | implements the media device API, they will automatically inherit from |
| 17 | media entities. Applications will be able to enumerate the sub-devices |
| 18 | and discover the hardware topology using the media entities, pads and |
| 19 | links enumeration API. |
| 20 | |
| 21 | In addition to make sub-devices discoverable, drivers can also choose to |
| 22 | make them directly configurable by applications. When both the |
| 23 | sub-device driver and the V4L2 device driver support this, sub-devices |
| 24 | will feature a character device node on which ioctls can be called to |
| 25 | |
| 26 | - query, read and write sub-devices controls |
| 27 | |
| 28 | - subscribe and unsubscribe to events and retrieve them |
| 29 | |
| 30 | - negotiate image formats on individual pads |
| 31 | |
| 32 | Sub-device character device nodes, conventionally named |
| 33 | ``/dev/v4l-subdev*``, use major number 81. |
| 34 | |
| 35 | |
| 36 | Controls |
| 37 | ======== |
| 38 | |
| 39 | Most V4L2 controls are implemented by sub-device hardware. Drivers |
| 40 | usually merge all controls and expose them through video device nodes. |
| 41 | Applications can control all sub-devices through a single interface. |
| 42 | |
| 43 | Complex devices sometimes implement the same control in different pieces |
| 44 | of hardware. This situation is common in embedded platforms, where both |
| 45 | sensors and image processing hardware implement identical functions, |
| 46 | such as contrast adjustment, white balance or faulty pixels correction. |
| 47 | As the V4L2 controls API doesn't support several identical controls in a |
| 48 | single device, all but one of the identical controls are hidden. |
| 49 | |
| 50 | Applications can access those hidden controls through the sub-device |
| 51 | node with the V4L2 control API described in :ref:`control`. The ioctls |
| 52 | behave identically as when issued on V4L2 device nodes, with the |
| 53 | exception that they deal only with controls implemented in the |
| 54 | sub-device. |
| 55 | |
| 56 | Depending on the driver, those controls might also be exposed through |
| 57 | one (or several) V4L2 device nodes. |
| 58 | |
| 59 | |
| 60 | Events |
| 61 | ====== |
| 62 | |
| 63 | V4L2 sub-devices can notify applications of events as described in |
| 64 | :ref:`event`. The API behaves identically as when used on V4L2 device |
| 65 | nodes, with the exception that it only deals with events generated by |
| 66 | the sub-device. Depending on the driver, those events might also be |
| 67 | reported on one (or several) V4L2 device nodes. |
| 68 | |
| 69 | |
| 70 | .. _pad-level-formats: |
| 71 | |
| 72 | Pad-level Formats |
| 73 | ================= |
| 74 | |
| 75 | **Warning** |
| 76 | |
| 77 | Pad-level formats are only applicable to very complex device that |
| 78 | need to expose low-level format configuration to user space. Generic |
| 79 | V4L2 applications do *not* need to use the API described in this |
| 80 | section. |
| 81 | |
| 82 | **Note** |
| 83 | |
| 84 | For the purpose of this section, the term *format* means the |
| 85 | combination of media bus data format, frame width and frame height. |
| 86 | |
| 87 | Image formats are typically negotiated on video capture and output |
| 88 | devices using the format and |
| 89 | :ref:`selection <vidioc-subdev-g-selection>` ioctls. The driver is |
| 90 | responsible for configuring every block in the video pipeline according |
| 91 | to the requested format at the pipeline input and/or output. |
| 92 | |
| 93 | For complex devices, such as often found in embedded systems, identical |
| 94 | image sizes at the output of a pipeline can be achieved using different |
| 95 | hardware configurations. One such example is shown on |
| 96 | :ref:`pipeline-scaling`, where image scaling can be performed on both |
| 97 | the video sensor and the host image processing hardware. |
| 98 | |
| 99 | |
| 100 | .. _pipeline-scaling: |
| 101 | |
| 102 | .. figure:: dev-subdev_files/pipeline.* |
| 103 | :alt: pipeline.pdf / pipeline.png |
| 104 | :align: center |
| 105 | |
| 106 | Image Format Negotiation on Pipelines |
| 107 | |
| 108 | High quality and high speed pipeline configuration |
| 109 | |
| 110 | |
| 111 | |
| 112 | The sensor scaler is usually of less quality than the host scaler, but |
| 113 | scaling on the sensor is required to achieve higher frame rates. |
| 114 | Depending on the use case (quality vs. speed), the pipeline must be |
| 115 | configured differently. Applications need to configure the formats at |
| 116 | every point in the pipeline explicitly. |
| 117 | |
| 118 | Drivers that implement the :ref:`media API <media-controller-intro>` |
| 119 | can expose pad-level image format configuration to applications. When |
| 120 | they do, applications can use the |
| 121 | :ref:`VIDIOC_SUBDEV_G_FMT <vidioc-subdev-g-fmt>` and |
| 122 | :ref:`VIDIOC_SUBDEV_S_FMT <vidioc-subdev-g-fmt>` ioctls. to |
| 123 | negotiate formats on a per-pad basis. |
| 124 | |
| 125 | Applications are responsible for configuring coherent parameters on the |
| 126 | whole pipeline and making sure that connected pads have compatible |
| 127 | formats. The pipeline is checked for formats mismatch at |
| 128 | :ref:`VIDIOC_STREAMON <vidioc-streamon>` time, and an EPIPE error |
| 129 | code is then returned if the configuration is invalid. |
| 130 | |
| 131 | Pad-level image format configuration support can be tested by calling |
| 132 | the :ref:`VIDIOC_SUBDEV_G_FMT <vidioc-subdev-g-fmt>` ioctl on pad |
| 133 | 0. If the driver returns an EINVAL error code pad-level format |
| 134 | configuration is not supported by the sub-device. |
| 135 | |
| 136 | |
| 137 | Format Negotiation |
| 138 | ------------------ |
| 139 | |
| 140 | Acceptable formats on pads can (and usually do) depend on a number of |
| 141 | external parameters, such as formats on other pads, active links, or |
| 142 | even controls. Finding a combination of formats on all pads in a video |
| 143 | pipeline, acceptable to both application and driver, can't rely on |
| 144 | formats enumeration only. A format negotiation mechanism is required. |
| 145 | |
| 146 | Central to the format negotiation mechanism are the get/set format |
| 147 | operations. When called with the ``which`` argument set to |
| 148 | ``V4L2_SUBDEV_FORMAT_TRY``, the |
| 149 | :ref:`VIDIOC_SUBDEV_G_FMT <vidioc-subdev-g-fmt>` and |
| 150 | :ref:`VIDIOC_SUBDEV_S_FMT <vidioc-subdev-g-fmt>` ioctls operate on |
| 151 | a set of formats parameters that are not connected to the hardware |
| 152 | configuration. Modifying those 'try' formats leaves the device state |
| 153 | untouched (this applies to both the software state stored in the driver |
| 154 | and the hardware state stored in the device itself). |
| 155 | |
| 156 | While not kept as part of the device state, try formats are stored in |
| 157 | the sub-device file handles. A |
| 158 | :ref:`VIDIOC_SUBDEV_G_FMT <vidioc-subdev-g-fmt>` call will return |
| 159 | the last try format set *on the same sub-device file handle*. Several |
| 160 | applications querying the same sub-device at the same time will thus not |
| 161 | interact with each other. |
| 162 | |
| 163 | To find out whether a particular format is supported by the device, |
| 164 | applications use the |
| 165 | :ref:`VIDIOC_SUBDEV_S_FMT <vidioc-subdev-g-fmt>` ioctl. Drivers |
| 166 | verify and, if needed, change the requested ``format`` based on device |
| 167 | requirements and return the possibly modified value. Applications can |
| 168 | then choose to try a different format or accept the returned value and |
| 169 | continue. |
| 170 | |
| 171 | Formats returned by the driver during a negotiation iteration are |
| 172 | guaranteed to be supported by the device. In particular, drivers |
| 173 | guarantee that a returned format will not be further changed if passed |
| 174 | to an :ref:`VIDIOC_SUBDEV_S_FMT <vidioc-subdev-g-fmt>` call as-is |
| 175 | (as long as external parameters, such as formats on other pads or links' |
| 176 | configuration are not changed). |
| 177 | |
| 178 | Drivers automatically propagate formats inside sub-devices. When a try |
| 179 | or active format is set on a pad, corresponding formats on other pads of |
| 180 | the same sub-device can be modified by the driver. Drivers are free to |
| 181 | modify formats as required by the device. However, they should comply |
| 182 | with the following rules when possible: |
| 183 | |
| 184 | - Formats should be propagated from sink pads to source pads. Modifying |
| 185 | a format on a source pad should not modify the format on any sink |
| 186 | pad. |
| 187 | |
| 188 | - Sub-devices that scale frames using variable scaling factors should |
| 189 | reset the scale factors to default values when sink pads formats are |
| 190 | modified. If the 1:1 scaling ratio is supported, this means that |
| 191 | source pads formats should be reset to the sink pads formats. |
| 192 | |
| 193 | Formats are not propagated across links, as that would involve |
| 194 | propagating them from one sub-device file handle to another. |
| 195 | Applications must then take care to configure both ends of every link |
| 196 | explicitly with compatible formats. Identical formats on the two ends of |
| 197 | a link are guaranteed to be compatible. Drivers are free to accept |
| 198 | different formats matching device requirements as being compatible. |
| 199 | |
| 200 | :ref:`sample-pipeline-config` shows a sample configuration sequence |
| 201 | for the pipeline described in :ref:`pipeline-scaling` (table columns |
| 202 | list entity names and pad numbers). |
| 203 | |
| 204 | |
| 205 | .. _sample-pipeline-config: |
| 206 | |
| 207 | .. flat-table:: Sample Pipeline Configuration |
| 208 | :header-rows: 1 |
| 209 | :stub-columns: 0 |
| 210 | |
| 211 | |
| 212 | - .. row 1 |
| 213 | |
| 214 | - |
| 215 | - Sensor/0 format |
| 216 | |
| 217 | - Frontend/0 format |
| 218 | |
| 219 | - Frontend/1 format |
| 220 | |
| 221 | - Scaler/0 format |
| 222 | |
| 223 | - Scaler/0 compose selection rectangle |
| 224 | |
| 225 | - Scaler/1 format |
| 226 | |
| 227 | - .. row 2 |
| 228 | |
| 229 | - Initial state |
| 230 | |
| 231 | - 2048x1536/SGRBG8_1X8 |
| 232 | |
| 233 | - (default) |
| 234 | |
| 235 | - (default) |
| 236 | |
| 237 | - (default) |
| 238 | |
| 239 | - (default) |
| 240 | |
| 241 | - (default) |
| 242 | |
| 243 | - .. row 3 |
| 244 | |
| 245 | - Configure frontend sink format |
| 246 | |
| 247 | - 2048x1536/SGRBG8_1X8 |
| 248 | |
| 249 | - *2048x1536/SGRBG8_1X8* |
| 250 | |
| 251 | - *2046x1534/SGRBG8_1X8* |
| 252 | |
| 253 | - (default) |
| 254 | |
| 255 | - (default) |
| 256 | |
| 257 | - (default) |
| 258 | |
| 259 | - .. row 4 |
| 260 | |
| 261 | - Configure scaler sink format |
| 262 | |
| 263 | - 2048x1536/SGRBG8_1X8 |
| 264 | |
| 265 | - 2048x1536/SGRBG8_1X8 |
| 266 | |
| 267 | - 2046x1534/SGRBG8_1X8 |
| 268 | |
| 269 | - *2046x1534/SGRBG8_1X8* |
| 270 | |
| 271 | - *0,0/2046x1534* |
| 272 | |
| 273 | - *2046x1534/SGRBG8_1X8* |
| 274 | |
| 275 | - .. row 5 |
| 276 | |
| 277 | - Configure scaler sink compose selection |
| 278 | |
| 279 | - 2048x1536/SGRBG8_1X8 |
| 280 | |
| 281 | - 2048x1536/SGRBG8_1X8 |
| 282 | |
| 283 | - 2046x1534/SGRBG8_1X8 |
| 284 | |
| 285 | - 2046x1534/SGRBG8_1X8 |
| 286 | |
| 287 | - *0,0/1280x960* |
| 288 | |
| 289 | - *1280x960/SGRBG8_1X8* |
| 290 | |
| 291 | |
| 292 | |
| 293 | 1. Initial state. The sensor source pad format is set to its native 3MP |
| 294 | size and V4L2_MBUS_FMT_SGRBG8_1X8 media bus code. Formats on the |
| 295 | host frontend and scaler sink and source pads have the default |
| 296 | values, as well as the compose rectangle on the scaler's sink pad. |
| 297 | |
| 298 | 2. The application configures the frontend sink pad format's size to |
| 299 | 2048x1536 and its media bus code to V4L2_MBUS_FMT_SGRBG_1X8. The |
| 300 | driver propagates the format to the frontend source pad. |
| 301 | |
| 302 | 3. The application configures the scaler sink pad format's size to |
| 303 | 2046x1534 and the media bus code to V4L2_MBUS_FMT_SGRBG_1X8 to |
| 304 | match the frontend source size and media bus code. The media bus code |
| 305 | on the sink pad is set to V4L2_MBUS_FMT_SGRBG_1X8. The driver |
| 306 | propagates the size to the compose selection rectangle on the |
| 307 | scaler's sink pad, and the format to the scaler source pad. |
| 308 | |
| 309 | 4. The application configures the size of the compose selection |
| 310 | rectangle of the scaler's sink pad 1280x960. The driver propagates |
| 311 | the size to the scaler's source pad format. |
| 312 | |
| 313 | When satisfied with the try results, applications can set the active |
| 314 | formats by setting the ``which`` argument to |
| 315 | ``V4L2_SUBDEV_FORMAT_ACTIVE``. Active formats are changed exactly as try |
| 316 | formats by drivers. To avoid modifying the hardware state during format |
| 317 | negotiation, applications should negotiate try formats first and then |
| 318 | modify the active settings using the try formats returned during the |
| 319 | last negotiation iteration. This guarantees that the active format will |
| 320 | be applied as-is by the driver without being modified. |
| 321 | |
| 322 | |
| 323 | .. _v4l2-subdev-selections: |
| 324 | |
| 325 | Selections: cropping, scaling and composition |
| 326 | --------------------------------------------- |
| 327 | |
| 328 | Many sub-devices support cropping frames on their input or output pads |
| 329 | (or possible even on both). Cropping is used to select the area of |
| 330 | interest in an image, typically on an image sensor or a video decoder. |
| 331 | It can also be used as part of digital zoom implementations to select |
| 332 | the area of the image that will be scaled up. |
| 333 | |
| 334 | Crop settings are defined by a crop rectangle and represented in a |
| 335 | struct :ref:`v4l2_rect <v4l2-rect>` by the coordinates of the top |
| 336 | left corner and the rectangle size. Both the coordinates and sizes are |
| 337 | expressed in pixels. |
| 338 | |
| 339 | As for pad formats, drivers store try and active rectangles for the |
| 340 | selection targets :ref:`v4l2-selections-common`. |
| 341 | |
| 342 | On sink pads, cropping is applied relative to the current pad format. |
| 343 | The pad format represents the image size as received by the sub-device |
| 344 | from the previous block in the pipeline, and the crop rectangle |
| 345 | represents the sub-image that will be transmitted further inside the |
| 346 | sub-device for processing. |
| 347 | |
| 348 | The scaling operation changes the size of the image by scaling it to new |
| 349 | dimensions. The scaling ratio isn't specified explicitly, but is implied |
| 350 | from the original and scaled image sizes. Both sizes are represented by |
| 351 | struct :ref:`v4l2_rect <v4l2-rect>`. |
| 352 | |
| 353 | Scaling support is optional. When supported by a subdev, the crop |
| 354 | rectangle on the subdev's sink pad is scaled to the size configured |
| 355 | using the |
| 356 | :ref:`VIDIOC_SUBDEV_S_SELECTION <vidioc-subdev-g-selection>` IOCTL |
| 357 | using ``V4L2_SEL_TGT_COMPOSE`` selection target on the same pad. If the |
| 358 | subdev supports scaling but not composing, the top and left values are |
| 359 | not used and must always be set to zero. |
| 360 | |
| 361 | On source pads, cropping is similar to sink pads, with the exception |
| 362 | that the source size from which the cropping is performed, is the |
| 363 | COMPOSE rectangle on the sink pad. In both sink and source pads, the |
| 364 | crop rectangle must be entirely contained inside the source image size |
| 365 | for the crop operation. |
| 366 | |
| 367 | The drivers should always use the closest possible rectangle the user |
| 368 | requests on all selection targets, unless specifically told otherwise. |
| 369 | ``V4L2_SEL_FLAG_GE`` and ``V4L2_SEL_FLAG_LE`` flags may be used to round |
| 370 | the image size either up or down. :ref:`v4l2-selection-flags` |
| 371 | |
| 372 | |
| 373 | Types of selection targets |
| 374 | -------------------------- |
| 375 | |
| 376 | |
| 377 | Actual targets |
| 378 | ^^^^^^^^^^^^^^ |
| 379 | |
| 380 | Actual targets (without a postfix) reflect the actual hardware |
| 381 | configuration at any point of time. There is a BOUNDS target |
| 382 | corresponding to every actual target. |
| 383 | |
| 384 | |
| 385 | BOUNDS targets |
| 386 | ^^^^^^^^^^^^^^ |
| 387 | |
| 388 | BOUNDS targets is the smallest rectangle that contains all valid actual |
| 389 | rectangles. It may not be possible to set the actual rectangle as large |
| 390 | as the BOUNDS rectangle, however. This may be because e.g. a sensor's |
| 391 | pixel array is not rectangular but cross-shaped or round. The maximum |
| 392 | size may also be smaller than the BOUNDS rectangle. |
| 393 | |
| 394 | |
| 395 | Order of configuration and format propagation |
| 396 | --------------------------------------------- |
| 397 | |
| 398 | Inside subdevs, the order of image processing steps will always be from |
| 399 | the sink pad towards the source pad. This is also reflected in the order |
| 400 | in which the configuration must be performed by the user: the changes |
| 401 | made will be propagated to any subsequent stages. If this behaviour is |
| 402 | not desired, the user must set ``V4L2_SEL_FLAG_KEEP_CONFIG`` flag. This |
| 403 | flag causes no propagation of the changes are allowed in any |
| 404 | circumstances. This may also cause the accessed rectangle to be adjusted |
| 405 | by the driver, depending on the properties of the underlying hardware. |
| 406 | |
| 407 | The coordinates to a step always refer to the actual size of the |
| 408 | previous step. The exception to this rule is the source compose |
| 409 | rectangle, which refers to the sink compose bounds rectangle --- if it |
| 410 | is supported by the hardware. |
| 411 | |
| 412 | 1. Sink pad format. The user configures the sink pad format. This format |
| 413 | defines the parameters of the image the entity receives through the |
| 414 | pad for further processing. |
| 415 | |
| 416 | 2. Sink pad actual crop selection. The sink pad crop defines the crop |
| 417 | performed to the sink pad format. |
| 418 | |
| 419 | 3. Sink pad actual compose selection. The size of the sink pad compose |
| 420 | rectangle defines the scaling ratio compared to the size of the sink |
| 421 | pad crop rectangle. The location of the compose rectangle specifies |
| 422 | the location of the actual sink compose rectangle in the sink compose |
| 423 | bounds rectangle. |
| 424 | |
| 425 | 4. Source pad actual crop selection. Crop on the source pad defines crop |
| 426 | performed to the image in the sink compose bounds rectangle. |
| 427 | |
| 428 | 5. Source pad format. The source pad format defines the output pixel |
| 429 | format of the subdev, as well as the other parameters with the |
| 430 | exception of the image width and height. Width and height are defined |
| 431 | by the size of the source pad actual crop selection. |
| 432 | |
| 433 | Accessing any of the above rectangles not supported by the subdev will |
| 434 | return ``EINVAL``. Any rectangle referring to a previous unsupported |
| 435 | rectangle coordinates will instead refer to the previous supported |
| 436 | rectangle. For example, if sink crop is not supported, the compose |
| 437 | selection will refer to the sink pad format dimensions instead. |
| 438 | |
| 439 | |
| 440 | .. _subdev-image-processing-crop: |
| 441 | |
| 442 | .. figure:: dev-subdev_files/subdev-image-processing-crop.* |
| 443 | :alt: subdev-image-processing-crop.svg |
| 444 | :align: center |
| 445 | |
| 446 | Image processing in subdevs: simple crop example |
| 447 | |
| 448 | In the above example, the subdev supports cropping on its sink pad. To |
| 449 | configure it, the user sets the media bus format on the subdev's sink |
| 450 | pad. Now the actual crop rectangle can be set on the sink pad --- the |
| 451 | location and size of this rectangle reflect the location and size of a |
| 452 | rectangle to be cropped from the sink format. The size of the sink crop |
| 453 | rectangle will also be the size of the format of the subdev's source |
| 454 | pad. |
| 455 | |
| 456 | |
| 457 | .. _subdev-image-processing-scaling-multi-source: |
| 458 | |
| 459 | .. figure:: dev-subdev_files/subdev-image-processing-scaling-multi-source.* |
| 460 | :alt: subdev-image-processing-scaling-multi-source.svg |
| 461 | :align: center |
| 462 | |
| 463 | Image processing in subdevs: scaling with multiple sources |
| 464 | |
| 465 | In this example, the subdev is capable of first cropping, then scaling |
| 466 | and finally cropping for two source pads individually from the resulting |
| 467 | scaled image. The location of the scaled image in the cropped image is |
| 468 | ignored in sink compose target. Both of the locations of the source crop |
| 469 | rectangles refer to the sink scaling rectangle, independently cropping |
| 470 | an area at location specified by the source crop rectangle from it. |
| 471 | |
| 472 | |
| 473 | .. _subdev-image-processing-full: |
| 474 | |
| 475 | .. figure:: dev-subdev_files/subdev-image-processing-full.* |
| 476 | :alt: subdev-image-processing-full.svg |
| 477 | :align: center |
| 478 | |
| 479 | Image processing in subdevs: scaling and composition with multiple sinks and sources |
| 480 | |
| 481 | The subdev driver supports two sink pads and two source pads. The images |
| 482 | from both of the sink pads are individually cropped, then scaled and |
| 483 | further composed on the composition bounds rectangle. From that, two |
| 484 | independent streams are cropped and sent out of the subdev from the |
| 485 | source pads. |
| 486 | |
| 487 | |
| 488 | .. toctree:: |
| 489 | :maxdepth: 1 |
| 490 | |
| 491 | subdev-formats |
| 492 | |
| 493 | |
| 494 | |
| 495 | |
| 496 | .. ------------------------------------------------------------------------------ |
| 497 | .. This file was automatically converted from DocBook-XML with the dbxml |
| 498 | .. library (https://github.com/return42/sphkerneldoc). The origin XML comes |
| 499 | .. from the linux kernel, refer to: |
| 500 | .. |
| 501 | .. * https://github.com/torvalds/linux/tree/master/Documentation/DocBook |
| 502 | .. ------------------------------------------------------------------------------ |