Laurent Pinchart | 333c8b9 | 2010-03-15 20:26:04 -0300 | [diff] [blame] | 1 | <title>Sub-device Interface</title> |
| 2 | |
| 3 | <note> |
| 4 | <title>Experimental</title> |
| 5 | <para>This is an <link linkend="experimental">experimental</link> |
| 6 | interface and may change in the future.</para> |
| 7 | </note> |
| 8 | |
| 9 | <para>The complex nature of V4L2 devices, where hardware is often made of |
| 10 | several integrated circuits that need to interact with each other in a |
| 11 | controlled way, leads to complex V4L2 drivers. The drivers usually reflect |
| 12 | the hardware model in software, and model the different hardware components |
| 13 | as software blocks called sub-devices.</para> |
| 14 | |
| 15 | <para>V4L2 sub-devices are usually kernel-only objects. If the V4L2 driver |
| 16 | implements the media device API, they will automatically inherit from media |
| 17 | entities. Applications will be able to enumerate the sub-devices and discover |
| 18 | the hardware topology using the media entities, pads and links enumeration |
| 19 | API.</para> |
| 20 | |
| 21 | <para>In addition to make sub-devices discoverable, drivers can also choose |
| 22 | to make them directly configurable by applications. When both the sub-device |
| 23 | driver and the V4L2 device driver support this, sub-devices will feature a |
| 24 | character device node on which ioctls can be called to |
| 25 | <itemizedlist> |
Hans Verkuil | 665bf36 | 2011-03-11 16:22:21 -0300 | [diff] [blame] | 26 | <listitem><para>query, read and write sub-devices controls</para></listitem> |
| 27 | <listitem><para>subscribe and unsubscribe to events and retrieve them</para></listitem> |
| 28 | <listitem><para>negotiate image formats on individual pads</para></listitem> |
Laurent Pinchart | 333c8b9 | 2010-03-15 20:26:04 -0300 | [diff] [blame] | 29 | </itemizedlist> |
| 30 | </para> |
| 31 | |
| 32 | <para>Sub-device character device nodes, conventionally named |
| 33 | <filename>/dev/v4l-subdev*</filename>, use major number 81.</para> |
| 34 | |
| 35 | <section> |
| 36 | <title>Controls</title> |
| 37 | <para>Most V4L2 controls are implemented by sub-device hardware. Drivers |
| 38 | usually merge all controls and expose them through video device nodes. |
| 39 | Applications can control all sub-devices through a single interface.</para> |
| 40 | |
| 41 | <para>Complex devices sometimes implement the same control in different |
| 42 | pieces of hardware. This situation is common in embedded platforms, where |
| 43 | both sensors and image processing hardware implement identical functions, |
| 44 | such as contrast adjustment, white balance or faulty pixels correction. As |
| 45 | the V4L2 controls API doesn't support several identical controls in a single |
| 46 | device, all but one of the identical controls are hidden.</para> |
| 47 | |
| 48 | <para>Applications can access those hidden controls through the sub-device |
| 49 | node with the V4L2 control API described in <xref linkend="control" />. The |
| 50 | ioctls behave identically as when issued on V4L2 device nodes, with the |
| 51 | exception that they deal only with controls implemented in the sub-device. |
| 52 | </para> |
| 53 | |
| 54 | <para>Depending on the driver, those controls might also be exposed through |
| 55 | one (or several) V4L2 device nodes.</para> |
| 56 | </section> |
| 57 | |
| 58 | <section> |
| 59 | <title>Events</title> |
| 60 | <para>V4L2 sub-devices can notify applications of events as described in |
| 61 | <xref linkend="event" />. The API behaves identically as when used on V4L2 |
| 62 | device nodes, with the exception that it only deals with events generated by |
| 63 | the sub-device. Depending on the driver, those events might also be reported |
| 64 | on one (or several) V4L2 device nodes.</para> |
| 65 | </section> |
| 66 | |
| 67 | <section id="pad-level-formats"> |
| 68 | <title>Pad-level Formats</title> |
| 69 | |
Hans Verkuil | 665bf36 | 2011-03-11 16:22:21 -0300 | [diff] [blame] | 70 | <warning><para>Pad-level formats are only applicable to very complex device that |
Laurent Pinchart | 333c8b9 | 2010-03-15 20:26:04 -0300 | [diff] [blame] | 71 | need to expose low-level format configuration to user space. Generic V4L2 |
| 72 | applications do <emphasis>not</emphasis> need to use the API described in |
Hans Verkuil | 665bf36 | 2011-03-11 16:22:21 -0300 | [diff] [blame] | 73 | this section.</para></warning> |
Laurent Pinchart | 333c8b9 | 2010-03-15 20:26:04 -0300 | [diff] [blame] | 74 | |
Hans Verkuil | 665bf36 | 2011-03-11 16:22:21 -0300 | [diff] [blame] | 75 | <note><para>For the purpose of this section, the term |
Laurent Pinchart | 333c8b9 | 2010-03-15 20:26:04 -0300 | [diff] [blame] | 76 | <wordasword>format</wordasword> means the combination of media bus data |
Hans Verkuil | 665bf36 | 2011-03-11 16:22:21 -0300 | [diff] [blame] | 77 | format, frame width and frame height.</para></note> |
Laurent Pinchart | 333c8b9 | 2010-03-15 20:26:04 -0300 | [diff] [blame] | 78 | |
| 79 | <para>Image formats are typically negotiated on video capture and output |
| 80 | devices using the <link linkend="crop">cropping and scaling</link> ioctls. |
| 81 | The driver is responsible for configuring every block in the video pipeline |
| 82 | according to the requested format at the pipeline input and/or |
| 83 | output.</para> |
| 84 | |
| 85 | <para>For complex devices, such as often found in embedded systems, |
| 86 | identical image sizes at the output of a pipeline can be achieved using |
Hans Verkuil | 665bf36 | 2011-03-11 16:22:21 -0300 | [diff] [blame] | 87 | different hardware configurations. One such example is shown on |
| 88 | <xref linkend="pipeline-scaling" />, where |
Laurent Pinchart | 333c8b9 | 2010-03-15 20:26:04 -0300 | [diff] [blame] | 89 | image scaling can be performed on both the video sensor and the host image |
| 90 | processing hardware.</para> |
| 91 | |
| 92 | <figure id="pipeline-scaling"> |
Lucas De Marchi | 25985ed | 2011-03-30 22:57:33 -0300 | [diff] [blame] | 93 | <title>Image Format Negotiation on Pipelines</title> |
Laurent Pinchart | 333c8b9 | 2010-03-15 20:26:04 -0300 | [diff] [blame] | 94 | <mediaobject> |
| 95 | <imageobject> |
| 96 | <imagedata fileref="pipeline.pdf" format="PS" /> |
| 97 | </imageobject> |
| 98 | <imageobject> |
| 99 | <imagedata fileref="pipeline.png" format="PNG" /> |
| 100 | </imageobject> |
| 101 | <textobject> |
| 102 | <phrase>High quality and high speed pipeline configuration</phrase> |
| 103 | </textobject> |
| 104 | </mediaobject> |
| 105 | </figure> |
| 106 | |
| 107 | <para>The sensor scaler is usually of less quality than the host scaler, but |
| 108 | scaling on the sensor is required to achieve higher frame rates. Depending |
| 109 | on the use case (quality vs. speed), the pipeline must be configured |
| 110 | differently. Applications need to configure the formats at every point in |
| 111 | the pipeline explicitly.</para> |
| 112 | |
| 113 | <para>Drivers that implement the <link linkend="media-controller-intro">media |
| 114 | API</link> can expose pad-level image format configuration to applications. |
| 115 | When they do, applications can use the &VIDIOC-SUBDEV-G-FMT; and |
| 116 | &VIDIOC-SUBDEV-S-FMT; ioctls. to negotiate formats on a per-pad basis.</para> |
| 117 | |
| 118 | <para>Applications are responsible for configuring coherent parameters on |
| 119 | the whole pipeline and making sure that connected pads have compatible |
| 120 | formats. The pipeline is checked for formats mismatch at &VIDIOC-STREAMON; |
| 121 | time, and an &EPIPE; is then returned if the configuration is |
| 122 | invalid.</para> |
| 123 | |
| 124 | <para>Pad-level image format configuration support can be tested by calling |
| 125 | the &VIDIOC-SUBDEV-G-FMT; ioctl on pad 0. If the driver returns an &EINVAL; |
| 126 | pad-level format configuration is not supported by the sub-device.</para> |
| 127 | |
| 128 | <section> |
| 129 | <title>Format Negotiation</title> |
| 130 | |
| 131 | <para>Acceptable formats on pads can (and usually do) depend on a number |
| 132 | of external parameters, such as formats on other pads, active links, or |
| 133 | even controls. Finding a combination of formats on all pads in a video |
| 134 | pipeline, acceptable to both application and driver, can't rely on formats |
| 135 | enumeration only. A format negotiation mechanism is required.</para> |
| 136 | |
| 137 | <para>Central to the format negotiation mechanism are the get/set format |
| 138 | operations. When called with the <structfield>which</structfield> argument |
| 139 | set to <constant>V4L2_SUBDEV_FORMAT_TRY</constant>, the |
| 140 | &VIDIOC-SUBDEV-G-FMT; and &VIDIOC-SUBDEV-S-FMT; ioctls operate on a set of |
| 141 | formats parameters that are not connected to the hardware configuration. |
| 142 | Modifying those 'try' formats leaves the device state untouched (this |
| 143 | applies to both the software state stored in the driver and the hardware |
| 144 | state stored in the device itself).</para> |
| 145 | |
| 146 | <para>While not kept as part of the device state, try formats are stored |
| 147 | in the sub-device file handles. A &VIDIOC-SUBDEV-G-FMT; call will return |
| 148 | the last try format set <emphasis>on the same sub-device file |
| 149 | handle</emphasis>. Several applications querying the same sub-device at |
| 150 | the same time will thus not interact with each other.</para> |
| 151 | |
| 152 | <para>To find out whether a particular format is supported by the device, |
| 153 | applications use the &VIDIOC-SUBDEV-S-FMT; ioctl. Drivers verify and, if |
| 154 | needed, change the requested <structfield>format</structfield> based on |
| 155 | device requirements and return the possibly modified value. Applications |
| 156 | can then choose to try a different format or accept the returned value and |
| 157 | continue.</para> |
| 158 | |
| 159 | <para>Formats returned by the driver during a negotiation iteration are |
| 160 | guaranteed to be supported by the device. In particular, drivers guarantee |
| 161 | that a returned format will not be further changed if passed to an |
| 162 | &VIDIOC-SUBDEV-S-FMT; call as-is (as long as external parameters, such as |
| 163 | formats on other pads or links' configuration are not changed).</para> |
| 164 | |
| 165 | <para>Drivers automatically propagate formats inside sub-devices. When a |
| 166 | try or active format is set on a pad, corresponding formats on other pads |
| 167 | of the same sub-device can be modified by the driver. Drivers are free to |
| 168 | modify formats as required by the device. However, they should comply with |
| 169 | the following rules when possible: |
| 170 | <itemizedlist> |
Hans Verkuil | 665bf36 | 2011-03-11 16:22:21 -0300 | [diff] [blame] | 171 | <listitem><para>Formats should be propagated from sink pads to source pads. |
Laurent Pinchart | 333c8b9 | 2010-03-15 20:26:04 -0300 | [diff] [blame] | 172 | Modifying a format on a source pad should not modify the format on any |
Hans Verkuil | 665bf36 | 2011-03-11 16:22:21 -0300 | [diff] [blame] | 173 | sink pad.</para></listitem> |
| 174 | <listitem><para>Sub-devices that scale frames using variable scaling factors |
Laurent Pinchart | 333c8b9 | 2010-03-15 20:26:04 -0300 | [diff] [blame] | 175 | should reset the scale factors to default values when sink pads formats |
| 176 | are modified. If the 1:1 scaling ratio is supported, this means that |
Hans Verkuil | 665bf36 | 2011-03-11 16:22:21 -0300 | [diff] [blame] | 177 | source pads formats should be reset to the sink pads formats.</para></listitem> |
Laurent Pinchart | 333c8b9 | 2010-03-15 20:26:04 -0300 | [diff] [blame] | 178 | </itemizedlist> |
| 179 | </para> |
| 180 | |
| 181 | <para>Formats are not propagated across links, as that would involve |
| 182 | propagating them from one sub-device file handle to another. Applications |
| 183 | must then take care to configure both ends of every link explicitly with |
| 184 | compatible formats. Identical formats on the two ends of a link are |
| 185 | guaranteed to be compatible. Drivers are free to accept different formats |
| 186 | matching device requirements as being compatible.</para> |
| 187 | |
Hans Verkuil | 665bf36 | 2011-03-11 16:22:21 -0300 | [diff] [blame] | 188 | <para><xref linkend="sample-pipeline-config" /> |
Laurent Pinchart | 333c8b9 | 2010-03-15 20:26:04 -0300 | [diff] [blame] | 189 | shows a sample configuration sequence for the pipeline described in |
Hans Verkuil | 665bf36 | 2011-03-11 16:22:21 -0300 | [diff] [blame] | 190 | <xref linkend="pipeline-scaling" /> (table |
Laurent Pinchart | 333c8b9 | 2010-03-15 20:26:04 -0300 | [diff] [blame] | 191 | columns list entity names and pad numbers).</para> |
| 192 | |
| 193 | <table pgwide="0" frame="none" id="sample-pipeline-config"> |
| 194 | <title>Sample Pipeline Configuration</title> |
| 195 | <tgroup cols="3"> |
| 196 | <colspec colname="what"/> |
| 197 | <colspec colname="sensor-0" /> |
| 198 | <colspec colname="frontend-0" /> |
| 199 | <colspec colname="frontend-1" /> |
| 200 | <colspec colname="scaler-0" /> |
| 201 | <colspec colname="scaler-1" /> |
| 202 | <thead> |
| 203 | <row> |
| 204 | <entry></entry> |
| 205 | <entry>Sensor/0</entry> |
| 206 | <entry>Frontend/0</entry> |
| 207 | <entry>Frontend/1</entry> |
| 208 | <entry>Scaler/0</entry> |
| 209 | <entry>Scaler/1</entry> |
| 210 | </row> |
| 211 | </thead> |
| 212 | <tbody valign="top"> |
| 213 | <row> |
| 214 | <entry>Initial state</entry> |
| 215 | <entry>2048x1536</entry> |
| 216 | <entry>-</entry> |
| 217 | <entry>-</entry> |
| 218 | <entry>-</entry> |
| 219 | <entry>-</entry> |
| 220 | </row> |
| 221 | <row> |
| 222 | <entry>Configure frontend input</entry> |
| 223 | <entry>2048x1536</entry> |
| 224 | <entry><emphasis>2048x1536</emphasis></entry> |
| 225 | <entry><emphasis>2046x1534</emphasis></entry> |
| 226 | <entry>-</entry> |
| 227 | <entry>-</entry> |
| 228 | </row> |
| 229 | <row> |
| 230 | <entry>Configure scaler input</entry> |
| 231 | <entry>2048x1536</entry> |
| 232 | <entry>2048x1536</entry> |
| 233 | <entry>2046x1534</entry> |
| 234 | <entry><emphasis>2046x1534</emphasis></entry> |
| 235 | <entry><emphasis>2046x1534</emphasis></entry> |
| 236 | </row> |
| 237 | <row> |
| 238 | <entry>Configure scaler output</entry> |
| 239 | <entry>2048x1536</entry> |
| 240 | <entry>2048x1536</entry> |
| 241 | <entry>2046x1534</entry> |
| 242 | <entry>2046x1534</entry> |
| 243 | <entry><emphasis>1280x960</emphasis></entry> |
| 244 | </row> |
| 245 | </tbody> |
| 246 | </tgroup> |
| 247 | </table> |
| 248 | |
| 249 | <para> |
| 250 | <orderedlist> |
Hans Verkuil | 665bf36 | 2011-03-11 16:22:21 -0300 | [diff] [blame] | 251 | <listitem><para>Initial state. The sensor output is set to its native 3MP |
Laurent Pinchart | 333c8b9 | 2010-03-15 20:26:04 -0300 | [diff] [blame] | 252 | resolution. Resolutions on the host frontend and scaler input and output |
Hans Verkuil | 665bf36 | 2011-03-11 16:22:21 -0300 | [diff] [blame] | 253 | pads are undefined.</para></listitem> |
| 254 | <listitem><para>The application configures the frontend input pad resolution to |
Laurent Pinchart | 333c8b9 | 2010-03-15 20:26:04 -0300 | [diff] [blame] | 255 | 2048x1536. The driver propagates the format to the frontend output pad. |
| 256 | Note that the propagated output format can be different, as in this case, |
| 257 | than the input format, as the hardware might need to crop pixels (for |
Hans Verkuil | 665bf36 | 2011-03-11 16:22:21 -0300 | [diff] [blame] | 258 | instance when converting a Bayer filter pattern to RGB or YUV).</para></listitem> |
| 259 | <listitem><para>The application configures the scaler input pad resolution to |
Laurent Pinchart | 333c8b9 | 2010-03-15 20:26:04 -0300 | [diff] [blame] | 260 | 2046x1534 to match the frontend output resolution. The driver propagates |
Hans Verkuil | 665bf36 | 2011-03-11 16:22:21 -0300 | [diff] [blame] | 261 | the format to the scaler output pad.</para></listitem> |
| 262 | <listitem><para>The application configures the scaler output pad resolution to |
| 263 | 1280x960.</para></listitem> |
Laurent Pinchart | 333c8b9 | 2010-03-15 20:26:04 -0300 | [diff] [blame] | 264 | </orderedlist> |
| 265 | </para> |
| 266 | |
| 267 | <para>When satisfied with the try results, applications can set the active |
| 268 | formats by setting the <structfield>which</structfield> argument to |
| 269 | <constant>V4L2_SUBDEV_FORMAT_TRY</constant>. Active formats are changed |
| 270 | exactly as try formats by drivers. To avoid modifying the hardware state |
| 271 | during format negotiation, applications should negotiate try formats first |
| 272 | and then modify the active settings using the try formats returned during |
| 273 | the last negotiation iteration. This guarantees that the active format |
| 274 | will be applied as-is by the driver without being modified. |
| 275 | </para> |
| 276 | </section> |
| 277 | |
Antti Koskipaa | f6a5cb1 | 2010-06-23 05:03:42 -0300 | [diff] [blame] | 278 | <section> |
| 279 | <title>Cropping and scaling</title> |
| 280 | |
| 281 | <para>Many sub-devices support cropping frames on their input or output |
| 282 | pads (or possible even on both). Cropping is used to select the area of |
| 283 | interest in an image, typically on a video sensor or video decoder. It can |
| 284 | also be used as part of digital zoom implementations to select the area of |
| 285 | the image that will be scaled up.</para> |
| 286 | |
| 287 | <para>Crop settings are defined by a crop rectangle and represented in a |
| 288 | &v4l2-rect; by the coordinates of the top left corner and the rectangle |
| 289 | size. Both the coordinates and sizes are expressed in pixels.</para> |
| 290 | |
| 291 | <para>The crop rectangle is retrieved and set using the |
| 292 | &VIDIOC-SUBDEV-G-CROP; and &VIDIOC-SUBDEV-S-CROP; ioctls. Like for pad |
| 293 | formats, drivers store try and active crop rectangles. The format |
| 294 | negotiation mechanism applies to crop settings as well.</para> |
| 295 | |
| 296 | <para>On input pads, cropping is applied relatively to the current pad |
| 297 | format. The pad format represents the image size as received by the |
| 298 | sub-device from the previous block in the pipeline, and the crop rectangle |
| 299 | represents the sub-image that will be transmitted further inside the |
| 300 | sub-device for processing. The crop rectangle be entirely containted |
| 301 | inside the input image size.</para> |
| 302 | |
| 303 | <para>Input crop rectangle are reset to their default value when the input |
| 304 | image format is modified. Drivers should use the input image size as the |
| 305 | crop rectangle default value, but hardware requirements may prevent this. |
| 306 | </para> |
| 307 | |
| 308 | <para>Cropping behaviour on output pads is not defined.</para> |
| 309 | |
| 310 | </section> |
Laurent Pinchart | 333c8b9 | 2010-03-15 20:26:04 -0300 | [diff] [blame] | 311 | </section> |
| 312 | |
| 313 | &sub-subdev-formats; |