Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 1 | Media Controller devices |
| 2 | ------------------------ |
| 3 | |
| 4 | Media Controller |
| 5 | ~~~~~~~~~~~~~~~~ |
| 6 | |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 7 | The media controller userspace API is documented in |
Mauro Carvalho Chehab | da83c88 | 2016-07-20 10:36:18 -0300 | [diff] [blame] | 8 | :ref:`the Media Controller uAPI book <media_controller>`. This document focus |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 9 | on the kernel-side implementation of the media framework. |
| 10 | |
| 11 | Abstract media device model |
| 12 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 13 | |
| 14 | Discovering a device internal topology, and configuring it at runtime, is one |
| 15 | of the goals of the media framework. To achieve this, hardware devices are |
| 16 | modelled as an oriented graph of building blocks called entities connected |
| 17 | through pads. |
| 18 | |
| 19 | An entity is a basic media hardware building block. It can correspond to |
| 20 | a large variety of logical blocks such as physical hardware devices |
| 21 | (CMOS sensor for instance), logical hardware devices (a building block |
| 22 | in a System-on-Chip image processing pipeline), DMA channels or physical |
| 23 | connectors. |
| 24 | |
| 25 | A pad is a connection endpoint through which an entity can interact with |
| 26 | other entities. Data (not restricted to video) produced by an entity |
| 27 | flows from the entity's output to one or more entity inputs. Pads should |
| 28 | not be confused with physical pins at chip boundaries. |
| 29 | |
| 30 | A link is a point-to-point oriented connection between two pads, either |
| 31 | on the same entity or on different entities. Data flows from a source |
| 32 | pad to a sink pad. |
| 33 | |
| 34 | Media device |
| 35 | ^^^^^^^^^^^^ |
| 36 | |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 37 | A media device is represented by a :c:type:`struct media_device <media_device>` |
| 38 | instance, defined in ``include/media/media-device.h``. |
| 39 | Allocation of the structure is handled by the media device driver, usually by |
| 40 | embedding the :c:type:`media_device` instance in a larger driver-specific |
| 41 | structure. |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 42 | |
| 43 | Drivers register media device instances by calling |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 44 | :cpp:func:`__media_device_register()` via the macro ``media_device_register()`` |
| 45 | and unregistered by calling :cpp:func:`media_device_unregister()`. |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 46 | |
| 47 | Entities |
| 48 | ^^^^^^^^ |
| 49 | |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 50 | Entities are represented by a :c:type:`struct media_entity <media_entity>` |
| 51 | instance, defined in ``include/media/media-entity.h``. The structure is usually |
| 52 | embedded into a higher-level structure, such as |
Mauro Carvalho Chehab | 041d821 | 2016-07-21 15:29:11 -0300 | [diff] [blame] | 53 | :c:type:`v4l2_subdev` or :c:type:`video_device` |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 54 | instances, although drivers can allocate entities directly. |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 55 | |
| 56 | Drivers initialize entity pads by calling |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 57 | :cpp:func:`media_entity_pads_init()`. |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 58 | |
| 59 | Drivers register entities with a media device by calling |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 60 | :cpp:func:`media_device_register_entity()` |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 61 | and unregistred by calling |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 62 | :cpp:func:`media_device_unregister_entity()`. |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 63 | |
| 64 | Interfaces |
| 65 | ^^^^^^^^^^ |
| 66 | |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 67 | Interfaces are represented by a |
| 68 | :c:type:`struct media_interface <media_interface>` instance, defined in |
| 69 | ``include/media/media-entity.h``. Currently, only one type of interface is |
| 70 | defined: a device node. Such interfaces are represented by a |
| 71 | :c:type:`struct media_intf_devnode <media_intf_devnode>`. |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 72 | |
| 73 | Drivers initialize and create device node interfaces by calling |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 74 | :cpp:func:`media_devnode_create()` |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 75 | and remove them by calling: |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 76 | :cpp:func:`media_devnode_remove()`. |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 77 | |
| 78 | Pads |
| 79 | ^^^^ |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 80 | Pads are represented by a :c:type:`struct media_pad <media_pad>` instance, |
| 81 | defined in ``include/media/media-entity.h``. Each entity stores its pads in |
| 82 | a pads array managed by the entity driver. Drivers usually embed the array in |
| 83 | a driver-specific structure. |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 84 | |
| 85 | Pads are identified by their entity and their 0-based index in the pads |
| 86 | array. |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 87 | |
| 88 | Both information are stored in the :c:type:`struct media_pad`, making the |
| 89 | :c:type:`media_pad` pointer the canonical way to store and pass link references. |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 90 | |
| 91 | Pads have flags that describe the pad capabilities and state. |
| 92 | |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 93 | ``MEDIA_PAD_FL_SINK`` indicates that the pad supports sinking data. |
| 94 | ``MEDIA_PAD_FL_SOURCE`` indicates that the pad supports sourcing data. |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 95 | |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 96 | .. note:: |
| 97 | |
| 98 | One and only one of ``MEDIA_PAD_FL_SINK`` or ``MEDIA_PAD_FL_SOURCE`` must |
| 99 | be set for each pad. |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 100 | |
| 101 | Links |
| 102 | ^^^^^ |
| 103 | |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 104 | Links are represented by a :c:type:`struct media_link <media_link>` instance, |
| 105 | defined in ``include/media/media-entity.h``. There are two types of links: |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 106 | |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 107 | **1. pad to pad links**: |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 108 | |
| 109 | Associate two entities via their PADs. Each entity has a list that points |
| 110 | to all links originating at or targeting any of its pads. |
| 111 | A given link is thus stored twice, once in the source entity and once in |
| 112 | the target entity. |
| 113 | |
| 114 | Drivers create pad to pad links by calling: |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 115 | :cpp:func:`media_create_pad_link()` and remove with |
| 116 | :cpp:func:`media_entity_remove_links()`. |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 117 | |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 118 | **2. interface to entity links**: |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 119 | |
| 120 | Associate one interface to a Link. |
| 121 | |
| 122 | Drivers create interface to entity links by calling: |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 123 | :cpp:func:`media_create_intf_link()` and remove with |
| 124 | :cpp:func:`media_remove_intf_links()`. |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 125 | |
| 126 | .. note:: |
| 127 | |
| 128 | Links can only be created after having both ends already created. |
| 129 | |
| 130 | Links have flags that describe the link capabilities and state. The |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 131 | valid values are described at :cpp:func:`media_create_pad_link()` and |
| 132 | :cpp:func:`media_create_intf_link()`. |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 133 | |
| 134 | Graph traversal |
| 135 | ^^^^^^^^^^^^^^^ |
| 136 | |
| 137 | The media framework provides APIs to iterate over entities in a graph. |
| 138 | |
| 139 | To iterate over all entities belonging to a media device, drivers can use |
| 140 | the media_device_for_each_entity macro, defined in |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 141 | ``include/media/media-device.h``. |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 142 | |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 143 | .. code-block:: c |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 144 | |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 145 | struct media_entity *entity; |
| 146 | |
| 147 | media_device_for_each_entity(entity, mdev) { |
| 148 | // entity will point to each entity in turn |
| 149 | ... |
| 150 | } |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 151 | |
| 152 | Drivers might also need to iterate over all entities in a graph that can be |
| 153 | reached only through enabled links starting at a given entity. The media |
| 154 | framework provides a depth-first graph traversal API for that purpose. |
| 155 | |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 156 | .. note:: |
| 157 | |
| 158 | Graphs with cycles (whether directed or undirected) are **NOT** |
| 159 | supported by the graph traversal API. To prevent infinite loops, the graph |
| 160 | traversal code limits the maximum depth to ``MEDIA_ENTITY_ENUM_MAX_DEPTH``, |
| 161 | currently defined as 16. |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 162 | |
| 163 | Drivers initiate a graph traversal by calling |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 164 | :cpp:func:`media_entity_graph_walk_start()` |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 165 | |
| 166 | The graph structure, provided by the caller, is initialized to start graph |
| 167 | traversal at the given entity. |
| 168 | |
| 169 | Drivers can then retrieve the next entity by calling |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 170 | :cpp:func:`media_entity_graph_walk_next()` |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 171 | |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 172 | When the graph traversal is complete the function will return ``NULL``. |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 173 | |
| 174 | Graph traversal can be interrupted at any moment. No cleanup function call |
| 175 | is required and the graph structure can be freed normally. |
| 176 | |
| 177 | Helper functions can be used to find a link between two given pads, or a pad |
| 178 | connected to another pad through an enabled link |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 179 | :cpp:func:`media_entity_find_link()` and |
| 180 | :cpp:func:`media_entity_remote_pad()`. |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 181 | |
| 182 | Use count and power handling |
| 183 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 184 | |
| 185 | Due to the wide differences between drivers regarding power management |
| 186 | needs, the media controller does not implement power management. However, |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 187 | the :c:type:`struct media_entity <media_entity>` includes a ``use_count`` |
| 188 | field that media drivers |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 189 | can use to track the number of users of every entity for power management |
| 190 | needs. |
| 191 | |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 192 | The :c:type:`media_entity<media_entity>`.\ ``use_count`` field is owned by |
| 193 | media drivers and must not be |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 194 | touched by entity drivers. Access to the field must be protected by the |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 195 | :c:type:`media_device`.\ ``graph_mutex`` lock. |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 196 | |
| 197 | Links setup |
| 198 | ^^^^^^^^^^^ |
| 199 | |
| 200 | Link properties can be modified at runtime by calling |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 201 | :cpp:func:`media_entity_setup_link()`. |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 202 | |
| 203 | Pipelines and media streams |
| 204 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 205 | |
| 206 | When starting streaming, drivers must notify all entities in the pipeline to |
| 207 | prevent link states from being modified during streaming by calling |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 208 | :cpp:func:`media_entity_pipeline_start()`. |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 209 | |
| 210 | The function will mark all entities connected to the given entity through |
| 211 | enabled links, either directly or indirectly, as streaming. |
| 212 | |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 213 | The :c:type:`struct media_pipeline <media_pipeline>` instance pointed to by |
| 214 | the pipe argument will be stored in every entity in the pipeline. |
| 215 | Drivers should embed the :c:type:`struct media_pipeline <media_pipeline>` |
| 216 | in higher-level pipeline structures and can then access the |
| 217 | pipeline through the :c:type:`struct media_entity <media_entity>` |
| 218 | pipe field. |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 219 | |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 220 | Calls to :cpp:func:`media_entity_pipeline_start()` can be nested. |
| 221 | The pipeline pointer must be identical for all nested calls to the function. |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 222 | |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 223 | :cpp:func:`media_entity_pipeline_start()` may return an error. In that case, |
| 224 | it will clean up any of the changes it did by itself. |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 225 | |
| 226 | When stopping the stream, drivers must notify the entities with |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 227 | :cpp:func:`media_entity_pipeline_stop()`. |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 228 | |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 229 | If multiple calls to :cpp:func:`media_entity_pipeline_start()` have been |
| 230 | made the same number of :cpp:func:`media_entity_pipeline_stop()` calls |
| 231 | are required to stop streaming. |
| 232 | The :c:type:`media_entity`.\ ``pipe`` field is reset to ``NULL`` on the last |
| 233 | nested stop call. |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 234 | |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 235 | Link configuration will fail with ``-EBUSY`` by default if either end of the |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 236 | link is a streaming entity. Links that can be modified while streaming must |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 237 | be marked with the ``MEDIA_LNK_FL_DYNAMIC`` flag. |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 238 | |
| 239 | If other operations need to be disallowed on streaming entities (such as |
| 240 | changing entities configuration parameters) drivers can explicitly check the |
| 241 | media_entity stream_count field to find out if an entity is streaming. This |
| 242 | operation must be done with the media_device graph_mutex held. |
| 243 | |
| 244 | Link validation |
| 245 | ^^^^^^^^^^^^^^^ |
| 246 | |
Mauro Carvalho Chehab | 74604b7 | 2016-07-17 09:18:03 -0300 | [diff] [blame] | 247 | Link validation is performed by :cpp:func:`media_entity_pipeline_start()` |
| 248 | for any entity which has sink pads in the pipeline. The |
| 249 | :c:type:`media_entity`.\ ``link_validate()`` callback is used for that |
| 250 | purpose. In ``link_validate()`` callback, entity driver should check |
| 251 | that the properties of the source pad of the connected entity and its own |
| 252 | sink pad match. It is up to the type of the entity (and in the end, the |
| 253 | properties of the hardware) what matching actually means. |
Mauro Carvalho Chehab | 684ffa2 | 2016-07-17 08:22:23 -0300 | [diff] [blame] | 254 | |
| 255 | Subsystems should facilitate link validation by providing subsystem specific |
| 256 | helper functions to provide easy access for commonly needed information, and |
| 257 | in the end provide a way to use driver-specific callbacks. |
| 258 | |
| 259 | .. kernel-doc:: include/media/media-device.h |
| 260 | |
| 261 | .. kernel-doc:: include/media/media-devnode.h |
| 262 | |
| 263 | .. kernel-doc:: include/media/media-entity.h |