Stuart Yoder | c9eda12 | 2015-08-06 20:09:24 -0500 | [diff] [blame] | 1 | Copyright (C) 2015 Freescale Semiconductor Inc. |
| 2 | |
| 3 | DPAA2 (Data Path Acceleration Architecture Gen2) |
| 4 | ------------------------------------------------ |
| 5 | |
| 6 | This document provides an overview of the Freescale DPAA2 architecture |
| 7 | and how it is integrated into the Linux kernel. |
| 8 | |
| 9 | Contents summary |
| 10 | -DPAA2 overview |
| 11 | -Overview of DPAA2 objects |
| 12 | -DPAA2 Linux driver architecture overview |
| 13 | -bus driver |
| 14 | -dprc driver |
| 15 | -allocator |
| 16 | -dpio driver |
| 17 | -Ethernet |
| 18 | -mac |
| 19 | |
| 20 | DPAA2 Overview |
| 21 | -------------- |
| 22 | |
| 23 | DPAA2 is a hardware architecture designed for high-speeed network |
| 24 | packet processing. DPAA2 consists of sophisticated mechanisms for |
| 25 | processing Ethernet packets, queue management, buffer management, |
| 26 | autonomous L2 switching, virtual Ethernet bridging, and accelerator |
| 27 | (e.g. crypto) sharing. |
| 28 | |
| 29 | A DPAA2 hardware component called the Management Complex (or MC) manages the |
| 30 | DPAA2 hardware resources. The MC provides an object-based abstraction for |
| 31 | software drivers to use the DPAA2 hardware. |
| 32 | |
| 33 | The MC uses DPAA2 hardware resources such as queues, buffer pools, and |
| 34 | network ports to create functional objects/devices such as network |
| 35 | interfaces, an L2 switch, or accelerator instances. |
| 36 | |
| 37 | The MC provides memory-mapped I/O command interfaces (MC portals) |
| 38 | which DPAA2 software drivers use to operate on DPAA2 objects: |
| 39 | |
| 40 | +--------------------------------------+ |
| 41 | | OS | |
| 42 | | DPAA2 drivers | |
| 43 | | | | |
| 44 | +-----------------------------|--------+ |
| 45 | | |
| 46 | | (create,discover,connect |
| 47 | | config,use,destroy) |
| 48 | | |
| 49 | DPAA2 | |
| 50 | +------------------------| mc portal |-+ |
| 51 | | | | |
| 52 | | +- - - - - - - - - - - - -V- - -+ | |
| 53 | | | | | |
| 54 | | | Management Complex (MC) | | |
| 55 | | | | | |
| 56 | | +- - - - - - - - - - - - - - - -+ | |
| 57 | | | |
| 58 | | Hardware Hardware | |
| 59 | | Resources Objects | |
| 60 | | --------- ------- | |
| 61 | | -queues -DPRC | |
| 62 | | -buffer pools -DPMCP | |
| 63 | | -Eth MACs/ports -DPIO | |
| 64 | | -network interface -DPNI | |
| 65 | | profiles -DPMAC | |
| 66 | | -queue portals -DPBP | |
| 67 | | -MC portals ... | |
| 68 | | ... | |
| 69 | | | |
| 70 | +--------------------------------------+ |
| 71 | |
| 72 | The MC mediates operations such as create, discover, |
| 73 | connect, configuration, and destroy. Fast-path operations |
| 74 | on data, such as packet transmit/receive, are not mediated by |
| 75 | the MC and are done directly using memory mapped regions in |
| 76 | DPIO objects. |
| 77 | |
| 78 | Overview of DPAA2 Objects |
| 79 | ------------------------- |
| 80 | The section provides a brief overview of some key objects |
| 81 | in the DPAA2 hardware. A simple scenario is described illustrating |
| 82 | the objects involved in creating a network interfaces. |
| 83 | |
| 84 | -DPRC (Datapath Resource Container) |
| 85 | |
| 86 | A DPRC is an container object that holds all the other |
| 87 | types of DPAA2 objects. In the example diagram below there |
| 88 | are 8 objects of 5 types (DPMCP, DPIO, DPBP, DPNI, and DPMAC) |
| 89 | in the container. |
| 90 | |
| 91 | +---------------------------------------------------------+ |
| 92 | | DPRC | |
| 93 | | | |
| 94 | | +-------+ +-------+ +-------+ +-------+ +-------+ | |
| 95 | | | DPMCP | | DPIO | | DPBP | | DPNI | | DPMAC | | |
| 96 | | +-------+ +-------+ +-------+ +---+---+ +---+---+ | |
| 97 | | | DPMCP | | DPIO | | |
| 98 | | +-------+ +-------+ | |
| 99 | | | DPMCP | | |
| 100 | | +-------+ | |
| 101 | | | |
| 102 | +---------------------------------------------------------+ |
| 103 | |
| 104 | From the point of view of an OS, a DPRC is bus-like. Like |
| 105 | a plug-and-play bus, such as PCI, DPRC commands can be used to |
| 106 | enumerate the contents of the DPRC, discover the hardware |
| 107 | objects present (including mappable regions and interrupts). |
| 108 | |
| 109 | dprc.1 (bus) |
| 110 | | |
| 111 | +--+--------+-------+-------+-------+ |
| 112 | | | | | | |
| 113 | dpmcp.1 dpio.1 dpbp.1 dpni.1 dpmac.1 |
| 114 | dpmcp.2 dpio.2 |
| 115 | dpmcp.3 |
| 116 | |
| 117 | Hardware objects can be created and destroyed dynamically, providing |
| 118 | the ability to hot plug/unplug objects in and out of the DPRC. |
| 119 | |
| 120 | A DPRC has a mappable mmio region (an MC portal) that can be used |
| 121 | to send MC commands. It has an interrupt for status events (like |
| 122 | hotplug). |
| 123 | |
| 124 | All objects in a container share the same hardware "isolation context". |
| 125 | This means that with respect to an IOMMU the isolation granularity |
| 126 | is at the DPRC (container) level, not at the individual object |
| 127 | level. |
| 128 | |
| 129 | DPRCs can be defined statically and populated with objects |
| 130 | via a config file passed to the MC when firmware starts |
| 131 | it. There is also a Linux user space tool called "restool" |
| 132 | that can be used to create/destroy containers and objects |
| 133 | dynamically. |
| 134 | |
| 135 | -DPAA2 Objects for an Ethernet Network Interface |
| 136 | |
| 137 | A typical Ethernet NIC is monolithic-- the NIC device contains TX/RX |
| 138 | queuing mechanisms, configuration mechanisms, buffer management, |
| 139 | physical ports, and interrupts. DPAA2 uses a more granular approach |
| 140 | utilizing multiple hardware objects. Each object has specialized |
| 141 | functions, and are used together by software to provide Ethernet network |
| 142 | interface functionality. This approach provides efficient use of finite |
| 143 | hardware resources, flexibility, and performance advantages. |
| 144 | |
| 145 | The diagram below shows the objects needed for a simple |
| 146 | network interface configuration on a system with 2 CPUs. |
| 147 | |
| 148 | +---+---+ +---+---+ |
| 149 | CPU0 CPU1 |
| 150 | +---+---+ +---+---+ |
| 151 | | | |
| 152 | +---+---+ +---+---+ |
| 153 | DPIO DPIO |
| 154 | +---+---+ +---+---+ |
| 155 | \ / |
| 156 | \ / |
| 157 | \ / |
| 158 | +---+---+ |
| 159 | DPNI --- DPBP,DPMCP |
| 160 | +---+---+ |
| 161 | | |
| 162 | | |
| 163 | +---+---+ |
| 164 | DPMAC |
| 165 | +---+---+ |
| 166 | | |
| 167 | port/PHY |
| 168 | |
| 169 | Below the objects are described. For each object a brief description |
| 170 | is provided along with a summary of the kinds of operations the object |
| 171 | supports and a summary of key resources of the object (mmio regions |
| 172 | and irqs). |
| 173 | |
| 174 | -DPMAC (Datapath Ethernet MAC): represents an Ethernet MAC, a |
| 175 | hardware device that connects to an Ethernet PHY and allows |
| 176 | physical transmission and reception of Ethernet frames. |
| 177 | -mmio regions: none |
| 178 | -irqs: dpni link change |
| 179 | -commands: set link up/down, link config, get stats, |
| 180 | irq config, enable, reset |
| 181 | |
| 182 | -DPNI (Datapath Network Interface): contains TX/RX queues, |
| 183 | network interface configuration, and rx buffer pool configuration |
| 184 | mechanisms. |
| 185 | -mmio regions: none |
| 186 | -irqs: link state |
| 187 | -commands: port config, offload config, queue config, |
| 188 | parse/classify config, irq config, enable, reset |
| 189 | |
| 190 | -DPIO (Datapath I/O): provides interfaces to enqueue and dequeue |
| 191 | packets and do hardware buffer pool management operations. For |
| 192 | optimum performance there is typically DPIO per CPU. This allows |
| 193 | each CPU to perform simultaneous enqueue/dequeue operations. |
| 194 | -mmio regions: queue operations, buffer mgmt |
| 195 | -irqs: data availability, congestion notification, buffer |
| 196 | pool depletion |
| 197 | -commands: irq config, enable, reset |
| 198 | |
| 199 | -DPBP (Datapath Buffer Pool): represents a hardware buffer |
| 200 | pool. |
| 201 | -mmio regions: none |
| 202 | -irqs: none |
| 203 | -commands: enable, reset |
| 204 | |
| 205 | -DPMCP (Datapath MC Portal): provides an MC command portal. |
| 206 | Used by drivers to send commands to the MC to manage |
| 207 | objects. |
| 208 | -mmio regions: MC command portal |
| 209 | -irqs: command completion |
| 210 | -commands: irq config, enable, reset |
| 211 | |
| 212 | Object Connections |
| 213 | ------------------ |
| 214 | Some objects have explicit relationships that must |
| 215 | be configured: |
| 216 | |
| 217 | -DPNI <--> DPMAC |
| 218 | -DPNI <--> DPNI |
| 219 | -DPNI <--> L2-switch-port |
| 220 | A DPNI must be connected to something such as a DPMAC, |
| 221 | another DPNI, or L2 switch port. The DPNI connection |
| 222 | is made via a DPRC command. |
| 223 | |
| 224 | +-------+ +-------+ |
| 225 | | DPNI | | DPMAC | |
| 226 | +---+---+ +---+---+ |
| 227 | | | |
| 228 | +==========+ |
| 229 | |
| 230 | -DPNI <--> DPBP |
| 231 | A network interface requires a 'buffer pool' (DPBP |
| 232 | object) which provides a list of pointers to memory |
| 233 | where received Ethernet data is to be copied. The |
| 234 | Ethernet driver configures the DPBPs associated with |
| 235 | the network interface. |
| 236 | |
| 237 | Interrupts |
| 238 | ---------- |
| 239 | All interrupts generated by DPAA2 objects are message |
| 240 | interrupts. At the hardware level message interrupts |
| 241 | generated by devices will normally have 3 components-- |
| 242 | 1) a non-spoofable 'device-id' expressed on the hardware |
| 243 | bus, 2) an address, 3) a data value. |
| 244 | |
| 245 | In the case of DPAA2 devices/objects, all objects in the |
| 246 | same container/DPRC share the same 'device-id'. |
| 247 | For ARM-based SoC this is the same as the stream ID. |
| 248 | |
| 249 | |
| 250 | DPAA2 Linux Driver Overview |
| 251 | --------------------------- |
| 252 | |
| 253 | This section provides an overview of the Linux kernel drivers for |
| 254 | DPAA2-- 1) the bus driver and associated "DPAA2 infrastructure" |
| 255 | drivers and 2) functional object drivers (such as Ethernet). |
| 256 | |
| 257 | As described previously, a DPRC is a container that holds the other |
| 258 | types of DPAA2 objects. It is functionally similar to a plug-and-play |
| 259 | bus controller. |
| 260 | |
| 261 | Each object in the DPRC is a Linux "device" and is bound to a driver. |
| 262 | The diagram below shows the Linux drivers involved in a networking |
| 263 | scenario and the objects bound to each driver. A brief description |
| 264 | of each driver follows. |
| 265 | |
| 266 | +------------+ |
| 267 | | OS Network | |
| 268 | | Stack | |
| 269 | +------------+ +------------+ |
| 270 | | Allocator |. . . . . . . | Ethernet | |
| 271 | |(dpmcp,dpbp)| | (dpni) | |
| 272 | +-.----------+ +---+---+----+ |
| 273 | . . ^ | |
| 274 | . . <data avail, | |<enqueue, |
| 275 | . . tx confirm> | | dequeue> |
| 276 | +-------------+ . | | |
| 277 | | DPRC driver | . +---+---V----+ +---------+ |
| 278 | | (dprc) | . . . . . .| DPIO driver| | MAC | |
| 279 | +----------+--+ | (dpio) | | (dpmac) | |
| 280 | | +------+-----+ +-----+---+ |
| 281 | |<dev add/remove> | | |
| 282 | | | | |
| 283 | +----+--------------+ | +--+---+ |
| 284 | | mc-bus driver | | | PHY | |
| 285 | | | | |driver| |
| 286 | | /fsl-mc@80c000000 | | +--+---+ |
| 287 | +-------------------+ | | |
| 288 | | | |
| 289 | ================================ HARDWARE =========|=================|====== |
| 290 | DPIO | |
| 291 | | | |
| 292 | DPNI---DPBP | |
| 293 | | | |
| 294 | DPMAC | |
| 295 | | | |
| 296 | PHY ---------------+ |
| 297 | ===================================================|======================== |
| 298 | |
| 299 | A brief description of each driver is provided below. |
| 300 | |
| 301 | mc-bus driver |
| 302 | ------------- |
| 303 | The mc-bus driver is a platform driver and is probed from an |
| 304 | "/fsl-mc@xxxx" node in the device tree passed in by boot firmware. |
| 305 | It is responsible for bootstrapping the DPAA2 kernel infrastructure. |
| 306 | Key functions include: |
| 307 | -registering a new bus type named "fsl-mc" with the kernel, |
| 308 | and implementing bus call-backs (e.g. match/uevent/dev_groups) |
| 309 | -implemeting APIs for DPAA2 driver registration and for device |
| 310 | add/remove |
| 311 | -creates an MSI irq domain |
| 312 | -do a device add of the 'root' DPRC device, which is needed |
| 313 | to bootstrap things |
| 314 | |
| 315 | DPRC driver |
| 316 | ----------- |
| 317 | The dprc-driver is bound DPRC objects and does runtime management |
| 318 | of a bus instance. It performs the initial bus scan of the DPRC |
| 319 | and handles interrupts for container events such as hot plug. |
| 320 | |
| 321 | Allocator |
| 322 | ---------- |
| 323 | Certain objects such as DPMCP and DPBP are generic and fungible, |
| 324 | and are intended to be used by other drivers. For example, |
| 325 | the DPAA2 Ethernet driver needs: |
| 326 | -DPMCPs to send MC commands, to configure network interfaces |
| 327 | -DPBPs for network buffer pools |
| 328 | |
| 329 | The allocator driver registers for these allocatable object types |
| 330 | and those objects are bound to the allocator when the bus is probed. |
| 331 | The allocator maintains a pool of objects that are available for |
| 332 | allocation by other DPAA2 drivers. |
| 333 | |
| 334 | DPIO driver |
| 335 | ----------- |
| 336 | The DPIO driver is bound to DPIO objects and provides services that allow |
| 337 | other drivers such as the Ethernet driver to receive and transmit data. |
| 338 | Key services include: |
| 339 | -data availability notifications |
| 340 | -hardware queuing operations (enqueue and dequeue of data) |
| 341 | -hardware buffer pool management |
| 342 | |
| 343 | There is typically one DPIO object per physical CPU for optimum |
| 344 | performance, allowing each CPU to simultaneously enqueue |
| 345 | and dequeue data. |
| 346 | |
| 347 | The DPIO driver operates on behalf of all DPAA2 drivers |
| 348 | active in the kernel-- Ethernet, crypto, compression, |
| 349 | etc. |
| 350 | |
| 351 | Ethernet |
| 352 | -------- |
| 353 | The Ethernet driver is bound to a DPNI and implements the kernel |
| 354 | interfaces needed to connect the DPAA2 network interface to |
| 355 | the network stack. |
| 356 | |
| 357 | Each DPNI corresponds to a Linux network interface. |
| 358 | |
| 359 | MAC driver |
| 360 | ---------- |
| 361 | An Ethernet PHY is an off-chip, board specific component and is managed |
| 362 | by the appropriate PHY driver via an mdio bus. The MAC driver |
| 363 | plays a role of being a proxy between the PHY driver and the |
| 364 | MC. It does this proxy via the MC commands to a DPMAC object. |