Alexandre Bounine | e15b4d6 | 2011-03-23 16:43:00 -0700 | [diff] [blame] | 1 | The Linux RapidIO Subsystem |
| 2 | |
| 3 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 4 | |
| 5 | The RapidIO standard is a packet-based fabric interconnect standard designed for |
| 6 | use in embedded systems. Development of the RapidIO standard is directed by the |
| 7 | RapidIO Trade Association (RTA). The current version of the RapidIO specification |
| 8 | is publicly available for download from the RTA web-site [1]. |
| 9 | |
| 10 | This document describes the basics of the Linux RapidIO subsystem and provides |
| 11 | information on its major components. |
| 12 | |
| 13 | 1 Overview |
| 14 | ---------- |
| 15 | |
| 16 | Because the RapidIO subsystem follows the Linux device model it is integrated |
| 17 | into the kernel similarly to other buses by defining RapidIO-specific device and |
| 18 | bus types and registering them within the device model. |
| 19 | |
| 20 | The Linux RapidIO subsystem is architecture independent and therefore defines |
| 21 | architecture-specific interfaces that provide support for common RapidIO |
| 22 | subsystem operations. |
| 23 | |
| 24 | 2. Core Components |
| 25 | ------------------ |
| 26 | |
| 27 | A typical RapidIO network is a combination of endpoints and switches. |
| 28 | Each of these components is represented in the subsystem by an associated data |
| 29 | structure. The core logical components of the RapidIO subsystem are defined |
| 30 | in include/linux/rio.h file. |
| 31 | |
| 32 | 2.1 Master Port |
| 33 | |
| 34 | A master port (or mport) is a RapidIO interface controller that is local to the |
| 35 | processor executing the Linux code. A master port generates and receives RapidIO |
| 36 | packets (transactions). In the RapidIO subsystem each master port is represented |
| 37 | by a rio_mport data structure. This structure contains master port specific |
| 38 | resources such as mailboxes and doorbells. The rio_mport also includes a unique |
| 39 | host device ID that is valid when a master port is configured as an enumerating |
| 40 | host. |
| 41 | |
| 42 | RapidIO master ports are serviced by subsystem specific mport device drivers |
| 43 | that provide functionality defined for this subsystem. To provide a hardware |
| 44 | independent interface for RapidIO subsystem operations, rio_mport structure |
| 45 | includes rio_ops data structure which contains pointers to hardware specific |
| 46 | implementations of RapidIO functions. |
| 47 | |
| 48 | 2.2 Device |
| 49 | |
| 50 | A RapidIO device is any endpoint (other than mport) or switch in the network. |
| 51 | All devices are presented in the RapidIO subsystem by corresponding rio_dev data |
| 52 | structure. Devices form one global device list and per-network device lists |
| 53 | (depending on number of available mports and networks). |
| 54 | |
| 55 | 2.3 Switch |
| 56 | |
| 57 | A RapidIO switch is a special class of device that routes packets between its |
| 58 | ports towards their final destination. The packet destination port within a |
| 59 | switch is defined by an internal routing table. A switch is presented in the |
| 60 | RapidIO subsystem by rio_dev data structure expanded by additional rio_switch |
| 61 | data structure, which contains switch specific information such as copy of the |
| 62 | routing table and pointers to switch specific functions. |
| 63 | |
| 64 | The RapidIO subsystem defines the format and initialization method for subsystem |
| 65 | specific switch drivers that are designed to provide hardware-specific |
| 66 | implementation of common switch management routines. |
| 67 | |
| 68 | 2.4 Network |
| 69 | |
| 70 | A RapidIO network is a combination of interconnected endpoint and switch devices. |
| 71 | Each RapidIO network known to the system is represented by corresponding rio_net |
| 72 | data structure. This structure includes lists of all devices and local master |
| 73 | ports that form the same network. It also contains a pointer to the default |
| 74 | master port that is used to communicate with devices within the network. |
| 75 | |
| 76 | 3. Subsystem Initialization |
| 77 | --------------------------- |
| 78 | |
| 79 | In order to initialize the RapidIO subsystem, a platform must initialize and |
| 80 | register at least one master port within the RapidIO network. To register mport |
| 81 | within the subsystem controller driver initialization code calls function |
Alexandre Bounine | 5eeb929 | 2013-05-24 15:55:07 -0700 | [diff] [blame] | 82 | rio_register_mport() for each available master port. |
Alexandre Bounine | e15b4d6 | 2011-03-23 16:43:00 -0700 | [diff] [blame] | 83 | |
Alexandre Bounine | 5eeb929 | 2013-05-24 15:55:07 -0700 | [diff] [blame] | 84 | RapidIO subsystem uses subsys_initcall() or device_initcall() to perform |
| 85 | controller initialization (depending on controller device type). |
| 86 | |
| 87 | After all active master ports are registered with a RapidIO subsystem, |
| 88 | an enumeration and/or discovery routine may be called automatically or |
| 89 | by user-space command. |
Alexandre Bounine | e15b4d6 | 2011-03-23 16:43:00 -0700 | [diff] [blame] | 90 | |
| 91 | 4. Enumeration and Discovery |
| 92 | ---------------------------- |
| 93 | |
Alexandre Bounine | 5eeb929 | 2013-05-24 15:55:07 -0700 | [diff] [blame] | 94 | 4.1 Overview |
| 95 | ------------ |
| 96 | |
| 97 | RapidIO subsystem configuration options allow users to specify enumeration and |
| 98 | discovery methods as statically linked components or loadable modules. |
| 99 | An enumeration/discovery method implementation and available input parameters |
| 100 | define how any given method can be attached to available RapidIO mports: |
| 101 | simply to all available mports OR individually to the specified mport device. |
| 102 | |
| 103 | Depending on selected enumeration/discovery build configuration, there are |
| 104 | several methods to initiate an enumeration and/or discovery process: |
| 105 | |
| 106 | (a) Statically linked enumeration and discovery process can be started |
| 107 | automatically during kernel initialization time using corresponding module |
| 108 | parameters. This was the original method used since introduction of RapidIO |
| 109 | subsystem. Now this method relies on enumerator module parameter which is |
| 110 | 'rio-scan.scan' for existing basic enumeration/discovery method. |
| 111 | When automatic start of enumeration/discovery is used a user has to ensure |
| 112 | that all discovering endpoints are started before the enumerating endpoint |
| 113 | and are waiting for enumeration to be completed. |
| 114 | Configuration option CONFIG_RAPIDIO_DISC_TIMEOUT defines time that discovering |
| 115 | endpoint waits for enumeration to be completed. If the specified timeout |
| 116 | expires the discovery process is terminated without obtaining RapidIO network |
| 117 | information. NOTE: a timed out discovery process may be restarted later using |
| 118 | a user-space command as it is described later if the given endpoint was |
| 119 | enumerated successfully. |
| 120 | |
| 121 | (b) Statically linked enumeration and discovery process can be started by |
| 122 | a command from user space. This initiation method provides more flexibility |
| 123 | for a system startup compared to the option (a) above. After all participating |
| 124 | endpoints have been successfully booted, an enumeration process shall be |
| 125 | started first by issuing a user-space command, after an enumeration is |
| 126 | completed a discovery process can be started on all remaining endpoints. |
| 127 | |
| 128 | (c) Modular enumeration and discovery process can be started by a command from |
| 129 | user space. After an enumeration/discovery module is loaded, a network scan |
| 130 | process can be started by issuing a user-space command. |
| 131 | Similar to the option (b) above, an enumerator has to be started first. |
| 132 | |
| 133 | (d) Modular enumeration and discovery process can be started by a module |
| 134 | initialization routine. In this case an enumerating module shall be loaded |
| 135 | first. |
| 136 | |
| 137 | When a network scan process is started it calls an enumeration or discovery |
| 138 | routine depending on the configured role of a master port: host or agent. |
Alexandre Bounine | e15b4d6 | 2011-03-23 16:43:00 -0700 | [diff] [blame] | 139 | |
| 140 | Enumeration is performed by a master port if it is configured as a host port by |
| 141 | assigning a host device ID greater than or equal to zero. A host device ID is |
| 142 | assigned to a master port through the kernel command line parameter "riohdid=", |
| 143 | or can be configured in a platform-specific manner. If the host device ID for |
| 144 | a specific master port is set to -1, the discovery process will be performed |
| 145 | for it. |
| 146 | |
| 147 | The enumeration and discovery routines use RapidIO maintenance transactions |
| 148 | to access the configuration space of devices. |
| 149 | |
Alexandre Bounine | 5eeb929 | 2013-05-24 15:55:07 -0700 | [diff] [blame] | 150 | 4.2 Automatic Start of Enumeration and Discovery |
| 151 | ------------------------------------------------ |
| 152 | |
| 153 | Automatic enumeration/discovery start method is applicable only to built-in |
| 154 | enumeration/discovery RapidIO configuration selection. To enable automatic |
| 155 | enumeration/discovery start by existing basic enumerator method set use boot |
| 156 | command line parameter "rio-scan.scan=1". |
| 157 | |
| 158 | This configuration requires synchronized start of all RapidIO endpoints that |
| 159 | form a network which will be enumerated/discovered. Discovering endpoints have |
| 160 | to be started before an enumeration starts to ensure that all RapidIO |
| 161 | controllers have been initialized and are ready to be discovered. Configuration |
| 162 | parameter CONFIG_RAPIDIO_DISC_TIMEOUT defines time (in seconds) which |
| 163 | a discovering endpoint will wait for enumeration to be completed. |
| 164 | |
| 165 | When automatic enumeration/discovery start is selected, basic method's |
| 166 | initialization routine calls rio_init_mports() to perform enumeration or |
| 167 | discovery for all known mport devices. |
| 168 | |
| 169 | Depending on RapidIO network size and configuration this automatic |
| 170 | enumeration/discovery start method may be difficult to use due to the |
| 171 | requirement for synchronized start of all endpoints. |
| 172 | |
| 173 | 4.3 User-space Start of Enumeration and Discovery |
| 174 | ------------------------------------------------- |
| 175 | |
| 176 | User-space start of enumeration and discovery can be used with built-in and |
| 177 | modular build configurations. For user-space controlled start RapidIO subsystem |
| 178 | creates the sysfs write-only attribute file '/sys/bus/rapidio/scan'. To initiate |
| 179 | an enumeration or discovery process on specific mport device, a user needs to |
| 180 | write mport_ID (not RapidIO destination ID) into that file. The mport_ID is a |
| 181 | sequential number (0 ... RIO_MAX_MPORTS) assigned during mport device |
| 182 | registration. For example for machine with single RapidIO controller, mport_ID |
| 183 | for that controller always will be 0. |
| 184 | |
| 185 | To initiate RapidIO enumeration/discovery on all available mports a user may |
| 186 | write '-1' (or RIO_MPORT_ANY) into the scan attribute file. |
| 187 | |
| 188 | 4.4 Basic Enumeration Method |
| 189 | ---------------------------- |
| 190 | |
| 191 | This is an original enumeration/discovery method which is available since |
| 192 | first release of RapidIO subsystem code. The enumeration process is |
| 193 | implemented according to the enumeration algorithm outlined in the RapidIO |
| 194 | Interconnect Specification: Annex I [1]. |
| 195 | |
| 196 | This method can be configured as statically linked or loadable module. |
| 197 | The method's single parameter "scan" allows to trigger the enumeration/discovery |
| 198 | process from module initialization routine. |
| 199 | |
| 200 | This enumeration/discovery method can be started only once and does not support |
| 201 | unloading if it is built as a module. |
Alexandre Bounine | e15b4d6 | 2011-03-23 16:43:00 -0700 | [diff] [blame] | 202 | |
| 203 | The enumeration process traverses the network using a recursive depth-first |
| 204 | algorithm. When a new device is found, the enumerator takes ownership of that |
| 205 | device by writing into the Host Device ID Lock CSR. It does this to ensure that |
| 206 | the enumerator has exclusive right to enumerate the device. If device ownership |
| 207 | is successfully acquired, the enumerator allocates a new rio_dev structure and |
| 208 | initializes it according to device capabilities. |
| 209 | |
| 210 | If the device is an endpoint, a unique device ID is assigned to it and its value |
| 211 | is written into the device's Base Device ID CSR. |
| 212 | |
| 213 | If the device is a switch, the enumerator allocates an additional rio_switch |
| 214 | structure to store switch specific information. Then the switch's vendor ID and |
| 215 | device ID are queried against a table of known RapidIO switches. Each switch |
| 216 | table entry contains a pointer to a switch-specific initialization routine that |
| 217 | initializes pointers to the rest of switch specific operations, and performs |
| 218 | hardware initialization if necessary. A RapidIO switch does not have a unique |
| 219 | device ID; it relies on hopcount and routing for device ID of an attached |
| 220 | endpoint if access to its configuration registers is required. If a switch (or |
| 221 | chain of switches) does not have any endpoint (except enumerator) attached to |
| 222 | it, a fake device ID will be assigned to configure a route to that switch. |
| 223 | In the case of a chain of switches without endpoint, one fake device ID is used |
| 224 | to configure a route through the entire chain and switches are differentiated by |
| 225 | their hopcount value. |
| 226 | |
| 227 | For both endpoints and switches the enumerator writes a unique component tag |
| 228 | into device's Component Tag CSR. That unique value is used by the error |
| 229 | management notification mechanism to identify a device that is reporting an |
| 230 | error management event. |
| 231 | |
| 232 | Enumeration beyond a switch is completed by iterating over each active egress |
| 233 | port of that switch. For each active link, a route to a default device ID |
| 234 | (0xFF for 8-bit systems and 0xFFFF for 16-bit systems) is temporarily written |
| 235 | into the routing table. The algorithm recurs by calling itself with hopcount + 1 |
| 236 | and the default device ID in order to access the device on the active port. |
| 237 | |
| 238 | After the host has completed enumeration of the entire network it releases |
| 239 | devices by clearing device ID locks (calls rio_clear_locks()). For each endpoint |
Alexandre Bounine | 088024b | 2011-11-02 13:39:19 -0700 | [diff] [blame] | 240 | in the system, it sets the Discovered bit in the Port General Control CSR |
Alexandre Bounine | e15b4d6 | 2011-03-23 16:43:00 -0700 | [diff] [blame] | 241 | to indicate that enumeration is completed and agents are allowed to execute |
| 242 | passive discovery of the network. |
| 243 | |
| 244 | The discovery process is performed by agents and is similar to the enumeration |
| 245 | process that is described above. However, the discovery process is performed |
| 246 | without changes to the existing routing because agents only gather information |
| 247 | about RapidIO network structure and are building an internal map of discovered |
| 248 | devices. This way each Linux-based component of the RapidIO subsystem has |
| 249 | a complete view of the network. The discovery process can be performed |
| 250 | simultaneously by several agents. After initializing its RapidIO master port |
| 251 | each agent waits for enumeration completion by the host for the configured wait |
| 252 | time period. If this wait time period expires before enumeration is completed, |
| 253 | an agent skips RapidIO discovery and continues with remaining kernel |
| 254 | initialization. |
| 255 | |
Alexandre Bounine | 5eeb929 | 2013-05-24 15:55:07 -0700 | [diff] [blame] | 256 | 4.5 Adding New Enumeration/Discovery Method |
| 257 | ------------------------------------------- |
| 258 | |
| 259 | RapidIO subsystem code organization allows addition of new enumeration/discovery |
| 260 | methods as new configuration options without significant impact to to the core |
| 261 | RapidIO code. |
| 262 | |
| 263 | A new enumeration/discovery method has to be attached to one or more mport |
| 264 | devices before an enumeration/discovery process can be started. Normally, |
| 265 | method's module initialization routine calls rio_register_scan() to attach |
| 266 | an enumerator to a specified mport device (or devices). The basic enumerator |
| 267 | implementation demonstrates this process. |
| 268 | |
Alexandre Bounine | e15b4d6 | 2011-03-23 16:43:00 -0700 | [diff] [blame] | 269 | 5. References |
| 270 | ------------- |
| 271 | |
| 272 | [1] RapidIO Trade Association. RapidIO Interconnect Specifications. |
| 273 | http://www.rapidio.org. |
| 274 | [2] Rapidio TA. Technology Comparisons. |
| 275 | http://www.rapidio.org/education/technology_comparisons/ |
| 276 | [3] RapidIO support for Linux. |
| 277 | http://lwn.net/Articles/139118/ |
| 278 | [4] Matt Porter. RapidIO for Linux. Ottawa Linux Symposium, 2005 |
| 279 | http://www.kernel.org/doc/ols/2005/ols2005v2-pages-43-56.pdf |