Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 1 | The PCI Express Port Bus Driver Guide HOWTO |
| 2 | Tom L Nguyen tom.l.nguyen@intel.com |
| 3 | 11/03/2004 |
| 4 | |
| 5 | 1. About this guide |
| 6 | |
| 7 | This guide describes the basics of the PCI Express Port Bus driver |
| 8 | and provides information on how to enable the service drivers to |
| 9 | register/unregister with the PCI Express Port Bus Driver. |
| 10 | |
| 11 | 2. Copyright 2004 Intel Corporation |
| 12 | |
| 13 | 3. What is the PCI Express Port Bus Driver |
| 14 | |
| 15 | A PCI Express Port is a logical PCI-PCI Bridge structure. There |
| 16 | are two types of PCI Express Port: the Root Port and the Switch |
| 17 | Port. The Root Port originates a PCI Express link from a PCI Express |
| 18 | Root Complex and the Switch Port connects PCI Express links to |
| 19 | internal logical PCI buses. The Switch Port, which has its secondary |
| 20 | bus representing the switch's internal routing logic, is called the |
| 21 | switch's Upstream Port. The switch's Downstream Port is bridging from |
| 22 | switch's internal routing bus to a bus representing the downstream |
| 23 | PCI Express link from the PCI Express Switch. |
| 24 | |
| 25 | A PCI Express Port can provide up to four distinct functions, |
| 26 | referred to in this document as services, depending on its port type. |
| 27 | PCI Express Port's services include native hotplug support (HP), |
| 28 | power management event support (PME), advanced error reporting |
| 29 | support (AER), and virtual channel support (VC). These services may |
| 30 | be handled by a single complex driver or be individually distributed |
| 31 | and handled by corresponding service drivers. |
| 32 | |
| 33 | 4. Why use the PCI Express Port Bus Driver? |
| 34 | |
| 35 | In existing Linux kernels, the Linux Device Driver Model allows a |
| 36 | physical device to be handled by only a single driver. The PCI |
| 37 | Express Port is a PCI-PCI Bridge device with multiple distinct |
| 38 | services. To maintain a clean and simple solution each service |
| 39 | may have its own software service driver. In this case several |
| 40 | service drivers will compete for a single PCI-PCI Bridge device. |
| 41 | For example, if the PCI Express Root Port native hotplug service |
| 42 | driver is loaded first, it claims a PCI-PCI Bridge Root Port. The |
| 43 | kernel therefore does not load other service drivers for that Root |
| 44 | Port. In other words, it is impossible to have multiple service |
| 45 | drivers load and run on a PCI-PCI Bridge device simultaneously |
| 46 | using the current driver model. |
| 47 | |
| 48 | To enable multiple service drivers running simultaneously requires |
| 49 | having a PCI Express Port Bus driver, which manages all populated |
| 50 | PCI Express Ports and distributes all provided service requests |
| 51 | to the corresponding service drivers as required. Some key |
| 52 | advantages of using the PCI Express Port Bus driver are listed below: |
| 53 | |
| 54 | - Allow multiple service drivers to run simultaneously on |
| 55 | a PCI-PCI Bridge Port device. |
| 56 | |
| 57 | - Allow service drivers implemented in an independent |
| 58 | staged approach. |
Randy Dunlap | 4b5ff46 | 2008-03-10 17:16:32 -0700 | [diff] [blame] | 59 | |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 60 | - Allow one service driver to run on multiple PCI-PCI Bridge |
Randy Dunlap | 4b5ff46 | 2008-03-10 17:16:32 -0700 | [diff] [blame] | 61 | Port devices. |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 62 | |
| 63 | - Manage and distribute resources of a PCI-PCI Bridge Port |
| 64 | device to requested service drivers. |
| 65 | |
| 66 | 5. Configuring the PCI Express Port Bus Driver vs. Service Drivers |
| 67 | |
| 68 | 5.1 Including the PCI Express Port Bus Driver Support into the Kernel |
| 69 | |
| 70 | Including the PCI Express Port Bus driver depends on whether the PCI |
| 71 | Express support is included in the kernel config. The kernel will |
| 72 | automatically include the PCI Express Port Bus driver as a kernel |
| 73 | driver when the PCI Express support is enabled in the kernel. |
| 74 | |
| 75 | 5.2 Enabling Service Driver Support |
| 76 | |
| 77 | PCI device drivers are implemented based on Linux Device Driver Model. |
| 78 | All service drivers are PCI device drivers. As discussed above, it is |
| 79 | impossible to load any service driver once the kernel has loaded the |
| 80 | PCI Express Port Bus Driver. To meet the PCI Express Port Bus Driver |
| 81 | Model requires some minimal changes on existing service drivers that |
| 82 | imposes no impact on the functionality of existing service drivers. |
| 83 | |
| 84 | A service driver is required to use the two APIs shown below to |
Randy Dunlap | 4b5ff46 | 2008-03-10 17:16:32 -0700 | [diff] [blame] | 85 | register its service with the PCI Express Port Bus driver (see |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 86 | section 5.2.1 & 5.2.2). It is important that a service driver |
| 87 | initializes the pcie_port_service_driver data structure, included in |
| 88 | header file /include/linux/pcieport_if.h, before calling these APIs. |
| 89 | Failure to do so will result an identity mismatch, which prevents |
| 90 | the PCI Express Port Bus driver from loading a service driver. |
| 91 | |
| 92 | 5.2.1 pcie_port_service_register |
| 93 | |
| 94 | int pcie_port_service_register(struct pcie_port_service_driver *new) |
| 95 | |
| 96 | This API replaces the Linux Driver Model's pci_module_init API. A |
| 97 | service driver should always calls pcie_port_service_register at |
| 98 | module init. Note that after service driver being loaded, calls |
| 99 | such as pci_enable_device(dev) and pci_set_master(dev) are no longer |
| 100 | necessary since these calls are executed by the PCI Port Bus driver. |
| 101 | |
| 102 | 5.2.2 pcie_port_service_unregister |
| 103 | |
| 104 | void pcie_port_service_unregister(struct pcie_port_service_driver *new) |
| 105 | |
| 106 | pcie_port_service_unregister replaces the Linux Driver Model's |
| 107 | pci_unregister_driver. It's always called by service driver when a |
| 108 | module exits. |
| 109 | |
| 110 | 5.2.3 Sample Code |
| 111 | |
| 112 | Below is sample service driver code to initialize the port service |
| 113 | driver data structure. |
| 114 | |
| 115 | static struct pcie_port_service_id service_id[] = { { |
| 116 | .vendor = PCI_ANY_ID, |
| 117 | .device = PCI_ANY_ID, |
| 118 | .port_type = PCIE_RC_PORT, |
| 119 | .service_type = PCIE_PORT_SERVICE_AER, |
| 120 | }, { /* end: all zeroes */ } |
| 121 | }; |
| 122 | |
| 123 | static struct pcie_port_service_driver root_aerdrv = { |
| 124 | .name = (char *)device_name, |
| 125 | .id_table = &service_id[0], |
| 126 | |
| 127 | .probe = aerdrv_load, |
| 128 | .remove = aerdrv_unload, |
| 129 | |
| 130 | .suspend = aerdrv_suspend, |
| 131 | .resume = aerdrv_resume, |
| 132 | }; |
| 133 | |
| 134 | Below is a sample code for registering/unregistering a service |
| 135 | driver. |
| 136 | |
| 137 | static int __init aerdrv_service_init(void) |
| 138 | { |
| 139 | int retval = 0; |
Randy Dunlap | 4b5ff46 | 2008-03-10 17:16:32 -0700 | [diff] [blame] | 140 | |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 141 | retval = pcie_port_service_register(&root_aerdrv); |
| 142 | if (!retval) { |
| 143 | /* |
| 144 | * FIX ME |
| 145 | */ |
| 146 | } |
| 147 | return retval; |
| 148 | } |
| 149 | |
Randy Dunlap | 4b5ff46 | 2008-03-10 17:16:32 -0700 | [diff] [blame] | 150 | static void __exit aerdrv_service_exit(void) |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 151 | { |
| 152 | pcie_port_service_unregister(&root_aerdrv); |
| 153 | } |
| 154 | |
| 155 | module_init(aerdrv_service_init); |
| 156 | module_exit(aerdrv_service_exit); |
| 157 | |
| 158 | 6. Possible Resource Conflicts |
| 159 | |
| 160 | Since all service drivers of a PCI-PCI Bridge Port device are |
| 161 | allowed to run simultaneously, below lists a few of possible resource |
| 162 | conflicts with proposed solutions. |
| 163 | |
| 164 | 6.1 MSI Vector Resource |
| 165 | |
| 166 | The MSI capability structure enables a device software driver to call |
| 167 | pci_enable_msi to request MSI based interrupts. Once MSI interrupts |
| 168 | are enabled on a device, it stays in this mode until a device driver |
| 169 | calls pci_disable_msi to disable MSI interrupts and revert back to |
| 170 | INTx emulation mode. Since service drivers of the same PCI-PCI Bridge |
| 171 | port share the same physical device, if an individual service driver |
| 172 | calls pci_enable_msi/pci_disable_msi it may result unpredictable |
| 173 | behavior. For example, two service drivers run simultaneously on the |
| 174 | same physical Root Port. Both service drivers call pci_enable_msi to |
| 175 | request MSI based interrupts. A service driver may not know whether |
| 176 | any other service drivers have run on this Root Port. If either one |
| 177 | of them calls pci_disable_msi, it puts the other service driver |
Randy Dunlap | 4b5ff46 | 2008-03-10 17:16:32 -0700 | [diff] [blame] | 178 | in a wrong interrupt mode. |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 179 | |
| 180 | To avoid this situation all service drivers are not permitted to |
| 181 | switch interrupt mode on its device. The PCI Express Port Bus driver |
| 182 | is responsible for determining the interrupt mode and this should be |
| 183 | transparent to service drivers. Service drivers need to know only |
| 184 | the vector IRQ assigned to the field irq of struct pcie_device, which |
| 185 | is passed in when the PCI Express Port Bus driver probes each service |
| 186 | driver. Service drivers should use (struct pcie_device*)dev->irq to |
| 187 | call request_irq/free_irq. In addition, the interrupt mode is stored |
| 188 | in the field interrupt_mode of struct pcie_device. |
| 189 | |
| 190 | 6.2 MSI-X Vector Resources |
| 191 | |
| 192 | Similar to the MSI a device driver for an MSI-X capable device can |
| 193 | call pci_enable_msix to request MSI-X interrupts. All service drivers |
| 194 | are not permitted to switch interrupt mode on its device. The PCI |
| 195 | Express Port Bus driver is responsible for determining the interrupt |
| 196 | mode and this should be transparent to service drivers. Any attempt |
| 197 | by service driver to call pci_enable_msix/pci_disable_msix may |
| 198 | result unpredictable behavior. Service drivers should use |
| 199 | (struct pcie_device*)dev->irq and call request_irq/free_irq. |
| 200 | |
| 201 | 6.3 PCI Memory/IO Mapped Regions |
| 202 | |
| 203 | Service drivers for PCI Express Power Management (PME), Advanced |
| 204 | Error Reporting (AER), Hot-Plug (HP) and Virtual Channel (VC) access |
| 205 | PCI configuration space on the PCI Express port. In all cases the |
| 206 | registers accessed are independent of each other. This patch assumes |
| 207 | that all service drivers will be well behaved and not overwrite |
| 208 | other service driver's configuration settings. |
| 209 | |
| 210 | 6.4 PCI Config Registers |
| 211 | |
| 212 | Each service driver runs its PCI config operations on its own |
| 213 | capability structure except the PCI Express capability structure, in |
| 214 | which Root Control register and Device Control register are shared |
| 215 | between PME and AER. This patch assumes that all service drivers |
| 216 | will be well behaved and not overwrite other service driver's |
| 217 | configuration settings. |