| Hal Rosenstock | 617b586 | 2005-07-27 11:45:47 -0700 | [diff] [blame] | 1 | INFINIBAND MIDLAYER LOCKING |
| 2 | |
| 3 | This guide is an attempt to make explicit the locking assumptions |
| 4 | made by the InfiniBand midlayer. It describes the requirements on |
| 5 | both low-level drivers that sit below the midlayer and upper level |
| 6 | protocols that use the midlayer. |
| 7 | |
| 8 | Sleeping and interrupt context |
| 9 | |
| 10 | With the following exceptions, a low-level driver implementation of |
| 11 | all of the methods in struct ib_device may sleep. The exceptions |
| 12 | are any methods from the list: |
| 13 | |
| 14 | create_ah |
| 15 | modify_ah |
| 16 | query_ah |
| 17 | destroy_ah |
| 18 | bind_mw |
| 19 | post_send |
| 20 | post_recv |
| 21 | poll_cq |
| 22 | req_notify_cq |
| 23 | map_phys_fmr |
| 24 | |
| 25 | which may not sleep and must be callable from any context. |
| 26 | |
| 27 | The corresponding functions exported to upper level protocol |
| 28 | consumers: |
| 29 | |
| 30 | ib_create_ah |
| 31 | ib_modify_ah |
| 32 | ib_query_ah |
| 33 | ib_destroy_ah |
| 34 | ib_bind_mw |
| 35 | ib_post_send |
| 36 | ib_post_recv |
| 37 | ib_req_notify_cq |
| 38 | ib_map_phys_fmr |
| 39 | |
| 40 | are therefore safe to call from any context. |
| 41 | |
| 42 | In addition, the function |
| 43 | |
| 44 | ib_dispatch_event |
| 45 | |
| 46 | used by low-level drivers to dispatch asynchronous events through |
| 47 | the midlayer is also safe to call from any context. |
| 48 | |
| 49 | Reentrancy |
| 50 | |
| 51 | All of the methods in struct ib_device exported by a low-level |
| 52 | driver must be fully reentrant. The low-level driver is required to |
| 53 | perform all synchronization necessary to maintain consistency, even |
| 54 | if multiple function calls using the same object are run |
| 55 | simultaneously. |
| 56 | |
| 57 | The IB midlayer does not perform any serialization of function calls. |
| 58 | |
| 59 | Because low-level drivers are reentrant, upper level protocol |
| 60 | consumers are not required to perform any serialization. However, |
| 61 | some serialization may be required to get sensible results. For |
| 62 | example, a consumer may safely call ib_poll_cq() on multiple CPUs |
| 63 | simultaneously. However, the ordering of the work completion |
| 64 | information between different calls of ib_poll_cq() is not defined. |
| 65 | |
| 66 | Callbacks |
| 67 | |
| 68 | A low-level driver must not perform a callback directly from the |
| 69 | same callchain as an ib_device method call. For example, it is not |
| 70 | allowed for a low-level driver to call a consumer's completion event |
| 71 | handler directly from its post_send method. Instead, the low-level |
| 72 | driver should defer this callback by, for example, scheduling a |
| 73 | tasklet to perform the callback. |
| 74 | |
| 75 | The low-level driver is responsible for ensuring that multiple |
| 76 | completion event handlers for the same CQ are not called |
| 77 | simultaneously. The driver must guarantee that only one CQ event |
| 78 | handler for a given CQ is running at a time. In other words, the |
| 79 | following situation is not allowed: |
| 80 | |
| 81 | CPU1 CPU2 |
| 82 | |
| 83 | low-level driver -> |
| 84 | consumer CQ event callback: |
| 85 | /* ... */ |
| 86 | ib_req_notify_cq(cq, ...); |
| 87 | low-level driver -> |
| 88 | /* ... */ consumer CQ event callback: |
| 89 | /* ... */ |
| 90 | return from CQ event handler |
| 91 | |
| 92 | The context in which completion event and asynchronous event |
| 93 | callbacks run is not defined. Depending on the low-level driver, it |
| 94 | may be process context, softirq context, or interrupt context. |
| 95 | Upper level protocol consumers may not sleep in a callback. |
| 96 | |
| 97 | Hot-plug |
| 98 | |
| 99 | A low-level driver announces that a device is ready for use by |
| 100 | consumers when it calls ib_register_device(), all initialization |
| 101 | must be complete before this call. The device must remain usable |
| 102 | until the driver's call to ib_unregister_device() has returned. |
| 103 | |
| 104 | A low-level driver must call ib_register_device() and |
| 105 | ib_unregister_device() from process context. It must not hold any |
| 106 | semaphores that could cause deadlock if a consumer calls back into |
| 107 | the driver across these calls. |
| 108 | |
| 109 | An upper level protocol consumer may begin using an IB device as |
| 110 | soon as the add method of its struct ib_client is called for that |
| 111 | device. A consumer must finish all cleanup and free all resources |
| 112 | relating to a device before returning from the remove method. |
| 113 | |
| 114 | A consumer is permitted to sleep in its add and remove methods. |