Andy Grover | ce87685 | 2014-10-01 16:07:04 -0700 | [diff] [blame] | 1 | Contents: |
| 2 | |
| 3 | 1) TCM Userspace Design |
| 4 | a) Background |
| 5 | b) Benefits |
| 6 | c) Design constraints |
| 7 | d) Implementation overview |
| 8 | i. Mailbox |
| 9 | ii. Command ring |
| 10 | iii. Data Area |
| 11 | e) Device discovery |
| 12 | f) Device events |
| 13 | g) Other contingencies |
| 14 | 2) Writing a user pass-through handler |
| 15 | a) Discovering and configuring TCMU uio devices |
| 16 | b) Waiting for events on the device(s) |
| 17 | c) Managing the command ring |
Andy Grover | 9c1cd1b | 2015-05-19 14:44:39 -0700 | [diff] [blame] | 18 | 3) A final note |
Andy Grover | ce87685 | 2014-10-01 16:07:04 -0700 | [diff] [blame] | 19 | |
| 20 | |
| 21 | TCM Userspace Design |
| 22 | -------------------- |
| 23 | |
| 24 | TCM is another name for LIO, an in-kernel iSCSI target (server). |
| 25 | Existing TCM targets run in the kernel. TCMU (TCM in Userspace) |
| 26 | allows userspace programs to be written which act as iSCSI targets. |
| 27 | This document describes the design. |
| 28 | |
| 29 | The existing kernel provides modules for different SCSI transport |
| 30 | protocols. TCM also modularizes the data storage. There are existing |
| 31 | modules for file, block device, RAM or using another SCSI device as |
| 32 | storage. These are called "backstores" or "storage engines". These |
| 33 | built-in modules are implemented entirely as kernel code. |
| 34 | |
| 35 | Background: |
| 36 | |
| 37 | In addition to modularizing the transport protocol used for carrying |
| 38 | SCSI commands ("fabrics"), the Linux kernel target, LIO, also modularizes |
| 39 | the actual data storage as well. These are referred to as "backstores" |
| 40 | or "storage engines". The target comes with backstores that allow a |
| 41 | file, a block device, RAM, or another SCSI device to be used for the |
| 42 | local storage needed for the exported SCSI LUN. Like the rest of LIO, |
| 43 | these are implemented entirely as kernel code. |
| 44 | |
| 45 | These backstores cover the most common use cases, but not all. One new |
| 46 | use case that other non-kernel target solutions, such as tgt, are able |
| 47 | to support is using Gluster's GLFS or Ceph's RBD as a backstore. The |
| 48 | target then serves as a translator, allowing initiators to store data |
| 49 | in these non-traditional networked storage systems, while still only |
| 50 | using standard protocols themselves. |
| 51 | |
| 52 | If the target is a userspace process, supporting these is easy. tgt, |
| 53 | for example, needs only a small adapter module for each, because the |
| 54 | modules just use the available userspace libraries for RBD and GLFS. |
| 55 | |
| 56 | Adding support for these backstores in LIO is considerably more |
| 57 | difficult, because LIO is entirely kernel code. Instead of undertaking |
| 58 | the significant work to port the GLFS or RBD APIs and protocols to the |
| 59 | kernel, another approach is to create a userspace pass-through |
| 60 | backstore for LIO, "TCMU". |
| 61 | |
| 62 | |
| 63 | Benefits: |
| 64 | |
| 65 | In addition to allowing relatively easy support for RBD and GLFS, TCMU |
| 66 | will also allow easier development of new backstores. TCMU combines |
| 67 | with the LIO loopback fabric to become something similar to FUSE |
| 68 | (Filesystem in Userspace), but at the SCSI layer instead of the |
| 69 | filesystem layer. A SUSE, if you will. |
| 70 | |
| 71 | The disadvantage is there are more distinct components to configure, and |
| 72 | potentially to malfunction. This is unavoidable, but hopefully not |
| 73 | fatal if we're careful to keep things as simple as possible. |
| 74 | |
| 75 | Design constraints: |
| 76 | |
| 77 | - Good performance: high throughput, low latency |
| 78 | - Cleanly handle if userspace: |
| 79 | 1) never attaches |
| 80 | 2) hangs |
| 81 | 3) dies |
| 82 | 4) misbehaves |
| 83 | - Allow future flexibility in user & kernel implementations |
| 84 | - Be reasonably memory-efficient |
| 85 | - Simple to configure & run |
| 86 | - Simple to write a userspace backend |
| 87 | |
| 88 | |
| 89 | Implementation overview: |
| 90 | |
| 91 | The core of the TCMU interface is a memory region that is shared |
| 92 | between kernel and userspace. Within this region is: a control area |
| 93 | (mailbox); a lockless producer/consumer circular buffer for commands |
| 94 | to be passed up, and status returned; and an in/out data buffer area. |
| 95 | |
| 96 | TCMU uses the pre-existing UIO subsystem. UIO allows device driver |
| 97 | development in userspace, and this is conceptually very close to the |
| 98 | TCMU use case, except instead of a physical device, TCMU implements a |
| 99 | memory-mapped layout designed for SCSI commands. Using UIO also |
| 100 | benefits TCMU by handling device introspection (e.g. a way for |
| 101 | userspace to determine how large the shared region is) and signaling |
| 102 | mechanisms in both directions. |
| 103 | |
| 104 | There are no embedded pointers in the memory region. Everything is |
| 105 | expressed as an offset from the region's starting address. This allows |
| 106 | the ring to still work if the user process dies and is restarted with |
| 107 | the region mapped at a different virtual address. |
| 108 | |
| 109 | See target_core_user.h for the struct definitions. |
| 110 | |
| 111 | The Mailbox: |
| 112 | |
| 113 | The mailbox is always at the start of the shared memory region, and |
| 114 | contains a version, details about the starting offset and size of the |
| 115 | command ring, and head and tail pointers to be used by the kernel and |
| 116 | userspace (respectively) to put commands on the ring, and indicate |
| 117 | when the commands are completed. |
| 118 | |
| 119 | version - 1 (userspace should abort if otherwise) |
| 120 | flags - none yet defined. |
| 121 | cmdr_off - The offset of the start of the command ring from the start |
| 122 | of the memory region, to account for the mailbox size. |
| 123 | cmdr_size - The size of the command ring. This does *not* need to be a |
| 124 | power of two. |
| 125 | cmd_head - Modified by the kernel to indicate when a command has been |
| 126 | placed on the ring. |
| 127 | cmd_tail - Modified by userspace to indicate when it has completed |
| 128 | processing of a command. |
| 129 | |
| 130 | The Command Ring: |
| 131 | |
| 132 | Commands are placed on the ring by the kernel incrementing |
| 133 | mailbox.cmd_head by the size of the command, modulo cmdr_size, and |
| 134 | then signaling userspace via uio_event_notify(). Once the command is |
| 135 | completed, userspace updates mailbox.cmd_tail in the same way and |
| 136 | signals the kernel via a 4-byte write(). When cmd_head equals |
| 137 | cmd_tail, the ring is empty -- no commands are currently waiting to be |
| 138 | processed by userspace. |
| 139 | |
Andy Grover | 0ad46af | 2015-04-14 17:30:04 -0700 | [diff] [blame] | 140 | TCMU commands are 8-byte aligned. They start with a common header |
| 141 | containing "len_op", a 32-bit value that stores the length, as well as |
| 142 | the opcode in the lowest unused bits. It also contains cmd_id and |
| 143 | flags fields for setting by the kernel (kflags) and userspace |
| 144 | (uflags). |
Andy Grover | ce87685 | 2014-10-01 16:07:04 -0700 | [diff] [blame] | 145 | |
Andy Grover | 0ad46af | 2015-04-14 17:30:04 -0700 | [diff] [blame] | 146 | Currently only two opcodes are defined, TCMU_OP_CMD and TCMU_OP_PAD. |
Andy Grover | ce87685 | 2014-10-01 16:07:04 -0700 | [diff] [blame] | 147 | |
Andy Grover | 0ad46af | 2015-04-14 17:30:04 -0700 | [diff] [blame] | 148 | When the opcode is CMD, the entry in the command ring is a struct |
| 149 | tcmu_cmd_entry. Userspace finds the SCSI CDB (Command Data Block) via |
| 150 | tcmu_cmd_entry.req.cdb_off. This is an offset from the start of the |
| 151 | overall shared memory region, not the entry. The data in/out buffers |
| 152 | are accessible via tht req.iov[] array. iov_cnt contains the number of |
| 153 | entries in iov[] needed to describe either the Data-In or Data-Out |
| 154 | buffers. For bidirectional commands, iov_cnt specifies how many iovec |
| 155 | entries cover the Data-Out area, and iov_bidi_count specifies how many |
| 156 | iovec entries immediately after that in iov[] cover the Data-In |
| 157 | area. Just like other fields, iov.iov_base is an offset from the start |
| 158 | of the region. |
Andy Grover | ce87685 | 2014-10-01 16:07:04 -0700 | [diff] [blame] | 159 | |
| 160 | When completing a command, userspace sets rsp.scsi_status, and |
| 161 | rsp.sense_buffer if necessary. Userspace then increments |
| 162 | mailbox.cmd_tail by entry.hdr.length (mod cmdr_size) and signals the |
| 163 | kernel via the UIO method, a 4-byte write to the file descriptor. |
| 164 | |
Andy Grover | 0ad46af | 2015-04-14 17:30:04 -0700 | [diff] [blame] | 165 | When the opcode is PAD, userspace only updates cmd_tail as above -- |
| 166 | it's a no-op. (The kernel inserts PAD entries to ensure each CMD entry |
| 167 | is contiguous within the command ring.) |
| 168 | |
| 169 | More opcodes may be added in the future. If userspace encounters an |
| 170 | opcode it does not handle, it must set UNKNOWN_OP bit (bit 0) in |
| 171 | hdr.uflags, update cmd_tail, and proceed with processing additional |
| 172 | commands, if any. |
| 173 | |
Andy Grover | ce87685 | 2014-10-01 16:07:04 -0700 | [diff] [blame] | 174 | The Data Area: |
| 175 | |
| 176 | This is shared-memory space after the command ring. The organization |
| 177 | of this area is not defined in the TCMU interface, and userspace |
| 178 | should access only the parts referenced by pending iovs. |
| 179 | |
| 180 | |
| 181 | Device Discovery: |
| 182 | |
| 183 | Other devices may be using UIO besides TCMU. Unrelated user processes |
| 184 | may also be handling different sets of TCMU devices. TCMU userspace |
| 185 | processes must find their devices by scanning sysfs |
| 186 | class/uio/uio*/name. For TCMU devices, these names will be of the |
| 187 | format: |
| 188 | |
| 189 | tcm-user/<hba_num>/<device_name>/<subtype>/<path> |
| 190 | |
| 191 | where "tcm-user" is common for all TCMU-backed UIO devices. <hba_num> |
| 192 | and <device_name> allow userspace to find the device's path in the |
| 193 | kernel target's configfs tree. Assuming the usual mount point, it is |
| 194 | found at: |
| 195 | |
| 196 | /sys/kernel/config/target/core/user_<hba_num>/<device_name> |
| 197 | |
| 198 | This location contains attributes such as "hw_block_size", that |
| 199 | userspace needs to know for correct operation. |
| 200 | |
| 201 | <subtype> will be a userspace-process-unique string to identify the |
| 202 | TCMU device as expecting to be backed by a certain handler, and <path> |
| 203 | will be an additional handler-specific string for the user process to |
| 204 | configure the device, if needed. The name cannot contain ':', due to |
| 205 | LIO limitations. |
| 206 | |
| 207 | For all devices so discovered, the user handler opens /dev/uioX and |
| 208 | calls mmap(): |
| 209 | |
| 210 | mmap(NULL, size, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0) |
| 211 | |
| 212 | where size must be equal to the value read from |
| 213 | /sys/class/uio/uioX/maps/map0/size. |
| 214 | |
| 215 | |
| 216 | Device Events: |
| 217 | |
| 218 | If a new device is added or removed, a notification will be broadcast |
| 219 | over netlink, using a generic netlink family name of "TCM-USER" and a |
| 220 | multicast group named "config". This will include the UIO name as |
| 221 | described in the previous section, as well as the UIO minor |
| 222 | number. This should allow userspace to identify both the UIO device and |
| 223 | the LIO device, so that after determining the device is supported |
| 224 | (based on subtype) it can take the appropriate action. |
| 225 | |
| 226 | |
| 227 | Other contingencies: |
| 228 | |
| 229 | Userspace handler process never attaches: |
| 230 | |
| 231 | - TCMU will post commands, and then abort them after a timeout period |
| 232 | (30 seconds.) |
| 233 | |
| 234 | Userspace handler process is killed: |
| 235 | |
| 236 | - It is still possible to restart and re-connect to TCMU |
| 237 | devices. Command ring is preserved. However, after the timeout period, |
| 238 | the kernel will abort pending tasks. |
| 239 | |
| 240 | Userspace handler process hangs: |
| 241 | |
| 242 | - The kernel will abort pending tasks after a timeout period. |
| 243 | |
| 244 | Userspace handler process is malicious: |
| 245 | |
| 246 | - The process can trivially break the handling of devices it controls, |
| 247 | but should not be able to access kernel memory outside its shared |
| 248 | memory areas. |
| 249 | |
| 250 | |
| 251 | Writing a user pass-through handler (with example code) |
| 252 | ------------------------------------------------------- |
| 253 | |
| 254 | A user process handing a TCMU device must support the following: |
| 255 | |
| 256 | a) Discovering and configuring TCMU uio devices |
| 257 | b) Waiting for events on the device(s) |
| 258 | c) Managing the command ring: Parsing operations and commands, |
| 259 | performing work as needed, setting response fields (scsi_status and |
| 260 | possibly sense_buffer), updating cmd_tail, and notifying the kernel |
| 261 | that work has been finished |
| 262 | |
| 263 | First, consider instead writing a plugin for tcmu-runner. tcmu-runner |
| 264 | implements all of this, and provides a higher-level API for plugin |
| 265 | authors. |
| 266 | |
| 267 | TCMU is designed so that multiple unrelated processes can manage TCMU |
| 268 | devices separately. All handlers should make sure to only open their |
| 269 | devices, based opon a known subtype string. |
| 270 | |
| 271 | a) Discovering and configuring TCMU UIO devices: |
| 272 | |
| 273 | (error checking omitted for brevity) |
| 274 | |
| 275 | int fd, dev_fd; |
| 276 | char buf[256]; |
| 277 | unsigned long long map_len; |
| 278 | void *map; |
| 279 | |
| 280 | fd = open("/sys/class/uio/uio0/name", O_RDONLY); |
| 281 | ret = read(fd, buf, sizeof(buf)); |
| 282 | close(fd); |
| 283 | buf[ret-1] = '\0'; /* null-terminate and chop off the \n */ |
| 284 | |
| 285 | /* we only want uio devices whose name is a format we expect */ |
| 286 | if (strncmp(buf, "tcm-user", 8)) |
| 287 | exit(-1); |
| 288 | |
| 289 | /* Further checking for subtype also needed here */ |
| 290 | |
| 291 | fd = open(/sys/class/uio/%s/maps/map0/size, O_RDONLY); |
| 292 | ret = read(fd, buf, sizeof(buf)); |
| 293 | close(fd); |
| 294 | str_buf[ret-1] = '\0'; /* null-terminate and chop off the \n */ |
| 295 | |
| 296 | map_len = strtoull(buf, NULL, 0); |
| 297 | |
| 298 | dev_fd = open("/dev/uio0", O_RDWR); |
| 299 | map = mmap(NULL, map_len, PROT_READ|PROT_WRITE, MAP_SHARED, dev_fd, 0); |
| 300 | |
| 301 | |
| 302 | b) Waiting for events on the device(s) |
| 303 | |
| 304 | while (1) { |
| 305 | char buf[4]; |
| 306 | |
| 307 | int ret = read(dev_fd, buf, 4); /* will block */ |
| 308 | |
| 309 | handle_device_events(dev_fd, map); |
| 310 | } |
| 311 | |
| 312 | |
| 313 | c) Managing the command ring |
| 314 | |
| 315 | #include <linux/target_core_user.h> |
| 316 | |
| 317 | int handle_device_events(int fd, void *map) |
| 318 | { |
| 319 | struct tcmu_mailbox *mb = map; |
| 320 | struct tcmu_cmd_entry *ent = (void *) mb + mb->cmdr_off + mb->cmd_tail; |
| 321 | int did_some_work = 0; |
| 322 | |
| 323 | /* Process events from cmd ring until we catch up with cmd_head */ |
| 324 | while (ent != (void *)mb + mb->cmdr_off + mb->cmd_head) { |
| 325 | |
Andy Grover | cf87edc | 2015-05-19 14:44:38 -0700 | [diff] [blame] | 326 | if (tcmu_hdr_get_op(ent->hdr.len_op) == TCMU_OP_CMD) { |
Andy Grover | ce87685 | 2014-10-01 16:07:04 -0700 | [diff] [blame] | 327 | uint8_t *cdb = (void *)mb + ent->req.cdb_off; |
| 328 | bool success = true; |
| 329 | |
| 330 | /* Handle command here. */ |
| 331 | printf("SCSI opcode: 0x%x\n", cdb[0]); |
| 332 | |
| 333 | /* Set response fields */ |
| 334 | if (success) |
| 335 | ent->rsp.scsi_status = SCSI_NO_SENSE; |
| 336 | else { |
| 337 | /* Also fill in rsp->sense_buffer here */ |
| 338 | ent->rsp.scsi_status = SCSI_CHECK_CONDITION; |
| 339 | } |
| 340 | } |
Andy Grover | cf87edc | 2015-05-19 14:44:38 -0700 | [diff] [blame] | 341 | else if (tcmu_hdr_get_op(ent->hdr.len_op) != TCMU_OP_PAD) { |
| 342 | /* Tell the kernel we didn't handle unknown opcodes */ |
| 343 | ent->hdr.uflags |= TCMU_UFLAG_UNKNOWN_OP; |
| 344 | } |
Andy Grover | ce87685 | 2014-10-01 16:07:04 -0700 | [diff] [blame] | 345 | else { |
Andy Grover | cf87edc | 2015-05-19 14:44:38 -0700 | [diff] [blame] | 346 | /* Do nothing for PAD entries except update cmd_tail */ |
Andy Grover | ce87685 | 2014-10-01 16:07:04 -0700 | [diff] [blame] | 347 | } |
| 348 | |
| 349 | /* update cmd_tail */ |
| 350 | mb->cmd_tail = (mb->cmd_tail + tcmu_hdr_get_len(&ent->hdr)) % mb->cmdr_size; |
| 351 | ent = (void *) mb + mb->cmdr_off + mb->cmd_tail; |
| 352 | did_some_work = 1; |
| 353 | } |
| 354 | |
| 355 | /* Notify the kernel that work has been finished */ |
| 356 | if (did_some_work) { |
| 357 | uint32_t buf = 0; |
| 358 | |
| 359 | write(fd, &buf, 4); |
| 360 | } |
| 361 | |
| 362 | return 0; |
| 363 | } |
| 364 | |
| 365 | |
Andy Grover | ce87685 | 2014-10-01 16:07:04 -0700 | [diff] [blame] | 366 | A final note |
| 367 | ------------ |
| 368 | |
| 369 | Please be careful to return codes as defined by the SCSI |
| 370 | specifications. These are different than some values defined in the |
| 371 | scsi/scsi.h include file. For example, CHECK CONDITION's status code |
| 372 | is 2, not 1. |