blob: 5518465290bf5d64b3ae9b94dd5440ed0a2b97c3 [file] [log] [blame]
Andy Groverce876852014-10-01 16:07:04 -07001Contents:
2
31) TCM Userspace Design
4 a) Background
5 b) Benefits
6 c) Design constraints
7 d) Implementation overview
8 i. Mailbox
9 ii. Command ring
10 iii. Data Area
11 e) Device discovery
12 f) Device events
13 g) Other contingencies
142) Writing a user pass-through handler
15 a) Discovering and configuring TCMU uio devices
16 b) Waiting for events on the device(s)
17 c) Managing the command ring
183) Command filtering and pass_level
194) A final note
20
21
22TCM Userspace Design
23--------------------
24
25TCM is another name for LIO, an in-kernel iSCSI target (server).
26Existing TCM targets run in the kernel. TCMU (TCM in Userspace)
27allows userspace programs to be written which act as iSCSI targets.
28This document describes the design.
29
30The existing kernel provides modules for different SCSI transport
31protocols. TCM also modularizes the data storage. There are existing
32modules for file, block device, RAM or using another SCSI device as
33storage. These are called "backstores" or "storage engines". These
34built-in modules are implemented entirely as kernel code.
35
36Background:
37
38In addition to modularizing the transport protocol used for carrying
39SCSI commands ("fabrics"), the Linux kernel target, LIO, also modularizes
40the actual data storage as well. These are referred to as "backstores"
41or "storage engines". The target comes with backstores that allow a
42file, a block device, RAM, or another SCSI device to be used for the
43local storage needed for the exported SCSI LUN. Like the rest of LIO,
44these are implemented entirely as kernel code.
45
46These backstores cover the most common use cases, but not all. One new
47use case that other non-kernel target solutions, such as tgt, are able
48to support is using Gluster's GLFS or Ceph's RBD as a backstore. The
49target then serves as a translator, allowing initiators to store data
50in these non-traditional networked storage systems, while still only
51using standard protocols themselves.
52
53If the target is a userspace process, supporting these is easy. tgt,
54for example, needs only a small adapter module for each, because the
55modules just use the available userspace libraries for RBD and GLFS.
56
57Adding support for these backstores in LIO is considerably more
58difficult, because LIO is entirely kernel code. Instead of undertaking
59the significant work to port the GLFS or RBD APIs and protocols to the
60kernel, another approach is to create a userspace pass-through
61backstore for LIO, "TCMU".
62
63
64Benefits:
65
66In addition to allowing relatively easy support for RBD and GLFS, TCMU
67will also allow easier development of new backstores. TCMU combines
68with the LIO loopback fabric to become something similar to FUSE
69(Filesystem in Userspace), but at the SCSI layer instead of the
70filesystem layer. A SUSE, if you will.
71
72The disadvantage is there are more distinct components to configure, and
73potentially to malfunction. This is unavoidable, but hopefully not
74fatal if we're careful to keep things as simple as possible.
75
76Design constraints:
77
78- Good performance: high throughput, low latency
79- Cleanly handle if userspace:
80 1) never attaches
81 2) hangs
82 3) dies
83 4) misbehaves
84- Allow future flexibility in user & kernel implementations
85- Be reasonably memory-efficient
86- Simple to configure & run
87- Simple to write a userspace backend
88
89
90Implementation overview:
91
92The core of the TCMU interface is a memory region that is shared
93between kernel and userspace. Within this region is: a control area
94(mailbox); a lockless producer/consumer circular buffer for commands
95to be passed up, and status returned; and an in/out data buffer area.
96
97TCMU uses the pre-existing UIO subsystem. UIO allows device driver
98development in userspace, and this is conceptually very close to the
99TCMU use case, except instead of a physical device, TCMU implements a
100memory-mapped layout designed for SCSI commands. Using UIO also
101benefits TCMU by handling device introspection (e.g. a way for
102userspace to determine how large the shared region is) and signaling
103mechanisms in both directions.
104
105There are no embedded pointers in the memory region. Everything is
106expressed as an offset from the region's starting address. This allows
107the ring to still work if the user process dies and is restarted with
108the region mapped at a different virtual address.
109
110See target_core_user.h for the struct definitions.
111
112The Mailbox:
113
114The mailbox is always at the start of the shared memory region, and
115contains a version, details about the starting offset and size of the
116command ring, and head and tail pointers to be used by the kernel and
117userspace (respectively) to put commands on the ring, and indicate
118when the commands are completed.
119
120version - 1 (userspace should abort if otherwise)
121flags - none yet defined.
122cmdr_off - The offset of the start of the command ring from the start
123of the memory region, to account for the mailbox size.
124cmdr_size - The size of the command ring. This does *not* need to be a
125power of two.
126cmd_head - Modified by the kernel to indicate when a command has been
127placed on the ring.
128cmd_tail - Modified by userspace to indicate when it has completed
129processing of a command.
130
131The Command Ring:
132
133Commands are placed on the ring by the kernel incrementing
134mailbox.cmd_head by the size of the command, modulo cmdr_size, and
135then signaling userspace via uio_event_notify(). Once the command is
136completed, userspace updates mailbox.cmd_tail in the same way and
137signals the kernel via a 4-byte write(). When cmd_head equals
138cmd_tail, the ring is empty -- no commands are currently waiting to be
139processed by userspace.
140
141TCMU commands start with a common header containing "len_op", a 32-bit
142value that stores the length, as well as the opcode in the lowest
143unused bits. Currently only two opcodes are defined, TCMU_OP_PAD and
144TCMU_OP_CMD. When userspace encounters a command with PAD opcode, it
145should skip ahead by the bytes in "length". (The kernel inserts PAD
146entries to ensure each CMD entry fits contigously into the circular
147buffer.)
148
149When userspace handles a CMD, it finds the SCSI CDB (Command Data
150Block) via tcmu_cmd_entry.req.cdb_off. This is an offset from the
151start of the overall shared memory region, not the entry. The data
152in/out buffers are accessible via tht req.iov[] array. Note that
153each iov.iov_base is also an offset from the start of the region.
154
155TCMU currently does not support BIDI operations.
156
157When completing a command, userspace sets rsp.scsi_status, and
158rsp.sense_buffer if necessary. Userspace then increments
159mailbox.cmd_tail by entry.hdr.length (mod cmdr_size) and signals the
160kernel via the UIO method, a 4-byte write to the file descriptor.
161
162The Data Area:
163
164This is shared-memory space after the command ring. The organization
165of this area is not defined in the TCMU interface, and userspace
166should access only the parts referenced by pending iovs.
167
168
169Device Discovery:
170
171Other devices may be using UIO besides TCMU. Unrelated user processes
172may also be handling different sets of TCMU devices. TCMU userspace
173processes must find their devices by scanning sysfs
174class/uio/uio*/name. For TCMU devices, these names will be of the
175format:
176
177tcm-user/<hba_num>/<device_name>/<subtype>/<path>
178
179where "tcm-user" is common for all TCMU-backed UIO devices. <hba_num>
180and <device_name> allow userspace to find the device's path in the
181kernel target's configfs tree. Assuming the usual mount point, it is
182found at:
183
184/sys/kernel/config/target/core/user_<hba_num>/<device_name>
185
186This location contains attributes such as "hw_block_size", that
187userspace needs to know for correct operation.
188
189<subtype> will be a userspace-process-unique string to identify the
190TCMU device as expecting to be backed by a certain handler, and <path>
191will be an additional handler-specific string for the user process to
192configure the device, if needed. The name cannot contain ':', due to
193LIO limitations.
194
195For all devices so discovered, the user handler opens /dev/uioX and
196calls mmap():
197
198mmap(NULL, size, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0)
199
200where size must be equal to the value read from
201/sys/class/uio/uioX/maps/map0/size.
202
203
204Device Events:
205
206If a new device is added or removed, a notification will be broadcast
207over netlink, using a generic netlink family name of "TCM-USER" and a
208multicast group named "config". This will include the UIO name as
209described in the previous section, as well as the UIO minor
210number. This should allow userspace to identify both the UIO device and
211the LIO device, so that after determining the device is supported
212(based on subtype) it can take the appropriate action.
213
214
215Other contingencies:
216
217Userspace handler process never attaches:
218
219- TCMU will post commands, and then abort them after a timeout period
220 (30 seconds.)
221
222Userspace handler process is killed:
223
224- It is still possible to restart and re-connect to TCMU
225 devices. Command ring is preserved. However, after the timeout period,
226 the kernel will abort pending tasks.
227
228Userspace handler process hangs:
229
230- The kernel will abort pending tasks after a timeout period.
231
232Userspace handler process is malicious:
233
234- The process can trivially break the handling of devices it controls,
235 but should not be able to access kernel memory outside its shared
236 memory areas.
237
238
239Writing a user pass-through handler (with example code)
240-------------------------------------------------------
241
242A user process handing a TCMU device must support the following:
243
244a) Discovering and configuring TCMU uio devices
245b) Waiting for events on the device(s)
246c) Managing the command ring: Parsing operations and commands,
247 performing work as needed, setting response fields (scsi_status and
248 possibly sense_buffer), updating cmd_tail, and notifying the kernel
249 that work has been finished
250
251First, consider instead writing a plugin for tcmu-runner. tcmu-runner
252implements all of this, and provides a higher-level API for plugin
253authors.
254
255TCMU is designed so that multiple unrelated processes can manage TCMU
256devices separately. All handlers should make sure to only open their
257devices, based opon a known subtype string.
258
259a) Discovering and configuring TCMU UIO devices:
260
261(error checking omitted for brevity)
262
263int fd, dev_fd;
264char buf[256];
265unsigned long long map_len;
266void *map;
267
268fd = open("/sys/class/uio/uio0/name", O_RDONLY);
269ret = read(fd, buf, sizeof(buf));
270close(fd);
271buf[ret-1] = '\0'; /* null-terminate and chop off the \n */
272
273/* we only want uio devices whose name is a format we expect */
274if (strncmp(buf, "tcm-user", 8))
275 exit(-1);
276
277/* Further checking for subtype also needed here */
278
279fd = open(/sys/class/uio/%s/maps/map0/size, O_RDONLY);
280ret = read(fd, buf, sizeof(buf));
281close(fd);
282str_buf[ret-1] = '\0'; /* null-terminate and chop off the \n */
283
284map_len = strtoull(buf, NULL, 0);
285
286dev_fd = open("/dev/uio0", O_RDWR);
287map = mmap(NULL, map_len, PROT_READ|PROT_WRITE, MAP_SHARED, dev_fd, 0);
288
289
290b) Waiting for events on the device(s)
291
292while (1) {
293 char buf[4];
294
295 int ret = read(dev_fd, buf, 4); /* will block */
296
297 handle_device_events(dev_fd, map);
298}
299
300
301c) Managing the command ring
302
303#include <linux/target_core_user.h>
304
305int handle_device_events(int fd, void *map)
306{
307 struct tcmu_mailbox *mb = map;
308 struct tcmu_cmd_entry *ent = (void *) mb + mb->cmdr_off + mb->cmd_tail;
309 int did_some_work = 0;
310
311 /* Process events from cmd ring until we catch up with cmd_head */
312 while (ent != (void *)mb + mb->cmdr_off + mb->cmd_head) {
313
314 if (tcmu_hdr_get_op(&ent->hdr) == TCMU_OP_CMD) {
315 uint8_t *cdb = (void *)mb + ent->req.cdb_off;
316 bool success = true;
317
318 /* Handle command here. */
319 printf("SCSI opcode: 0x%x\n", cdb[0]);
320
321 /* Set response fields */
322 if (success)
323 ent->rsp.scsi_status = SCSI_NO_SENSE;
324 else {
325 /* Also fill in rsp->sense_buffer here */
326 ent->rsp.scsi_status = SCSI_CHECK_CONDITION;
327 }
328 }
329 else {
330 /* Do nothing for PAD entries */
331 }
332
333 /* update cmd_tail */
334 mb->cmd_tail = (mb->cmd_tail + tcmu_hdr_get_len(&ent->hdr)) % mb->cmdr_size;
335 ent = (void *) mb + mb->cmdr_off + mb->cmd_tail;
336 did_some_work = 1;
337 }
338
339 /* Notify the kernel that work has been finished */
340 if (did_some_work) {
341 uint32_t buf = 0;
342
343 write(fd, &buf, 4);
344 }
345
346 return 0;
347}
348
349
350Command filtering and pass_level
351--------------------------------
352
353TCMU supports a "pass_level" option with valid values of 0 or 1. When
354the value is 0 (the default), nearly all SCSI commands received for
355the device are passed through to the handler. This allows maximum
356flexibility but increases the amount of code required by the handler,
357to support all mandatory SCSI commands. If pass_level is set to 1,
358then only IO-related commands are presented, and the rest are handled
359by LIO's in-kernel command emulation. The commands presented at level
3601 include all versions of:
361
362READ
363WRITE
364WRITE_VERIFY
365XDWRITEREAD
366WRITE_SAME
367COMPARE_AND_WRITE
368SYNCHRONIZE_CACHE
369UNMAP
370
371
372A final note
373------------
374
375Please be careful to return codes as defined by the SCSI
376specifications. These are different than some values defined in the
377scsi/scsi.h include file. For example, CHECK CONDITION's status code
378is 2, not 1.