blob: 197a0b6b058215c60f3d79f3f40cc2aca4504013 [file] [log] [blame]
Dan Williamsbc301962015-06-25 04:48:19 -04001 LIBNVDIMM: Non-Volatile Devices
2 libnvdimm - kernel / libndctl - userspace helper library
3 linux-nvdimm@lists.01.org
4 v13
5
6
7 Glossary
8 Overview
9 Supporting Documents
10 Git Trees
11 LIBNVDIMM PMEM and BLK
12 Why BLK?
13 PMEM vs BLK
14 BLK-REGIONs, PMEM-REGIONs, Atomic Sectors, and DAX
15 Example NVDIMM Platform
16 LIBNVDIMM Kernel Device Model and LIBNDCTL Userspace API
17 LIBNDCTL: Context
18 libndctl: instantiate a new library context example
19 LIBNVDIMM/LIBNDCTL: Bus
20 libnvdimm: control class device in /sys/class
21 libnvdimm: bus
22 libndctl: bus enumeration example
23 LIBNVDIMM/LIBNDCTL: DIMM (NMEM)
24 libnvdimm: DIMM (NMEM)
25 libndctl: DIMM enumeration example
26 LIBNVDIMM/LIBNDCTL: Region
27 libnvdimm: region
28 libndctl: region enumeration example
29 Why Not Encode the Region Type into the Region Name?
30 How Do I Determine the Major Type of a Region?
31 LIBNVDIMM/LIBNDCTL: Namespace
32 libnvdimm: namespace
33 libndctl: namespace enumeration example
34 libndctl: namespace creation example
35 Why the Term "namespace"?
36 LIBNVDIMM/LIBNDCTL: Block Translation Table "btt"
37 libnvdimm: btt layout
38 libndctl: btt creation example
39 Summary LIBNDCTL Diagram
40
41
42Glossary
43--------
44
45PMEM: A system-physical-address range where writes are persistent. A
46block device composed of PMEM is capable of DAX. A PMEM address range
47may span an interleave of several DIMMs.
48
49BLK: A set of one or more programmable memory mapped apertures provided
50by a DIMM to access its media. This indirection precludes the
51performance benefit of interleaving, but enables DIMM-bounded failure
52modes.
53
54DPA: DIMM Physical Address, is a DIMM-relative offset. With one DIMM in
55the system there would be a 1:1 system-physical-address:DPA association.
56Once more DIMMs are added a memory controller interleave must be
57decoded to determine the DPA associated with a given
58system-physical-address. BLK capacity always has a 1:1 relationship
59with a single-DIMM's DPA range.
60
61DAX: File system extensions to bypass the page cache and block layer to
62mmap persistent memory, from a PMEM block device, directly into a
63process address space.
64
65BTT: Block Translation Table: Persistent memory is byte addressable.
66Existing software may have an expectation that the power-fail-atomicity
67of writes is at least one sector, 512 bytes. The BTT is an indirection
68table with atomic update semantics to front a PMEM/BLK block device
69driver and present arbitrary atomic sector sizes.
70
71LABEL: Metadata stored on a DIMM device that partitions and identifies
72(persistently names) storage between PMEM and BLK. It also partitions
73BLK storage to host BTTs with different parameters per BLK-partition.
74Note that traditional partition tables, GPT/MBR, are layered on top of a
75BLK or PMEM device.
76
77
78Overview
79--------
80
81The LIBNVDIMM subsystem provides support for three types of NVDIMMs, namely,
82PMEM, BLK, and NVDIMM devices that can simultaneously support both PMEM
83and BLK mode access. These three modes of operation are described by
84the "NVDIMM Firmware Interface Table" (NFIT) in ACPI 6. While the LIBNVDIMM
85implementation is generic and supports pre-NFIT platforms, it was guided
86by the superset of capabilities need to support this ACPI 6 definition
87for NVDIMM resources. The bulk of the kernel implementation is in place
88to handle the case where DPA accessible via PMEM is aliased with DPA
89accessible via BLK. When that occurs a LABEL is needed to reserve DPA
90for exclusive access via one mode a time.
91
92Supporting Documents
93ACPI 6: http://www.uefi.org/sites/default/files/resources/ACPI_6.0.pdf
94NVDIMM Namespace: http://pmem.io/documents/NVDIMM_Namespace_Spec.pdf
95DSM Interface Example: http://pmem.io/documents/NVDIMM_DSM_Interface_Example.pdf
96Driver Writer's Guide: http://pmem.io/documents/NVDIMM_Driver_Writers_Guide.pdf
97
98Git Trees
99LIBNVDIMM: https://git.kernel.org/cgit/linux/kernel/git/djbw/nvdimm.git
100LIBNDCTL: https://github.com/pmem/ndctl.git
101PMEM: https://github.com/01org/prd
102
103
104LIBNVDIMM PMEM and BLK
105------------------
106
107Prior to the arrival of the NFIT, non-volatile memory was described to a
108system in various ad-hoc ways. Usually only the bare minimum was
109provided, namely, a single system-physical-address range where writes
110are expected to be durable after a system power loss. Now, the NFIT
111specification standardizes not only the description of PMEM, but also
112BLK and platform message-passing entry points for control and
113configuration.
114
115For each NVDIMM access method (PMEM, BLK), LIBNVDIMM provides a block
116device driver:
117
118 1. PMEM (nd_pmem.ko): Drives a system-physical-address range. This
119 range is contiguous in system memory and may be interleaved (hardware
120 memory controller striped) across multiple DIMMs. When interleaved the
121 platform may optionally provide details of which DIMMs are participating
122 in the interleave.
123
124 Note that while LIBNVDIMM describes system-physical-address ranges that may
125 alias with BLK access as ND_NAMESPACE_PMEM ranges and those without
126 alias as ND_NAMESPACE_IO ranges, to the nd_pmem driver there is no
127 distinction. The different device-types are an implementation detail
128 that userspace can exploit to implement policies like "only interface
129 with address ranges from certain DIMMs". It is worth noting that when
130 aliasing is present and a DIMM lacks a label, then no block device can
131 be created by default as userspace needs to do at least one allocation
132 of DPA to the PMEM range. In contrast ND_NAMESPACE_IO ranges, once
133 registered, can be immediately attached to nd_pmem.
134
135 2. BLK (nd_blk.ko): This driver performs I/O using a set of platform
136 defined apertures. A set of apertures will all access just one DIMM.
137 Multiple windows allow multiple concurrent accesses, much like
138 tagged-command-queuing, and would likely be used by different threads or
139 different CPUs.
140
141 The NFIT specification defines a standard format for a BLK-aperture, but
142 the spec also allows for vendor specific layouts, and non-NFIT BLK
143 implementations may other designs for BLK I/O. For this reason "nd_blk"
144 calls back into platform-specific code to perform the I/O. One such
145 implementation is defined in the "Driver Writer's Guide" and "DSM
146 Interface Example".
147
148
149Why BLK?
150--------
151
152While PMEM provides direct byte-addressable CPU-load/store access to
153NVDIMM storage, it does not provide the best system RAS (recovery,
154availability, and serviceability) model. An access to a corrupted
155system-physical-address address causes a cpu exception while an access
156to a corrupted address through an BLK-aperture causes that block window
157to raise an error status in a register. The latter is more aligned with
158the standard error model that host-bus-adapter attached disks present.
159Also, if an administrator ever wants to replace a memory it is easier to
160service a system at DIMM module boundaries. Compare this to PMEM where
161data could be interleaved in an opaque hardware specific manner across
162several DIMMs.
163
164PMEM vs BLK
165BLK-apertures solve this RAS problem, but their presence is also the
166major contributing factor to the complexity of the ND subsystem. They
167complicate the implementation because PMEM and BLK alias in DPA space.
168Any given DIMM's DPA-range may contribute to one or more
169system-physical-address sets of interleaved DIMMs, *and* may also be
170accessed in its entirety through its BLK-aperture. Accessing a DPA
171through a system-physical-address while simultaneously accessing the
172same DPA through a BLK-aperture has undefined results. For this reason,
173DIMMs with this dual interface configuration include a DSM function to
174store/retrieve a LABEL. The LABEL effectively partitions the DPA-space
175into exclusive system-physical-address and BLK-aperture accessible
176regions. For simplicity a DIMM is allowed a PMEM "region" per each
177interleave set in which it is a member. The remaining DPA space can be
178carved into an arbitrary number of BLK devices with discontiguous
179extents.
180
181BLK-REGIONs, PMEM-REGIONs, Atomic Sectors, and DAX
182--------------------------------------------------
183
184One of the few
185reasons to allow multiple BLK namespaces per REGION is so that each
186BLK-namespace can be configured with a BTT with unique atomic sector
187sizes. While a PMEM device can host a BTT the LABEL specification does
188not provide for a sector size to be specified for a PMEM namespace.
189This is due to the expectation that the primary usage model for PMEM is
190via DAX, and the BTT is incompatible with DAX. However, for the cases
191where an application or filesystem still needs atomic sector update
192guarantees it can register a BTT on a PMEM device or partition. See
193LIBNVDIMM/NDCTL: Block Translation Table "btt"
194
195
196Example NVDIMM Platform
197-----------------------
198
199For the remainder of this document the following diagram will be
200referenced for any example sysfs layouts.
201
202
203 (a) (b) DIMM BLK-REGION
204 +-------------------+--------+--------+--------+
205+------+ | pm0.0 | blk2.0 | pm1.0 | blk2.1 | 0 region2
206| imc0 +--+- - - region0- - - +--------+ +--------+
207+--+---+ | pm0.0 | blk3.0 | pm1.0 | blk3.1 | 1 region3
208 | +-------------------+--------v v--------+
209+--+---+ | |
210| cpu0 | region1
211+--+---+ | |
212 | +----------------------------^ ^--------+
213+--+---+ | blk4.0 | pm1.0 | blk4.0 | 2 region4
214| imc1 +--+----------------------------| +--------+
215+------+ | blk5.0 | pm1.0 | blk5.0 | 3 region5
216 +----------------------------+--------+--------+
217
218In this platform we have four DIMMs and two memory controllers in one
219socket. Each unique interface (BLK or PMEM) to DPA space is identified
220by a region device with a dynamically assigned id (REGION0 - REGION5).
221
222 1. The first portion of DIMM0 and DIMM1 are interleaved as REGION0. A
223 single PMEM namespace is created in the REGION0-SPA-range that spans
224 DIMM0 and DIMM1 with a user-specified name of "pm0.0". Some of that
225 interleaved system-physical-address range is reclaimed as BLK-aperture
226 accessed space starting at DPA-offset (a) into each DIMM. In that
227 reclaimed space we create two BLK-aperture "namespaces" from REGION2 and
228 REGION3 where "blk2.0" and "blk3.0" are just human readable names that
229 could be set to any user-desired name in the LABEL.
230
231 2. In the last portion of DIMM0 and DIMM1 we have an interleaved
232 system-physical-address range, REGION1, that spans those two DIMMs as
233 well as DIMM2 and DIMM3. Some of REGION1 allocated to a PMEM namespace
234 named "pm1.0" the rest is reclaimed in 4 BLK-aperture namespaces (for
235 each DIMM in the interleave set), "blk2.1", "blk3.1", "blk4.0", and
236 "blk5.0".
237
238 3. The portion of DIMM2 and DIMM3 that do not participate in the REGION1
239 interleaved system-physical-address range (i.e. the DPA address below
240 offset (b) are also included in the "blk4.0" and "blk5.0" namespaces.
241 Note, that this example shows that BLK-aperture namespaces don't need to
242 be contiguous in DPA-space.
243
244 This bus is provided by the kernel under the device
245 /sys/devices/platform/nfit_test.0 when CONFIG_NFIT_TEST is enabled and
246 the nfit_test.ko module is loaded. This not only test LIBNVDIMM but the
247 acpi_nfit.ko driver as well.
248
249
250LIBNVDIMM Kernel Device Model and LIBNDCTL Userspace API
251----------------------------------------------------
252
253What follows is a description of the LIBNVDIMM sysfs layout and a
254corresponding object hierarchy diagram as viewed through the LIBNDCTL
255api. The example sysfs paths and diagrams are relative to the Example
256NVDIMM Platform which is also the LIBNVDIMM bus used in the LIBNDCTL unit
257test.
258
259LIBNDCTL: Context
260Every api call in the LIBNDCTL library requires a context that holds the
261logging parameters and other library instance state. The library is
262based on the libabc template:
263https://git.kernel.org/cgit/linux/kernel/git/kay/libabc.git/
264
265LIBNDCTL: instantiate a new library context example
266
267 struct ndctl_ctx *ctx;
268
269 if (ndctl_new(&ctx) == 0)
270 return ctx;
271 else
272 return NULL;
273
274LIBNVDIMM/LIBNDCTL: Bus
275-------------------
276
277A bus has a 1:1 relationship with an NFIT. The current expectation for
278ACPI based systems is that there is only ever one platform-global NFIT.
279That said, it is trivial to register multiple NFITs, the specification
280does not preclude it. The infrastructure supports multiple busses and
281we we use this capability to test multiple NFIT configurations in the
282unit test.
283
284LIBNVDIMM: control class device in /sys/class
285
286This character device accepts DSM messages to be passed to DIMM
287identified by its NFIT handle.
288
289 /sys/class/nd/ndctl0
290 |-- dev
291 |-- device -> ../../../ndbus0
292 |-- subsystem -> ../../../../../../../class/nd
293
294
295
296LIBNVDIMM: bus
297
298 struct nvdimm_bus *nvdimm_bus_register(struct device *parent,
299 struct nvdimm_bus_descriptor *nfit_desc);
300
301 /sys/devices/platform/nfit_test.0/ndbus0
302 |-- commands
303 |-- nd
304 |-- nfit
305 |-- nmem0
306 |-- nmem1
307 |-- nmem2
308 |-- nmem3
309 |-- power
310 |-- provider
311 |-- region0
312 |-- region1
313 |-- region2
314 |-- region3
315 |-- region4
316 |-- region5
317 |-- uevent
318 `-- wait_probe
319
320LIBNDCTL: bus enumeration example
321Find the bus handle that describes the bus from Example NVDIMM Platform
322
323 static struct ndctl_bus *get_bus_by_provider(struct ndctl_ctx *ctx,
324 const char *provider)
325 {
326 struct ndctl_bus *bus;
327
328 ndctl_bus_foreach(ctx, bus)
329 if (strcmp(provider, ndctl_bus_get_provider(bus)) == 0)
330 return bus;
331
332 return NULL;
333 }
334
335 bus = get_bus_by_provider(ctx, "nfit_test.0");
336
337
338LIBNVDIMM/LIBNDCTL: DIMM (NMEM)
339---------------------------
340
341The DIMM device provides a character device for sending commands to
342hardware, and it is a container for LABELs. If the DIMM is defined by
343NFIT then an optional 'nfit' attribute sub-directory is available to add
344NFIT-specifics.
345
346Note that the kernel device name for "DIMMs" is "nmemX". The NFIT
347describes these devices via "Memory Device to System Physical Address
348Range Mapping Structure", and there is no requirement that they actually
349be physical DIMMs, so we use a more generic name.
350
351LIBNVDIMM: DIMM (NMEM)
352
353 struct nvdimm *nvdimm_create(struct nvdimm_bus *nvdimm_bus, void *provider_data,
354 const struct attribute_group **groups, unsigned long flags,
355 unsigned long *dsm_mask);
356
357 /sys/devices/platform/nfit_test.0/ndbus0
358 |-- nmem0
359 | |-- available_slots
360 | |-- commands
361 | |-- dev
362 | |-- devtype
363 | |-- driver -> ../../../../../bus/nd/drivers/nvdimm
364 | |-- modalias
365 | |-- nfit
366 | | |-- device
367 | | |-- format
368 | | |-- handle
369 | | |-- phys_id
370 | | |-- rev_id
371 | | |-- serial
372 | | `-- vendor
373 | |-- state
374 | |-- subsystem -> ../../../../../bus/nd
375 | `-- uevent
376 |-- nmem1
377 [..]
378
379
380LIBNDCTL: DIMM enumeration example
381
382Note, in this example we are assuming NFIT-defined DIMMs which are
383identified by an "nfit_handle" a 32-bit value where:
384Bit 3:0 DIMM number within the memory channel
385Bit 7:4 memory channel number
386Bit 11:8 memory controller ID
387Bit 15:12 socket ID (within scope of a Node controller if node controller is present)
388Bit 27:16 Node Controller ID
389Bit 31:28 Reserved
390
391 static struct ndctl_dimm *get_dimm_by_handle(struct ndctl_bus *bus,
392 unsigned int handle)
393 {
394 struct ndctl_dimm *dimm;
395
396 ndctl_dimm_foreach(bus, dimm)
397 if (ndctl_dimm_get_handle(dimm) == handle)
398 return dimm;
399
400 return NULL;
401 }
402
403 #define DIMM_HANDLE(n, s, i, c, d) \
404 (((n & 0xfff) << 16) | ((s & 0xf) << 12) | ((i & 0xf) << 8) \
405 | ((c & 0xf) << 4) | (d & 0xf))
406
407 dimm = get_dimm_by_handle(bus, DIMM_HANDLE(0, 0, 0, 0, 0));
408
409LIBNVDIMM/LIBNDCTL: Region
410----------------------
411
412A generic REGION device is registered for each PMEM range orBLK-aperture
413set. Per the example there are 6 regions: 2 PMEM and 4 BLK-aperture
414sets on the "nfit_test.0" bus. The primary role of regions are to be a
415container of "mappings". A mapping is a tuple of <DIMM,
416DPA-start-offset, length>.
417
418LIBNVDIMM provides a built-in driver for these REGION devices. This driver
419is responsible for reconciling the aliased DPA mappings across all
420regions, parsing the LABEL, if present, and then emitting NAMESPACE
421devices with the resolved/exclusive DPA-boundaries for the nd_pmem or
422nd_blk device driver to consume.
423
424In addition to the generic attributes of "mapping"s, "interleave_ways"
425and "size" the REGION device also exports some convenience attributes.
426"nstype" indicates the integer type of namespace-device this region
427emits, "devtype" duplicates the DEVTYPE variable stored by udev at the
428'add' event, "modalias" duplicates the MODALIAS variable stored by udev
429at the 'add' event, and finally, the optional "spa_index" is provided in
430the case where the region is defined by a SPA.
431
432LIBNVDIMM: region
433
434 struct nd_region *nvdimm_pmem_region_create(struct nvdimm_bus *nvdimm_bus,
435 struct nd_region_desc *ndr_desc);
436 struct nd_region *nvdimm_blk_region_create(struct nvdimm_bus *nvdimm_bus,
437 struct nd_region_desc *ndr_desc);
438
439 /sys/devices/platform/nfit_test.0/ndbus0
440 |-- region0
441 | |-- available_size
442 | |-- btt0
443 | |-- btt_seed
444 | |-- devtype
445 | |-- driver -> ../../../../../bus/nd/drivers/nd_region
446 | |-- init_namespaces
447 | |-- mapping0
448 | |-- mapping1
449 | |-- mappings
450 | |-- modalias
451 | |-- namespace0.0
452 | |-- namespace_seed
453 | |-- numa_node
454 | |-- nfit
455 | | `-- spa_index
456 | |-- nstype
457 | |-- set_cookie
458 | |-- size
459 | |-- subsystem -> ../../../../../bus/nd
460 | `-- uevent
461 |-- region1
462 [..]
463
464LIBNDCTL: region enumeration example
465
466Sample region retrieval routines based on NFIT-unique data like
467"spa_index" (interleave set id) for PMEM and "nfit_handle" (dimm id) for
468BLK.
469
470 static struct ndctl_region *get_pmem_region_by_spa_index(struct ndctl_bus *bus,
471 unsigned int spa_index)
472 {
473 struct ndctl_region *region;
474
475 ndctl_region_foreach(bus, region) {
476 if (ndctl_region_get_type(region) != ND_DEVICE_REGION_PMEM)
477 continue;
478 if (ndctl_region_get_spa_index(region) == spa_index)
479 return region;
480 }
481 return NULL;
482 }
483
484 static struct ndctl_region *get_blk_region_by_dimm_handle(struct ndctl_bus *bus,
485 unsigned int handle)
486 {
487 struct ndctl_region *region;
488
489 ndctl_region_foreach(bus, region) {
490 struct ndctl_mapping *map;
491
492 if (ndctl_region_get_type(region) != ND_DEVICE_REGION_BLOCK)
493 continue;
494 ndctl_mapping_foreach(region, map) {
495 struct ndctl_dimm *dimm = ndctl_mapping_get_dimm(map);
496
497 if (ndctl_dimm_get_handle(dimm) == handle)
498 return region;
499 }
500 }
501 return NULL;
502 }
503
504
505Why Not Encode the Region Type into the Region Name?
506----------------------------------------------------
507
508At first glance it seems since NFIT defines just PMEM and BLK interface
509types that we should simply name REGION devices with something derived
510from those type names. However, the ND subsystem explicitly keeps the
511REGION name generic and expects userspace to always consider the
512region-attributes for 4 reasons:
513
514 1. There are already more than two REGION and "namespace" types. For
515 PMEM there are two subtypes. As mentioned previously we have PMEM where
516 the constituent DIMM devices are known and anonymous PMEM. For BLK
517 regions the NFIT specification already anticipates vendor specific
518 implementations. The exact distinction of what a region contains is in
519 the region-attributes not the region-name or the region-devtype.
520
521 2. A region with zero child-namespaces is a possible configuration. For
522 example, the NFIT allows for a DCR to be published without a
523 corresponding BLK-aperture. This equates to a DIMM that can only accept
524 control/configuration messages, but no i/o through a descendant block
525 device. Again, this "type" is advertised in the attributes ('mappings'
526 == 0) and the name does not tell you much.
527
528 3. What if a third major interface type arises in the future? Outside
529 of vendor specific implementations, it's not difficult to envision a
530 third class of interface type beyond BLK and PMEM. With a generic name
531 for the REGION level of the device-hierarchy old userspace
532 implementations can still make sense of new kernel advertised
533 region-types. Userspace can always rely on the generic region
534 attributes like "mappings", "size", etc and the expected child devices
535 named "namespace". This generic format of the device-model hierarchy
536 allows the LIBNVDIMM and LIBNDCTL implementations to be more uniform and
537 future-proof.
538
539 4. There are more robust mechanisms for determining the major type of a
540 region than a device name. See the next section, How Do I Determine the
541 Major Type of a Region?
542
543How Do I Determine the Major Type of a Region?
544----------------------------------------------
545
546Outside of the blanket recommendation of "use libndctl", or simply
547looking at the kernel header (/usr/include/linux/ndctl.h) to decode the
548"nstype" integer attribute, here are some other options.
549
550 1. module alias lookup:
551
552 The whole point of region/namespace device type differentiation is to
553 decide which block-device driver will attach to a given LIBNVDIMM namespace.
554 One can simply use the modalias to lookup the resulting module. It's
555 important to note that this method is robust in the presence of a
556 vendor-specific driver down the road. If a vendor-specific
557 implementation wants to supplant the standard nd_blk driver it can with
558 minimal impact to the rest of LIBNVDIMM.
559
560 In fact, a vendor may also want to have a vendor-specific region-driver
561 (outside of nd_region). For example, if a vendor defined its own LABEL
562 format it would need its own region driver to parse that LABEL and emit
563 the resulting namespaces. The output from module resolution is more
564 accurate than a region-name or region-devtype.
565
566 2. udev:
567
568 The kernel "devtype" is registered in the udev database
569 # udevadm info --path=/devices/platform/nfit_test.0/ndbus0/region0
570 P: /devices/platform/nfit_test.0/ndbus0/region0
571 E: DEVPATH=/devices/platform/nfit_test.0/ndbus0/region0
572 E: DEVTYPE=nd_pmem
573 E: MODALIAS=nd:t2
574 E: SUBSYSTEM=nd
575
576 # udevadm info --path=/devices/platform/nfit_test.0/ndbus0/region4
577 P: /devices/platform/nfit_test.0/ndbus0/region4
578 E: DEVPATH=/devices/platform/nfit_test.0/ndbus0/region4
579 E: DEVTYPE=nd_blk
580 E: MODALIAS=nd:t3
581 E: SUBSYSTEM=nd
582
583 ...and is available as a region attribute, but keep in mind that the
584 "devtype" does not indicate sub-type variations and scripts should
585 really be understanding the other attributes.
586
587 3. type specific attributes:
588
589 As it currently stands a BLK-aperture region will never have a
590 "nfit/spa_index" attribute, but neither will a non-NFIT PMEM region. A
591 BLK region with a "mappings" value of 0 is, as mentioned above, a DIMM
592 that does not allow I/O. A PMEM region with a "mappings" value of zero
593 is a simple system-physical-address range.
594
595
596LIBNVDIMM/LIBNDCTL: Namespace
597-------------------------
598
599A REGION, after resolving DPA aliasing and LABEL specified boundaries,
600surfaces one or more "namespace" devices. The arrival of a "namespace"
601device currently triggers either the nd_blk or nd_pmem driver to load
602and register a disk/block device.
603
604LIBNVDIMM: namespace
605Here is a sample layout from the three major types of NAMESPACE where
606namespace0.0 represents DIMM-info-backed PMEM (note that it has a 'uuid'
607attribute), namespace2.0 represents a BLK namespace (note it has a
608'sector_size' attribute) that, and namespace6.0 represents an anonymous
609PMEM namespace (note that has no 'uuid' attribute due to not support a
610LABEL).
611
612 /sys/devices/platform/nfit_test.0/ndbus0/region0/namespace0.0
613 |-- alt_name
614 |-- devtype
615 |-- dpa_extents
616 |-- force_raw
617 |-- modalias
618 |-- numa_node
619 |-- resource
620 |-- size
621 |-- subsystem -> ../../../../../../bus/nd
622 |-- type
623 |-- uevent
624 `-- uuid
625 /sys/devices/platform/nfit_test.0/ndbus0/region2/namespace2.0
626 |-- alt_name
627 |-- devtype
628 |-- dpa_extents
629 |-- force_raw
630 |-- modalias
631 |-- numa_node
632 |-- sector_size
633 |-- size
634 |-- subsystem -> ../../../../../../bus/nd
635 |-- type
636 |-- uevent
637 `-- uuid
638 /sys/devices/platform/nfit_test.1/ndbus1/region6/namespace6.0
639 |-- block
640 | `-- pmem0
641 |-- devtype
642 |-- driver -> ../../../../../../bus/nd/drivers/pmem
643 |-- force_raw
644 |-- modalias
645 |-- numa_node
646 |-- resource
647 |-- size
648 |-- subsystem -> ../../../../../../bus/nd
649 |-- type
650 `-- uevent
651
652LIBNDCTL: namespace enumeration example
653Namespaces are indexed relative to their parent region, example below.
654These indexes are mostly static from boot to boot, but subsystem makes
655no guarantees in this regard. For a static namespace identifier use its
656'uuid' attribute.
657
658static struct ndctl_namespace *get_namespace_by_id(struct ndctl_region *region,
659 unsigned int id)
660{
661 struct ndctl_namespace *ndns;
662
663 ndctl_namespace_foreach(region, ndns)
664 if (ndctl_namespace_get_id(ndns) == id)
665 return ndns;
666
667 return NULL;
668}
669
670LIBNDCTL: namespace creation example
671Idle namespaces are automatically created by the kernel if a given
672region has enough available capacity to create a new namespace.
673Namespace instantiation involves finding an idle namespace and
674configuring it. For the most part the setting of namespace attributes
675can occur in any order, the only constraint is that 'uuid' must be set
676before 'size'. This enables the kernel to track DPA allocations
677internally with a static identifier.
678
679static int configure_namespace(struct ndctl_region *region,
680 struct ndctl_namespace *ndns,
681 struct namespace_parameters *parameters)
682{
683 char devname[50];
684
685 snprintf(devname, sizeof(devname), "namespace%d.%d",
686 ndctl_region_get_id(region), paramaters->id);
687
688 ndctl_namespace_set_alt_name(ndns, devname);
689 /* 'uuid' must be set prior to setting size! */
690 ndctl_namespace_set_uuid(ndns, paramaters->uuid);
691 ndctl_namespace_set_size(ndns, paramaters->size);
692 /* unlike pmem namespaces, blk namespaces have a sector size */
693 if (parameters->lbasize)
694 ndctl_namespace_set_sector_size(ndns, parameters->lbasize);
695 ndctl_namespace_enable(ndns);
696}
697
698
699Why the Term "namespace"?
700
701 1. Why not "volume" for instance? "volume" ran the risk of confusing ND
702 as a volume manager like device-mapper.
703
704 2. The term originated to describe the sub-devices that can be created
705 within a NVME controller (see the nvme specification:
706 http://www.nvmexpress.org/specifications/), and NFIT namespaces are
707 meant to parallel the capabilities and configurability of
708 NVME-namespaces.
709
710
711LIBNVDIMM/LIBNDCTL: Block Translation Table "btt"
712---------------------------------------------
713
714A BTT (design document: http://pmem.io/2014/09/23/btt.html) is a stacked
715block device driver that fronts either the whole block device or a
716partition of a block device emitted by either a PMEM or BLK NAMESPACE.
717
718LIBNVDIMM: btt layout
719Every region will start out with at least one BTT device which is the
720seed device. To activate it set the "namespace", "uuid", and
721"sector_size" attributes and then bind the device to the nd_pmem or
722nd_blk driver depending on the region type.
723
724 /sys/devices/platform/nfit_test.1/ndbus0/region0/btt0/
725 |-- namespace
726 |-- delete
727 |-- devtype
728 |-- modalias
729 |-- numa_node
730 |-- sector_size
731 |-- subsystem -> ../../../../../bus/nd
732 |-- uevent
733 `-- uuid
734
735LIBNDCTL: btt creation example
736Similar to namespaces an idle BTT device is automatically created per
737region. Each time this "seed" btt device is configured and enabled a new
738seed is created. Creating a BTT configuration involves two steps of
739finding and idle BTT and assigning it to consume a PMEM or BLK namespace.
740
741 static struct ndctl_btt *get_idle_btt(struct ndctl_region *region)
742 {
743 struct ndctl_btt *btt;
744
745 ndctl_btt_foreach(region, btt)
746 if (!ndctl_btt_is_enabled(btt)
747 && !ndctl_btt_is_configured(btt))
748 return btt;
749
750 return NULL;
751 }
752
753 static int configure_btt(struct ndctl_region *region,
754 struct btt_parameters *parameters)
755 {
756 btt = get_idle_btt(region);
757
758 ndctl_btt_set_uuid(btt, parameters->uuid);
759 ndctl_btt_set_sector_size(btt, parameters->sector_size);
760 ndctl_btt_set_namespace(btt, parameters->ndns);
761 /* turn off raw mode device */
762 ndctl_namespace_disable(parameters->ndns);
763 /* turn on btt access */
764 ndctl_btt_enable(btt);
765 }
766
767Once instantiated a new inactive btt seed device will appear underneath
768the region.
769
770Once a "namespace" is removed from a BTT that instance of the BTT device
771will be deleted or otherwise reset to default values. This deletion is
772only at the device model level. In order to destroy a BTT the "info
773block" needs to be destroyed. Note, that to destroy a BTT the media
774needs to be written in raw mode. By default, the kernel will autodetect
775the presence of a BTT and disable raw mode. This autodetect behavior
776can be suppressed by enabling raw mode for the namespace via the
777ndctl_namespace_set_raw_mode() api.
778
779
780Summary LIBNDCTL Diagram
781------------------------
782
783For the given example above, here is the view of the objects as seen by the LIBNDCTL api:
784 +---+
785 |CTX| +---------+ +--------------+ +---------------+
786 +-+-+ +-> REGION0 +---> NAMESPACE0.0 +--> PMEM8 "pm0.0" |
787 | | +---------+ +--------------+ +---------------+
788+-------+ | | +---------+ +--------------+ +---------------+
789| DIMM0 <-+ | +-> REGION1 +---> NAMESPACE1.0 +--> PMEM6 "pm1.0" |
790+-------+ | | | +---------+ +--------------+ +---------------+
791| DIMM1 <-+ +-v--+ | +---------+ +--------------+ +---------------+
792+-------+ +-+BUS0+---> REGION2 +-+-> NAMESPACE2.0 +--> ND6 "blk2.0" |
793| DIMM2 <-+ +----+ | +---------+ | +--------------+ +----------------------+
794+-------+ | | +-> NAMESPACE2.1 +--> ND5 "blk2.1" | BTT2 |
795| DIMM3 <-+ | +--------------+ +----------------------+
796+-------+ | +---------+ +--------------+ +---------------+
797 +-> REGION3 +-+-> NAMESPACE3.0 +--> ND4 "blk3.0" |
798 | +---------+ | +--------------+ +----------------------+
799 | +-> NAMESPACE3.1 +--> ND3 "blk3.1" | BTT1 |
800 | +--------------+ +----------------------+
801 | +---------+ +--------------+ +---------------+
802 +-> REGION4 +---> NAMESPACE4.0 +--> ND2 "blk4.0" |
803 | +---------+ +--------------+ +---------------+
804 | +---------+ +--------------+ +----------------------+
805 +-> REGION5 +---> NAMESPACE5.0 +--> ND1 "blk5.0" | BTT0 |
806 +---------+ +--------------+ +---------------+------+
807
808