blob: 801d9d1cf82be56facc9a8e9f99b41d35739374a [file] [log] [blame]
Joe Thornber991d9fa2011-10-31 20:21:18 +00001Introduction
2============
3
4This document descibes a collection of device-mapper targets that
5between them implement thin-provisioning and snapshots.
6
7The main highlight of this implementation, compared to the previous
8implementation of snapshots, is that it allows many virtual devices to
9be stored on the same data volume. This simplifies administration and
10allows the sharing of data between volumes, thus reducing disk usage.
11
12Another significant feature is support for an arbitrary depth of
13recursive snapshots (snapshots of snapshots of snapshots ...). The
14previous implementation of snapshots did this by chaining together
15lookup tables, and so performance was O(depth). This new
16implementation uses a single data structure to avoid this degradation
17with depth. Fragmentation may still be an issue, however, in some
18scenarios.
19
20Metadata is stored on a separate device from data, giving the
21administrator some freedom, for example to:
22
23- Improve metadata resilience by storing metadata on a mirrored volume
24 but data on a non-mirrored one.
25
26- Improve performance by storing the metadata on SSD.
27
28Status
29======
30
31These targets are very much still in the EXPERIMENTAL state. Please
32do not yet rely on them in production. But do experiment and offer us
33feedback. Different use cases will have different performance
34characteristics, for example due to fragmentation of the data volume.
35
36If you find this software is not performing as expected please mail
37dm-devel@redhat.com with details and we'll try our best to improve
38things for you.
39
40Userspace tools for checking and repairing the metadata are under
41development.
42
43Cookbook
44========
45
46This section describes some quick recipes for using thin provisioning.
47They use the dmsetup program to control the device-mapper driver
48directly. End users will be advised to use a higher-level volume
49manager such as LVM2 once support has been added.
50
51Pool device
52-----------
53
54The pool device ties together the metadata volume and the data volume.
55It maps I/O linearly to the data volume and updates the metadata via
56two mechanisms:
57
58- Function calls from the thin targets
59
60- Device-mapper 'messages' from userspace which control the creation of new
61 virtual devices amongst other things.
62
63Setting up a fresh pool device
64------------------------------
65
66Setting up a pool device requires a valid metadata device, and a
67data device. If you do not have an existing metadata device you can
68make one by zeroing the first 4k to indicate empty metadata.
69
70 dd if=/dev/zero of=$metadata_dev bs=4096 count=1
71
72The amount of metadata you need will vary according to how many blocks
73are shared between thin devices (i.e. through snapshots). If you have
74less sharing than average you'll need a larger-than-average metadata device.
75
76As a guide, we suggest you calculate the number of bytes to use in the
77metadata device as 48 * $data_dev_size / $data_block_size but round it up
78to 2MB if the answer is smaller. The largest size supported is 16GB.
79
80If you're creating large numbers of snapshots which are recording large
81amounts of change, you may need find you need to increase this.
82
83Reloading a pool table
84----------------------
85
86You may reload a pool's table, indeed this is how the pool is resized
87if it runs out of space. (N.B. While specifying a different metadata
88device when reloading is not forbidden at the moment, things will go
89wrong if it does not route I/O to exactly the same on-disk location as
90previously.)
91
92Using an existing pool device
93-----------------------------
94
95 dmsetup create pool \
96 --table "0 20971520 thin-pool $metadata_dev $data_dev \
97 $data_block_size $low_water_mark"
98
99$data_block_size gives the smallest unit of disk space that can be
100allocated at a time expressed in units of 512-byte sectors. People
101primarily interested in thin provisioning may want to use a value such
102as 1024 (512KB). People doing lots of snapshotting may want a smaller value
103such as 128 (64KB). If you are not zeroing newly-allocated data,
104a larger $data_block_size in the region of 256000 (128MB) is suggested.
105$data_block_size must be the same for the lifetime of the
106metadata device.
107
108$low_water_mark is expressed in blocks of size $data_block_size. If
109free space on the data device drops below this level then a dm event
110will be triggered which a userspace daemon should catch allowing it to
111extend the pool device. Only one such event will be sent.
112Resuming a device with a new table itself triggers an event so the
113userspace daemon can use this to detect a situation where a new table
114already exceeds the threshold.
115
116Thin provisioning
117-----------------
118
119i) Creating a new thinly-provisioned volume.
120
121 To create a new thinly- provisioned volume you must send a message to an
122 active pool device, /dev/mapper/pool in this example.
123
124 dmsetup message /dev/mapper/pool 0 "create_thin 0"
125
126 Here '0' is an identifier for the volume, a 24-bit number. It's up
127 to the caller to allocate and manage these identifiers. If the
128 identifier is already in use, the message will fail with -EEXIST.
129
130ii) Using a thinly-provisioned volume.
131
132 Thinly-provisioned volumes are activated using the 'thin' target:
133
134 dmsetup create thin --table "0 2097152 thin /dev/mapper/pool 0"
135
136 The last parameter is the identifier for the thinp device.
137
138Internal snapshots
139------------------
140
141i) Creating an internal snapshot.
142
143 Snapshots are created with another message to the pool.
144
145 N.B. If the origin device that you wish to snapshot is active, you
146 must suspend it before creating the snapshot to avoid corruption.
147 This is NOT enforced at the moment, so please be careful!
148
149 dmsetup suspend /dev/mapper/thin
150 dmsetup message /dev/mapper/pool 0 "create_snap 1 0"
151 dmsetup resume /dev/mapper/thin
152
153 Here '1' is the identifier for the volume, a 24-bit number. '0' is the
154 identifier for the origin device.
155
156ii) Using an internal snapshot.
157
158 Once created, the user doesn't have to worry about any connection
159 between the origin and the snapshot. Indeed the snapshot is no
160 different from any other thinly-provisioned device and can be
161 snapshotted itself via the same method. It's perfectly legal to
162 have only one of them active, and there's no ordering requirement on
163 activating or removing them both. (This differs from conventional
164 device-mapper snapshots.)
165
166 Activate it exactly the same way as any other thinly-provisioned volume:
167
168 dmsetup create snap --table "0 2097152 thin /dev/mapper/pool 1"
169
170Deactivation
171------------
172
173All devices using a pool must be deactivated before the pool itself
174can be.
175
176 dmsetup remove thin
177 dmsetup remove snap
178 dmsetup remove pool
179
180Reference
181=========
182
183'thin-pool' target
184------------------
185
186i) Constructor
187
188 thin-pool <metadata dev> <data dev> <data block size (sectors)> \
189 <low water mark (blocks)> [<number of feature args> [<arg>]*]
190
191 Optional feature arguments:
192 - 'skip_block_zeroing': skips the zeroing of newly-provisioned blocks.
193
194 Data block size must be between 64KB (128 sectors) and 1GB
195 (2097152 sectors) inclusive.
196
197
198ii) Status
199
200 <transaction id> <used metadata blocks>/<total metadata blocks>
201 <used data blocks>/<total data blocks> <held metadata root>
202
203
204 transaction id:
205 A 64-bit number used by userspace to help synchronise with metadata
206 from volume managers.
207
208 used data blocks / total data blocks
209 If the number of free blocks drops below the pool's low water mark a
210 dm event will be sent to userspace. This event is edge-triggered and
211 it will occur only once after each resume so volume manager writers
212 should register for the event and then check the target's status.
213
214 held metadata root:
215 The location, in sectors, of the metadata root that has been
216 'held' for userspace read access. '-' indicates there is no
217 held root. This feature is not yet implemented so '-' is
218 always returned.
219
220iii) Messages
221
222 create_thin <dev id>
223
224 Create a new thinly-provisioned device.
225 <dev id> is an arbitrary unique 24-bit identifier chosen by
226 the caller.
227
228 create_snap <dev id> <origin id>
229
230 Create a new snapshot of another thinly-provisioned device.
231 <dev id> is an arbitrary unique 24-bit identifier chosen by
232 the caller.
233 <origin id> is the identifier of the thinly-provisioned device
234 of which the new device will be a snapshot.
235
236 delete <dev id>
237
238 Deletes a thin device. Irreversible.
239
240 trim <dev id> <new size in sectors>
241
242 Delete mappings from the end of a thin device. Irreversible.
243 You might want to use this if you're reducing the size of
244 your thinly-provisioned device. In many cases, due to the
245 sharing of blocks between devices, it is not possible to
246 determine in advance how much space 'trim' will release. (In
247 future a userspace tool might be able to perform this
248 calculation.)
249
250 set_transaction_id <current id> <new id>
251
252 Userland volume managers, such as LVM, need a way to
253 synchronise their external metadata with the internal metadata of the
254 pool target. The thin-pool target offers to store an
255 arbitrary 64-bit transaction id and return it on the target's
256 status line. To avoid races you must provide what you think
257 the current transaction id is when you change it with this
258 compare-and-swap message.
259
260'thin' target
261-------------
262
263i) Constructor
264
265 thin <pool dev> <dev id>
266
267 pool dev:
268 the thin-pool device, e.g. /dev/mapper/my_pool or 253:0
269
270 dev id:
271 the internal device identifier of the device to be
272 activated.
273
274The pool doesn't store any size against the thin devices. If you
275load a thin target that is smaller than you've been using previously,
276then you'll have no access to blocks mapped beyond the end. If you
277load a target that is bigger than before, then extra blocks will be
278provisioned as and when needed.
279
280If you wish to reduce the size of your thin device and potentially
281regain some space then send the 'trim' message to the pool.
282
283ii) Status
284
285 <nr mapped sectors> <highest mapped sector>