blob: 68c0f517c60edb1ab7a61e990720c7c8f097bb16 [file] [log] [blame]
Joe Thornberc6b4fcb2013-03-01 22:45:51 +00001Introduction
2============
3
4dm-cache is a device mapper target written by Joe Thornber, Heinz
5Mauelshagen, and Mike Snitzer.
6
7It aims to improve performance of a block device (eg, a spindle) by
8dynamically migrating some of its data to a faster, smaller device
9(eg, an SSD).
10
11This device-mapper solution allows us to insert this caching at
12different levels of the dm stack, for instance above the data device for
13a thin-provisioning pool. Caching solutions that are integrated more
14closely with the virtual memory system should give better performance.
15
16The target reuses the metadata library used in the thin-provisioning
17library.
18
19The decision as to what data to migrate and when is left to a plug-in
20policy module. Several of these have been written as we experiment,
21and we hope other people will contribute others for specific io
22scenarios (eg. a vm image server).
23
24Glossary
25========
26
27 Migration - Movement of the primary copy of a logical block from one
28 device to the other.
29 Promotion - Migration from slow device to fast device.
30 Demotion - Migration from fast device to slow device.
31
32The origin device always contains a copy of the logical block, which
33may be out of date or kept in sync with the copy on the cache device
34(depending on policy).
35
36Design
37======
38
39Sub-devices
40-----------
41
42The target is constructed by passing three devices to it (along with
43other parameters detailed later):
44
451. An origin device - the big, slow one.
46
472. A cache device - the small, fast one.
48
493. A small metadata device - records which blocks are in the cache,
50 which are dirty, and extra hints for use by the policy object.
51 This information could be put on the cache device, but having it
52 separate allows the volume manager to configure it differently,
Mike Snitzer66bb2642013-08-16 10:54:20 -040053 e.g. as a mirror for extra robustness. This metadata device may only
54 be used by a single cache device.
Joe Thornberc6b4fcb2013-03-01 22:45:51 +000055
56Fixed block size
57----------------
58
59The origin is divided up into blocks of a fixed size. This block size
60is configurable when you first create the cache. Typically we've been
Mike Snitzer05473042013-08-16 10:54:19 -040061using block sizes of 256KB - 1024KB. The block size must be between 64
62(32KB) and 2097152 (1GB) and a multiple of 64 (32KB).
Joe Thornberc6b4fcb2013-03-01 22:45:51 +000063
64Having a fixed block size simplifies the target a lot. But it is
65something of a compromise. For instance, a small part of a block may be
66getting hit a lot, yet the whole block will be promoted to the cache.
67So large block sizes are bad because they waste cache space. And small
68block sizes are bad because they increase the amount of metadata (both
69in core and on disk).
70
Joe Thornber2ee57d52013-10-24 14:10:29 -040071Cache operating modes
72---------------------
Joe Thornberc6b4fcb2013-03-01 22:45:51 +000073
Joe Thornber2ee57d52013-10-24 14:10:29 -040074The cache has three operating modes: writeback, writethrough and
75passthrough.
Joe Thornberc6b4fcb2013-03-01 22:45:51 +000076
77If writeback, the default, is selected then a write to a block that is
78cached will go only to the cache and the block will be marked dirty in
79the metadata.
80
81If writethrough is selected then a write to a cached block will not
82complete until it has hit both the origin and cache devices. Clean
83blocks should remain clean.
84
Joe Thornber2ee57d52013-10-24 14:10:29 -040085If passthrough is selected, useful when the cache contents are not known
86to be coherent with the origin device, then all reads are served from
87the origin device (all reads miss the cache) and all writes are
88forwarded to the origin device; additionally, write hits cause cache
Mike Snitzer7b6b2bc2013-11-12 12:17:43 -050089block invalidates. To enable passthrough mode the cache must be clean.
90Passthrough mode allows a cache device to be activated without having to
91worry about coherency. Coherency that exists is maintained, although
92the cache will gradually cool as writes take place. If the coherency of
93the cache can later be verified, or established through use of the
94"invalidate_cblocks" message, the cache device can be transitioned to
95writethrough or writeback mode while still warm. Otherwise, the cache
96contents can be discarded prior to transitioning to the desired
97operating mode.
Joe Thornber2ee57d52013-10-24 14:10:29 -040098
Joe Thornberc6b4fcb2013-03-01 22:45:51 +000099A simple cleaner policy is provided, which will clean (write back) all
Mike Snitzer7b6b2bc2013-11-12 12:17:43 -0500100dirty blocks in a cache. Useful for decommissioning a cache or when
101shrinking a cache. Shrinking the cache's fast device requires all cache
102blocks, in the area of the cache being removed, to be clean. If the
103area being removed from the cache still contains dirty blocks the resize
104will fail. Care must be taken to never reduce the volume used for the
105cache's fast device until the cache is clean. This is of particular
106importance if writeback mode is used. Writethrough and passthrough
107modes already maintain a clean cache. Future support to partially clean
108the cache, above a specified threshold, will allow for keeping the cache
109warm and in writeback mode during resize.
Joe Thornberc6b4fcb2013-03-01 22:45:51 +0000110
111Migration throttling
112--------------------
113
114Migrating data between the origin and cache device uses bandwidth.
115The user can set a throttle to prevent more than a certain amount of
Anatol Pomozovf884ab12013-05-08 16:56:16 -0700116migration occurring at any one time. Currently we're not taking any
Joe Thornberc6b4fcb2013-03-01 22:45:51 +0000117account of normal io traffic going to the devices. More work needs
118doing here to avoid migrating during those peak io moments.
119
120For the time being, a message "migration_threshold <#sectors>"
121can be used to set the maximum number of sectors being migrated,
122the default being 204800 sectors (or 100MB).
123
124Updating on-disk metadata
125-------------------------
126
Mike Snitzer07f2b6e2014-02-14 11:58:41 -0500127On-disk metadata is committed every time a FLUSH or FUA bio is written.
128If no such requests are made then commits will occur every second. This
129means the cache behaves like a physical disk that has a volatile write
130cache. If power is lost you may lose some recent writes. The metadata
131should always be consistent in spite of any crash.
Joe Thornberc6b4fcb2013-03-01 22:45:51 +0000132
133The 'dirty' state for a cache block changes far too frequently for us
134to keep updating it on the fly. So we treat it as a hint. In normal
135operation it will be written when the dm device is suspended. If the
136system crashes all cache blocks will be assumed dirty when restarted.
137
138Per-block policy hints
139----------------------
140
141Policy plug-ins can store a chunk of data per cache block. It's up to
142the policy how big this chunk is, but it should be kept small. Like the
143dirty flags this data is lost if there's a crash so a safe fallback
144value should always be possible.
145
146For instance, the 'mq' policy, which is currently the default policy,
147uses this facility to store the hit count of the cache blocks. If
148there's a crash this information will be lost, which means the cache
149may be less efficient until those hit counts are regenerated.
150
151Policy hints affect performance, not correctness.
152
153Policy messaging
154----------------
155
156Policies will have different tunables, specific to each one, so we
157need a generic way of getting and setting these. Device-mapper
158messages are used. Refer to cache-policies.txt.
159
160Discard bitset resolution
161-------------------------
162
163We can avoid copying data during migration if we know the block has
164been discarded. A prime example of this is when mkfs discards the
165whole block device. We store a bitset tracking the discard state of
166blocks. However, we allow this bitset to have a different block size
167from the cache blocks. This is because we need to track the discard
168state for all of the origin device (compare with the dirty bitset
169which is just for the smaller cache device).
170
171Target interface
172================
173
174Constructor
175-----------
176
177 cache <metadata dev> <cache dev> <origin dev> <block size>
178 <#feature args> [<feature arg>]*
179 <policy> <#policy args> [policy args]*
180
181 metadata dev : fast device holding the persistent metadata
182 cache dev : fast device holding cached data blocks
183 origin dev : slow device holding original data blocks
184 block size : cache unit size in sectors
185
186 #feature args : number of feature arguments passed
Mike Snitzer7b6b2bc2013-11-12 12:17:43 -0500187 feature args : writethrough or passthrough (The default is writeback.)
Joe Thornberc6b4fcb2013-03-01 22:45:51 +0000188
189 policy : the replacement policy to use
190 #policy args : an even number of arguments corresponding to
191 key/value pairs passed to the policy
192 policy args : key/value pairs passed to the policy
193 E.g. 'sequential_threshold 1024'
194 See cache-policies.txt for details.
195
196Optional feature arguments are:
197 writethrough : write through caching that prohibits cache block
198 content from being different from origin block content.
199 Without this argument, the default behaviour is to write
200 back cache block contents later for performance reasons,
201 so they may differ from the corresponding origin blocks.
202
Mike Snitzer7b6b2bc2013-11-12 12:17:43 -0500203 passthrough : a degraded mode useful for various cache coherency
204 situations (e.g., rolling back snapshots of
205 underlying storage). Reads and writes always go to
206 the origin. If a write goes to a cached origin
207 block, then the cache block is invalidated.
208 To enable passthrough mode the cache must be clean.
209
Joe Thornberc6b4fcb2013-03-01 22:45:51 +0000210A policy called 'default' is always registered. This is an alias for
211the policy we currently think is giving best all round performance.
212
213As the default policy could vary between kernels, if you are relying on
214the characteristics of a specific policy, always request it by name.
215
216Status
217------
218
Mike Snitzer6a388612014-01-09 16:04:12 -0500219<metadata block size> <#used metadata blocks>/<#total metadata blocks>
220<cache block size> <#used cache blocks>/<#total cache blocks>
221<#read hits> <#read misses> <#write hits> <#write misses>
222<#demotions> <#promotions> <#dirty> <#features> <features>*
Mike Snitzer2e68c4e2014-01-15 21:06:55 -0500223<#core args> <core args>* <policy name> <#policy args> <policy args>*
Joe Thornberc6b4fcb2013-03-01 22:45:51 +0000224
Mike Snitzer6a388612014-01-09 16:04:12 -0500225metadata block size : Fixed block size for each metadata block in
226 sectors
227#used metadata blocks : Number of metadata blocks used
228#total metadata blocks : Total number of metadata blocks
229cache block size : Configurable block size for the cache device
230 in sectors
231#used cache blocks : Number of blocks resident in the cache
232#total cache blocks : Total number of cache blocks
233#read hits : Number of times a READ bio has been mapped
Joe Thornberc6b4fcb2013-03-01 22:45:51 +0000234 to the cache
Mike Snitzer6a388612014-01-09 16:04:12 -0500235#read misses : Number of times a READ bio has been mapped
Joe Thornberc6b4fcb2013-03-01 22:45:51 +0000236 to the origin
Mike Snitzer6a388612014-01-09 16:04:12 -0500237#write hits : Number of times a WRITE bio has been mapped
Joe Thornberc6b4fcb2013-03-01 22:45:51 +0000238 to the cache
Mike Snitzer6a388612014-01-09 16:04:12 -0500239#write misses : Number of times a WRITE bio has been
Joe Thornberc6b4fcb2013-03-01 22:45:51 +0000240 mapped to the origin
Mike Snitzer6a388612014-01-09 16:04:12 -0500241#demotions : Number of times a block has been removed
Joe Thornberc6b4fcb2013-03-01 22:45:51 +0000242 from the cache
Mike Snitzer6a388612014-01-09 16:04:12 -0500243#promotions : Number of times a block has been moved to
Joe Thornberc6b4fcb2013-03-01 22:45:51 +0000244 the cache
Mike Snitzer6a388612014-01-09 16:04:12 -0500245#dirty : Number of blocks in the cache that differ
Joe Thornberc6b4fcb2013-03-01 22:45:51 +0000246 from the origin
Mike Snitzer6a388612014-01-09 16:04:12 -0500247#feature args : Number of feature args to follow
248feature args : 'writethrough' (optional)
249#core args : Number of core arguments (must be even)
250core args : Key/value pairs for tuning the core
Joe Thornberc6b4fcb2013-03-01 22:45:51 +0000251 e.g. migration_threshold
Mike Snitzer2e68c4e2014-01-15 21:06:55 -0500252policy name : Name of the policy
Mike Snitzer6a388612014-01-09 16:04:12 -0500253#policy args : Number of policy arguments to follow (must be even)
254policy args : Key/value pairs
255 e.g. sequential_threshold
Joe Thornberc6b4fcb2013-03-01 22:45:51 +0000256
257Messages
258--------
259
260Policies will have different tunables, specific to each one, so we
261need a generic way of getting and setting these. Device-mapper
262messages are used. (A sysfs interface would also be possible.)
263
264The message format is:
265
266 <key> <value>
267
268E.g.
269 dmsetup message my_cache 0 sequential_threshold 1024
270
Joe Thornber65790ff2013-11-08 16:39:50 +0000271
272Invalidation is removing an entry from the cache without writing it
273back. Cache blocks can be invalidated via the invalidate_cblocks
Mike Snitzer7b6b2bc2013-11-12 12:17:43 -0500274message, which takes an arbitrary number of cblock ranges. Each cblock
Mike Snitzer83f539e2013-11-26 11:03:54 -0500275range's end value is "one past the end", meaning 5-10 expresses a range
276of values from 5 to 9. Each cblock must be expressed as a decimal
277value, in the future a variant message that takes cblock ranges
278expressed in hexidecimal may be needed to better support efficient
279invalidation of larger caches. The cache must be in passthrough mode
280when invalidate_cblocks is used.
Joe Thornber65790ff2013-11-08 16:39:50 +0000281
282 invalidate_cblocks [<cblock>|<cblock begin>-<cblock end>]*
283
284E.g.
285 dmsetup message my_cache 0 invalidate_cblocks 2345 3456-4567 5678-6789
286
Joe Thornberc6b4fcb2013-03-01 22:45:51 +0000287Examples
288========
289
290The test suite can be found here:
291
Joe Thornber65790ff2013-11-08 16:39:50 +0000292https://github.com/jthornber/device-mapper-test-suite
Joe Thornberc6b4fcb2013-03-01 22:45:51 +0000293
294dmsetup create my_cache --table '0 41943040 cache /dev/mapper/metadata \
295 /dev/mapper/ssd /dev/mapper/origin 512 1 writeback default 0'
296dmsetup create my_cache --table '0 41943040 cache /dev/mapper/metadata \
297 /dev/mapper/ssd /dev/mapper/origin 1024 1 writeback \
298 mq 4 sequential_threshold 1024 random_threshold 8'