blob: e8cdf7241b66b3b413b0253042b4d4a54700dc4e [file] [log] [blame]
Joe Thornberc6b4fcb2013-03-01 22:45:51 +00001Introduction
2============
3
4dm-cache is a device mapper target written by Joe Thornber, Heinz
5Mauelshagen, and Mike Snitzer.
6
7It aims to improve performance of a block device (eg, a spindle) by
8dynamically migrating some of its data to a faster, smaller device
9(eg, an SSD).
10
11This device-mapper solution allows us to insert this caching at
12different levels of the dm stack, for instance above the data device for
13a thin-provisioning pool. Caching solutions that are integrated more
14closely with the virtual memory system should give better performance.
15
16The target reuses the metadata library used in the thin-provisioning
17library.
18
19The decision as to what data to migrate and when is left to a plug-in
20policy module. Several of these have been written as we experiment,
21and we hope other people will contribute others for specific io
22scenarios (eg. a vm image server).
23
24Glossary
25========
26
27 Migration - Movement of the primary copy of a logical block from one
28 device to the other.
29 Promotion - Migration from slow device to fast device.
30 Demotion - Migration from fast device to slow device.
31
32The origin device always contains a copy of the logical block, which
33may be out of date or kept in sync with the copy on the cache device
34(depending on policy).
35
36Design
37======
38
39Sub-devices
40-----------
41
42The target is constructed by passing three devices to it (along with
43other parameters detailed later):
44
451. An origin device - the big, slow one.
46
472. A cache device - the small, fast one.
48
493. A small metadata device - records which blocks are in the cache,
50 which are dirty, and extra hints for use by the policy object.
51 This information could be put on the cache device, but having it
52 separate allows the volume manager to configure it differently,
53 e.g. as a mirror for extra robustness.
54
55Fixed block size
56----------------
57
58The origin is divided up into blocks of a fixed size. This block size
59is configurable when you first create the cache. Typically we've been
60using block sizes of 256k - 1024k.
61
62Having a fixed block size simplifies the target a lot. But it is
63something of a compromise. For instance, a small part of a block may be
64getting hit a lot, yet the whole block will be promoted to the cache.
65So large block sizes are bad because they waste cache space. And small
66block sizes are bad because they increase the amount of metadata (both
67in core and on disk).
68
69Writeback/writethrough
70----------------------
71
72The cache has two modes, writeback and writethrough.
73
74If writeback, the default, is selected then a write to a block that is
75cached will go only to the cache and the block will be marked dirty in
76the metadata.
77
78If writethrough is selected then a write to a cached block will not
79complete until it has hit both the origin and cache devices. Clean
80blocks should remain clean.
81
82A simple cleaner policy is provided, which will clean (write back) all
83dirty blocks in a cache. Useful for decommissioning a cache.
84
85Migration throttling
86--------------------
87
88Migrating data between the origin and cache device uses bandwidth.
89The user can set a throttle to prevent more than a certain amount of
Anatol Pomozovf884ab12013-05-08 16:56:16 -070090migration occurring at any one time. Currently we're not taking any
Joe Thornberc6b4fcb2013-03-01 22:45:51 +000091account of normal io traffic going to the devices. More work needs
92doing here to avoid migrating during those peak io moments.
93
94For the time being, a message "migration_threshold <#sectors>"
95can be used to set the maximum number of sectors being migrated,
96the default being 204800 sectors (or 100MB).
97
98Updating on-disk metadata
99-------------------------
100
101On-disk metadata is committed every time a REQ_SYNC or REQ_FUA bio is
102written. If no such requests are made then commits will occur every
103second. This means the cache behaves like a physical disk that has a
104write cache (the same is true of the thin-provisioning target). If
105power is lost you may lose some recent writes. The metadata should
106always be consistent in spite of any crash.
107
108The 'dirty' state for a cache block changes far too frequently for us
109to keep updating it on the fly. So we treat it as a hint. In normal
110operation it will be written when the dm device is suspended. If the
111system crashes all cache blocks will be assumed dirty when restarted.
112
113Per-block policy hints
114----------------------
115
116Policy plug-ins can store a chunk of data per cache block. It's up to
117the policy how big this chunk is, but it should be kept small. Like the
118dirty flags this data is lost if there's a crash so a safe fallback
119value should always be possible.
120
121For instance, the 'mq' policy, which is currently the default policy,
122uses this facility to store the hit count of the cache blocks. If
123there's a crash this information will be lost, which means the cache
124may be less efficient until those hit counts are regenerated.
125
126Policy hints affect performance, not correctness.
127
128Policy messaging
129----------------
130
131Policies will have different tunables, specific to each one, so we
132need a generic way of getting and setting these. Device-mapper
133messages are used. Refer to cache-policies.txt.
134
135Discard bitset resolution
136-------------------------
137
138We can avoid copying data during migration if we know the block has
139been discarded. A prime example of this is when mkfs discards the
140whole block device. We store a bitset tracking the discard state of
141blocks. However, we allow this bitset to have a different block size
142from the cache blocks. This is because we need to track the discard
143state for all of the origin device (compare with the dirty bitset
144which is just for the smaller cache device).
145
146Target interface
147================
148
149Constructor
150-----------
151
152 cache <metadata dev> <cache dev> <origin dev> <block size>
153 <#feature args> [<feature arg>]*
154 <policy> <#policy args> [policy args]*
155
156 metadata dev : fast device holding the persistent metadata
157 cache dev : fast device holding cached data blocks
158 origin dev : slow device holding original data blocks
159 block size : cache unit size in sectors
160
161 #feature args : number of feature arguments passed
162 feature args : writethrough. (The default is writeback.)
163
164 policy : the replacement policy to use
165 #policy args : an even number of arguments corresponding to
166 key/value pairs passed to the policy
167 policy args : key/value pairs passed to the policy
168 E.g. 'sequential_threshold 1024'
169 See cache-policies.txt for details.
170
171Optional feature arguments are:
172 writethrough : write through caching that prohibits cache block
173 content from being different from origin block content.
174 Without this argument, the default behaviour is to write
175 back cache block contents later for performance reasons,
176 so they may differ from the corresponding origin blocks.
177
178A policy called 'default' is always registered. This is an alias for
179the policy we currently think is giving best all round performance.
180
181As the default policy could vary between kernels, if you are relying on
182the characteristics of a specific policy, always request it by name.
183
184Status
185------
186
187<#used metadata blocks>/<#total metadata blocks> <#read hits> <#read misses>
188<#write hits> <#write misses> <#demotions> <#promotions> <#blocks in cache>
189<#dirty> <#features> <features>* <#core args> <core args>* <#policy args>
190<policy args>*
191
192#used metadata blocks : Number of metadata blocks used
193#total metadata blocks : Total number of metadata blocks
194#read hits : Number of times a READ bio has been mapped
195 to the cache
196#read misses : Number of times a READ bio has been mapped
197 to the origin
198#write hits : Number of times a WRITE bio has been mapped
199 to the cache
200#write misses : Number of times a WRITE bio has been
201 mapped to the origin
202#demotions : Number of times a block has been removed
203 from the cache
204#promotions : Number of times a block has been moved to
205 the cache
206#blocks in cache : Number of blocks resident in the cache
207#dirty : Number of blocks in the cache that differ
208 from the origin
209#feature args : Number of feature args to follow
210feature args : 'writethrough' (optional)
211#core args : Number of core arguments (must be even)
212core args : Key/value pairs for tuning the core
213 e.g. migration_threshold
214#policy args : Number of policy arguments to follow (must be even)
215policy args : Key/value pairs
216 e.g. 'sequential_threshold 1024
217
218Messages
219--------
220
221Policies will have different tunables, specific to each one, so we
222need a generic way of getting and setting these. Device-mapper
223messages are used. (A sysfs interface would also be possible.)
224
225The message format is:
226
227 <key> <value>
228
229E.g.
230 dmsetup message my_cache 0 sequential_threshold 1024
231
232Examples
233========
234
235The test suite can be found here:
236
237https://github.com/jthornber/thinp-test-suite
238
239dmsetup create my_cache --table '0 41943040 cache /dev/mapper/metadata \
240 /dev/mapper/ssd /dev/mapper/origin 512 1 writeback default 0'
241dmsetup create my_cache --table '0 41943040 cache /dev/mapper/metadata \
242 /dev/mapper/ssd /dev/mapper/origin 1024 1 writeback \
243 mq 4 sequential_threshold 1024 random_threshold 8'