blob: d3ca8af21a31adf4eed39eade2067a48bdb8dda6 [file] [log] [blame]
Joe Thornberf2836352013-03-01 22:45:51 +00001Guidance for writing policies
2=============================
3
4Try to keep transactionality out of it. The core is careful to
5avoid asking about anything that is migrating. This is a pain, but
6makes it easier to write the policies.
7
8Mappings are loaded into the policy at construction time.
9
10Every bio that is mapped by the target is referred to the policy.
11The policy can return a simple HIT or MISS or issue a migration.
12
13Currently there's no way for the policy to issue background work,
Mike Snitzer492d48d2016-04-20 21:11:25 -040014e.g. to start writing back dirty blocks that are going to be evicted
Joe Thornberf2836352013-03-01 22:45:51 +000015soon.
16
17Because we map bios, rather than requests it's easy for the policy
18to get fooled by many small bios. For this reason the core target
19issues periodic ticks to the policy. It's suggested that the policy
20doesn't update states (eg, hit counts) for a block more than once
21for each tick. The core ticks by watching bios complete, and so
22trying to see when the io scheduler has let the ios run.
23
24
25Overview of supplied cache replacement policies
26===============================================
27
Mike Snitzerbccab6a2015-06-17 11:43:38 -040028multiqueue (mq)
29---------------
Joe Thornberf2836352013-03-01 22:45:51 +000030
Joe Thornber9ed84692016-02-10 10:18:10 +000031This policy is now an alias for smq (see below).
Joe Thornberf2836352013-03-01 22:45:51 +000032
Joe Thornber9ed84692016-02-10 10:18:10 +000033The following tunables are accepted, but have no effect:
Joe Thornber01911c12013-10-24 14:10:28 -040034
Joe Thornber78e03d62013-12-09 12:53:05 +000035 'sequential_threshold <#nr_sequential_ios>'
36 'random_threshold <#nr_random_ios>'
37 'read_promote_adjustment <value>'
38 'write_promote_adjustment <value>'
39 'discard_promote_adjustment <value>'
Joe Thornberf2836352013-03-01 22:45:51 +000040
Mike Snitzerbccab6a2015-06-17 11:43:38 -040041Stochastic multiqueue (smq)
42---------------------------
43
44This policy is the default.
45
46The stochastic multi-queue (smq) policy addresses some of the problems
47with the multiqueue (mq) policy.
48
49The smq policy (vs mq) offers the promise of less memory utilization,
50improved performance and increased adaptability in the face of changing
Mike Snitzer492d48d2016-04-20 21:11:25 -040051workloads. smq also does not have any cumbersome tuning knobs.
Mike Snitzerbccab6a2015-06-17 11:43:38 -040052
53Users may switch from "mq" to "smq" simply by appropriately reloading a
54DM table that is using the cache target. Doing so will cause all of the
55mq policy's hints to be dropped. Also, performance of the cache may
56degrade slightly until smq recalculates the origin device's hotspots
57that should be cached.
58
59Memory usage:
Mike Snitzer492d48d2016-04-20 21:11:25 -040060The mq policy used a lot of memory; 88 bytes per cache block on a 64
Mike Snitzerbccab6a2015-06-17 11:43:38 -040061bit machine.
62
Mike Snitzer492d48d2016-04-20 21:11:25 -040063smq uses 28bit indexes to implement it's data structures rather than
Mike Snitzerbccab6a2015-06-17 11:43:38 -040064pointers. It avoids storing an explicit hit count for each block. It
Mike Snitzer492d48d2016-04-20 21:11:25 -040065has a 'hotspot' queue, rather than a pre-cache, which uses a quarter of
Mike Snitzerbccab6a2015-06-17 11:43:38 -040066the entries (each hotspot block covers a larger area than a single
67cache block).
68
Mike Snitzer492d48d2016-04-20 21:11:25 -040069All this means smq uses ~25bytes per cache block. Still a lot of
Mike Snitzerbccab6a2015-06-17 11:43:38 -040070memory, but a substantial improvement nontheless.
71
72Level balancing:
Mike Snitzer492d48d2016-04-20 21:11:25 -040073mq placed entries in different levels of the multiqueue structures
74based on their hit count (~ln(hit count)). This meant the bottom
75levels generally had the most entries, and the top ones had very
76few. Having unbalanced levels like this reduced the efficacy of the
Mike Snitzerbccab6a2015-06-17 11:43:38 -040077multiqueue.
78
Mike Snitzer492d48d2016-04-20 21:11:25 -040079smq does not maintain a hit count, instead it swaps hit entries with
80the least recently used entry from the level above. The overall
Mike Snitzerbccab6a2015-06-17 11:43:38 -040081ordering being a side effect of this stochastic process. With this
82scheme we can decide how many entries occupy each multiqueue level,
83resulting in better promotion/demotion decisions.
84
85Adaptability:
Mike Snitzer492d48d2016-04-20 21:11:25 -040086The mq policy maintained a hit count for each cache block. For a
Mike Snitzerbccab6a2015-06-17 11:43:38 -040087different block to get promoted to the cache it's hit count has to
Mike Snitzer492d48d2016-04-20 21:11:25 -040088exceed the lowest currently in the cache. This meant it could take a
Mike Snitzerbccab6a2015-06-17 11:43:38 -040089long time for the cache to adapt between varying IO patterns.
Mike Snitzerbccab6a2015-06-17 11:43:38 -040090
Mike Snitzer492d48d2016-04-20 21:11:25 -040091smq doesn't maintain hit counts, so a lot of this problem just goes
Mike Snitzerbccab6a2015-06-17 11:43:38 -040092away. In addition it tracks performance of the hotspot queue, which
93is used to decide which blocks to promote. If the hotspot queue is
94performing badly then it starts moving entries more quickly between
95levels. This lets it adapt to new IO patterns very quickly.
96
97Performance:
Mike Snitzer492d48d2016-04-20 21:11:25 -040098Testing smq shows substantially better performance than mq.
Mike Snitzerbccab6a2015-06-17 11:43:38 -040099
Heinz Mauelshagen8735a812013-03-01 22:45:52 +0000100cleaner
101-------
102
103The cleaner writes back all dirty blocks in a cache to decommission it.
104
Joe Thornberf2836352013-03-01 22:45:51 +0000105Examples
106========
107
108The syntax for a table is:
109 cache <metadata dev> <cache dev> <origin dev> <block size>
110 <#feature_args> [<feature arg>]*
111 <policy> <#policy_args> [<policy arg>]*
112
113The syntax to send a message using the dmsetup command is:
114 dmsetup message <mapped device> 0 sequential_threshold 1024
115 dmsetup message <mapped device> 0 random_threshold 8
116
117Using dmsetup:
118 dmsetup create blah --table "0 268435456 cache /dev/sdb /dev/sdc \
119 /dev/sdd 512 0 mq 4 sequential_threshold 1024 random_threshold 8"
120 creates a 128GB large mapped device named 'blah' with the
121 sequential threshold set to 1024 and the random_threshold set to 8.