blob: 2622115276aa028266ae376ff9116401409f433a [file] [log] [blame]
Balbir Singh00f0b822008-03-04 14:28:39 -08001Memory Resource Controller
2
Jörg Sommer67de0162011-06-15 13:00:47 -07003NOTE: The Memory Resource Controller has generically been referred to as the
4 memory controller in this document. Do not confuse memory controller
5 used here with the memory controller that is used in hardware.
Balbir Singh1b6df3a2008-02-07 00:13:46 -08006
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -07007(For editors)
8In this document:
9 When we mention a cgroup (cgroupfs's directory) with memory controller,
10 we call it "memory cgroup". When you see git-log and source code, you'll
11 see patch's title and function names tend to use "memcg".
12 In this document, we avoid using it.
Balbir Singh1b6df3a2008-02-07 00:13:46 -080013
Balbir Singh1b6df3a2008-02-07 00:13:46 -080014Benefits and Purpose of the memory controller
15
16The memory controller isolates the memory behaviour of a group of tasks
17from the rest of the system. The article on LWN [12] mentions some probable
18uses of the memory controller. The memory controller can be used to
19
20a. Isolate an application or a group of applications
Michael Kerrisk1939c552012-10-08 16:33:09 -070021 Memory-hungry applications can be isolated and limited to a smaller
Balbir Singh1b6df3a2008-02-07 00:13:46 -080022 amount of memory.
Michael Kerrisk1939c552012-10-08 16:33:09 -070023b. Create a cgroup with a limited amount of memory; this can be used
Balbir Singh1b6df3a2008-02-07 00:13:46 -080024 as a good alternative to booting with mem=XXXX.
25c. Virtualization solutions can control the amount of memory they want
26 to assign to a virtual machine instance.
27d. A CD/DVD burner could control the amount of memory used by the
28 rest of the system to ensure that burning does not fail due to lack
29 of available memory.
Michael Kerrisk1939c552012-10-08 16:33:09 -070030e. There are several other use cases; find one or use the controller just
Balbir Singh1b6df3a2008-02-07 00:13:46 -080031 for fun (to learn and hack on the VM subsystem).
32
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -070033Current Status: linux-2.6.34-mmotm(development version of 2010/April)
34
35Features:
36 - accounting anonymous pages, file caches, swap caches usage and limiting them.
Ying Han6252efc2012-04-12 12:49:10 -070037 - pages are linked to per-memcg LRU exclusively, and there is no global LRU.
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -070038 - optionally, memory+swap usage can be accounted and limited.
39 - hierarchical accounting
40 - soft limit
Michael Kerrisk1939c552012-10-08 16:33:09 -070041 - moving (recharging) account at moving a task is selectable.
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -070042 - usage threshold notifier
Anton Vorontsov70ddf632013-04-29 15:08:31 -070043 - memory pressure notifier
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -070044 - oom-killer disable knob and oom-notifier
45 - Root cgroup has no limit controls.
46
Michael Kerrisk1939c552012-10-08 16:33:09 -070047 Kernel memory support is a work in progress, and the current version provides
Glauber Costa65c64ce2011-12-22 01:02:27 +000048 basically functionality. (See Section 2.7)
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -070049
50Brief summary of control files.
51
52 tasks # attach a task(thread) and show list of threads
53 cgroup.procs # show list of processes
54 cgroup.event_control # an interface for event_fd()
Daisuke Nishimuraa111c962011-04-27 15:26:48 -070055 memory.usage_in_bytes # show current res_counter usage for memory
56 (See 5.5 for details)
57 memory.memsw.usage_in_bytes # show current res_counter usage for memory+Swap
58 (See 5.5 for details)
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -070059 memory.limit_in_bytes # set/show limit of memory usage
60 memory.memsw.limit_in_bytes # set/show limit of memory+Swap usage
61 memory.failcnt # show the number of memory usage hits limits
62 memory.memsw.failcnt # show the number of memory+Swap hits limits
63 memory.max_usage_in_bytes # show max memory usage recorded
Zhu Yanhaid66c1ce2012-01-12 17:18:24 -080064 memory.memsw.max_usage_in_bytes # show max memory+Swap usage recorded
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -070065 memory.soft_limit_in_bytes # set/show soft limit of memory usage
66 memory.stat # show various statistics
67 memory.use_hierarchy # set/show hierarchical account enabled
68 memory.force_empty # trigger forced move charge to parent
Anton Vorontsov70ddf632013-04-29 15:08:31 -070069 memory.pressure_level # set memory pressure notifications
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -070070 memory.swappiness # set/show swappiness parameter of vmscan
71 (See sysctl's vm.swappiness)
72 memory.move_charge_at_immigrate # set/show controls of moving charges
73 memory.oom_control # set/show oom controls.
Ying Han50c35e52011-06-15 15:08:16 -070074 memory.numa_stat # show the number of memory usage per numa node
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -070075
Glauber Costad5bdae72012-12-18 14:22:22 -080076 memory.kmem.limit_in_bytes # set/show hard limit for kernel memory
77 memory.kmem.usage_in_bytes # show current kernel memory allocation
78 memory.kmem.failcnt # show the number of kernel memory usage hits limits
79 memory.kmem.max_usage_in_bytes # show max kernel memory usage recorded
80
Glauber Costa3aaabe22011-12-11 21:47:06 +000081 memory.kmem.tcp.limit_in_bytes # set/show hard limit for tcp buf memory
Glauber Costa5a6dd342011-12-11 21:47:07 +000082 memory.kmem.tcp.usage_in_bytes # show current tcp buf memory allocation
Wanpeng Li05a73ed2012-07-31 16:43:21 -070083 memory.kmem.tcp.failcnt # show the number of tcp buf memory usage hits limits
84 memory.kmem.tcp.max_usage_in_bytes # show max tcp buf memory usage recorded
Glauber Costae5671df2011-12-11 21:47:01 +000085
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800861. History
87
88The memory controller has a long history. A request for comments for the memory
89controller was posted by Balbir Singh [1]. At the time the RFC was posted
90there were several implementations for memory control. The goal of the
91RFC was to build consensus and agreement for the minimal features required
92for memory control. The first RSS controller was posted by Balbir Singh[2]
93in Feb 2007. Pavel Emelianov [3][4][5] has since posted three versions of the
94RSS controller. At OLS, at the resource management BoF, everyone suggested
95that we handle both page cache and RSS together. Another request was raised
96to allow user space handling of OOM. The current memory controller is
97at version 6; it combines both mapped (RSS) and unmapped Page
98Cache Control [11].
99
1002. Memory Control
101
102Memory is a unique resource in the sense that it is present in a limited
103amount. If a task requires a lot of CPU processing, the task can spread
104its processing over a period of hours, days, months or years, but with
105memory, the same physical memory needs to be reused to accomplish the task.
106
107The memory controller implementation has been divided into phases. These
108are:
109
1101. Memory controller
1112. mlock(2) controller
1123. Kernel user memory accounting and slab control
1134. user mappings length controller
114
115The memory controller is the first controller developed.
116
1172.1. Design
118
119The core of the design is a counter called the res_counter. The res_counter
120tracks the current memory usage and limit of the group of processes associated
121with the controller. Each cgroup has a memory controller specific data
122structure (mem_cgroup) associated with it.
123
1242.2. Accounting
125
126 +--------------------+
127 | mem_cgroup |
128 | (res_counter) |
129 +--------------------+
130 / ^ \
131 / | \
132 +---------------+ | +---------------+
133 | mm_struct | |.... | mm_struct |
134 | | | | |
135 +---------------+ | +---------------+
136 |
137 + --------------+
138 |
139 +---------------+ +------+--------+
140 | page +----------> page_cgroup|
141 | | | |
142 +---------------+ +---------------+
143
144 (Figure 1: Hierarchy of Accounting)
145
146
147Figure 1 shows the important aspects of the controller
148
1491. Accounting happens per cgroup
1502. Each mm_struct knows about which cgroup it belongs to
1513. Each page has a pointer to the page_cgroup, which in turn knows the
152 cgroup it belongs to
153
Jeff Liu348b4652012-12-11 16:01:28 -0800154The accounting is done as follows: mem_cgroup_charge_common() is invoked to
155set up the necessary data structures and check if the cgroup that is being
156charged is over its limit. If it is, then reclaim is invoked on the cgroup.
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800157More details can be found in the reclaim section of this document.
158If everything goes well, a page meta-data-structure called page_cgroup is
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700159updated. page_cgroup has its own LRU on cgroup.
160(*) page_cgroup structure is allocated at boot/memory-hotplug time.
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800161
1622.2.1 Accounting details
163
KAMEZAWA Hiroyuki5b4e6552008-10-18 20:28:10 -0700164All mapped anon pages (RSS) and cache pages (Page Cache) are accounted.
Ying Han6252efc2012-04-12 12:49:10 -0700165Some pages which are never reclaimable and will not be on the LRU
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700166are not accounted. We just account pages under usual VM management.
KAMEZAWA Hiroyuki5b4e6552008-10-18 20:28:10 -0700167
168RSS pages are accounted at page_fault unless they've already been accounted
169for earlier. A file page will be accounted for as Page Cache when it's
170inserted into inode (radix-tree). While it's mapped into the page tables of
171processes, duplicate accounting is carefully avoided.
172
Michael Kerrisk1939c552012-10-08 16:33:09 -0700173An RSS page is unaccounted when it's fully unmapped. A PageCache page is
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700174unaccounted when it's removed from radix-tree. Even if RSS pages are fully
175unmapped (by kswapd), they may exist as SwapCache in the system until they
Michael Kerrisk1939c552012-10-08 16:33:09 -0700176are really freed. Such SwapCaches are also accounted.
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700177A swapped-in page is not accounted until it's mapped.
178
Michael Kerrisk1939c552012-10-08 16:33:09 -0700179Note: The kernel does swapin-readahead and reads multiple swaps at once.
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700180This means swapped-in pages may contain pages for other tasks than a task
181causing page fault. So, we avoid accounting at swap-in I/O.
KAMEZAWA Hiroyuki5b4e6552008-10-18 20:28:10 -0700182
183At page migration, accounting information is kept.
184
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700185Note: we just account pages-on-LRU because our purpose is to control amount
186of used pages; not-on-LRU pages tend to be out-of-control from VM view.
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800187
1882.3 Shared Page Accounting
189
190Shared pages are accounted on the basis of the first touch approach. The
191cgroup that first touches a page is accounted for the page. The principle
192behind this approach is that a cgroup that aggressively uses a shared
193page will eventually get charged for it (once it is uncharged from
194the cgroup that brought it in -- this will happen on memory pressure).
195
KAMEZAWA Hiroyuki4b913552012-05-29 15:06:51 -0700196But see section 8.2: when moving a task to another cgroup, its pages may
197be recharged to the new cgroup, if move_charge_at_immigrate has been chosen.
198
Paul Bolledf7c6b92013-03-25 23:59:16 +0100199Exception: If CONFIG_MEMCG_SWAP is not used.
KAMEZAWA Hiroyuki8c7c6e342009-01-07 18:08:00 -0800200When you do swapoff and make swapped-out pages of shmem(tmpfs) to
KAMEZAWA Hiroyukid13d1442009-01-07 18:07:56 -0800201be backed into memory in force, charges for pages are accounted against the
202caller of swapoff rather than the users of shmem.
203
Andrew Mortonc255a452012-07-31 16:43:02 -07002042.4 Swap Extension (CONFIG_MEMCG_SWAP)
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700205
KAMEZAWA Hiroyuki8c7c6e342009-01-07 18:08:00 -0800206Swap Extension allows you to record charge for swap. A swapped-in page is
207charged back to original page allocator if possible.
208
209When swap is accounted, following files are added.
210 - memory.memsw.usage_in_bytes.
211 - memory.memsw.limit_in_bytes.
212
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700213memsw means memory+swap. Usage of memory+swap is limited by
214memsw.limit_in_bytes.
KAMEZAWA Hiroyuki8c7c6e342009-01-07 18:08:00 -0800215
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700216Example: Assume a system with 4G of swap. A task which allocates 6G of memory
217(by mistake) under 2G memory limitation will use all swap.
218In this case, setting memsw.limit_in_bytes=3G will prevent bad use of swap.
Michael Kerrisk1939c552012-10-08 16:33:09 -0700219By using the memsw limit, you can avoid system OOM which can be caused by swap
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700220shortage.
221
222* why 'memory+swap' rather than swap.
KAMEZAWA Hiroyuki8c7c6e342009-01-07 18:08:00 -0800223The global LRU(kswapd) can swap out arbitrary pages. Swap-out means
224to move account from memory to swap...there is no change in usage of
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700225memory+swap. In other words, when we want to limit the usage of swap without
226affecting global LRU, memory+swap limit is better than just limiting swap from
Michael Kerrisk1939c552012-10-08 16:33:09 -0700227an OS point of view.
KAMEZAWA Hiroyuki8c7c6e342009-01-07 18:08:00 -0800228
KAMEZAWA Hiroyuki22a668d2009-06-17 16:27:19 -0700229* What happens when a cgroup hits memory.memsw.limit_in_bytes
Jörg Sommer67de0162011-06-15 13:00:47 -0700230When a cgroup hits memory.memsw.limit_in_bytes, it's useless to do swap-out
KAMEZAWA Hiroyuki22a668d2009-06-17 16:27:19 -0700231in this cgroup. Then, swap-out will not be done by cgroup routine and file
232caches are dropped. But as mentioned above, global LRU can do swapout memory
233from it for sanity of the system's memory management state. You can't forbid
234it by cgroup.
KAMEZAWA Hiroyuki8c7c6e342009-01-07 18:08:00 -0800235
2362.5 Reclaim
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800237
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700238Each cgroup maintains a per cgroup LRU which has the same structure as
239global VM. When a cgroup goes over its limit, we first try
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800240to reclaim memory from the cgroup so as to make space for the new
241pages that the cgroup has touched. If the reclaim is unsuccessful,
242an OOM routine is invoked to select and kill the bulkiest task in the
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700243cgroup. (See 10. OOM Control below.)
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800244
245The reclaim algorithm has not been modified for cgroups, except that
Michael Kerrisk1939c552012-10-08 16:33:09 -0700246pages that are selected for reclaiming come from the per-cgroup LRU
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800247list.
248
Balbir Singh4b3bde42009-09-23 15:56:32 -0700249NOTE: Reclaim does not work for the root cgroup, since we cannot set any
250limits on the root cgroup.
251
KAMEZAWA Hiroyukidaaf1e62010-03-10 15:22:32 -0800252Note2: When panic_on_oom is set to "2", the whole system will panic.
253
KAMEZAWA Hiroyuki9490ff22010-05-26 14:42:36 -0700254When oom event notifier is registered, event will be delivered.
255(See oom_control section)
256
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -07002572.6 Locking
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800258
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700259 lock_page_cgroup()/unlock_page_cgroup() should not be called under
260 mapping->tree_lock.
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800261
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700262 Other lock order is following:
263 PG_locked.
264 mm->page_table_lock
265 zone->lru_lock
266 lock_page_cgroup.
267 In many cases, just lock_page_cgroup() is called.
268 per-zone-per-cgroup LRU (cgroup's private LRU) is just guarded by
269 zone->lru_lock, it has no lock of its own.
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800270
Andrew Mortonc255a452012-07-31 16:43:02 -07002712.7 Kernel Memory Extension (CONFIG_MEMCG_KMEM)
Glauber Costae5671df2011-12-11 21:47:01 +0000272
273With the Kernel memory extension, the Memory Controller is able to limit
274the amount of kernel memory used by the system. Kernel memory is fundamentally
275different than user memory, since it can't be swapped out, which makes it
276possible to DoS the system by consuming too much of this precious resource.
277
Glauber Costad5bdae72012-12-18 14:22:22 -0800278Kernel memory won't be accounted at all until limit on a group is set. This
279allows for existing setups to continue working without disruption. The limit
280cannot be set if the cgroup have children, or if there are already tasks in the
281cgroup. Attempting to set the limit under those conditions will return -EBUSY.
282When use_hierarchy == 1 and a group is accounted, its children will
283automatically be accounted regardless of their limit value.
284
285After a group is first limited, it will be kept being accounted until it
286is removed. The memory limitation itself, can of course be removed by writing
287-1 to memory.kmem.limit_in_bytes. In this case, kmem will be accounted, but not
288limited.
289
Glauber Costae5671df2011-12-11 21:47:01 +0000290Kernel memory limits are not imposed for the root cgroup. Usage for the root
Glauber Costad5bdae72012-12-18 14:22:22 -0800291cgroup may or may not be accounted. The memory used is accumulated into
292memory.kmem.usage_in_bytes, or in a separate counter when it makes sense.
293(currently only for tcp).
294The main "kmem" counter is fed into the main counter, so kmem charges will
295also be visible from the user counter.
Glauber Costae5671df2011-12-11 21:47:01 +0000296
Glauber Costae5671df2011-12-11 21:47:01 +0000297Currently no soft limit is implemented for kernel memory. It is future work
298to trigger slab reclaim when those limits are reached.
299
3002.7.1 Current Kernel Memory resources accounted
301
Glauber Costad5bdae72012-12-18 14:22:22 -0800302* stack pages: every process consumes some stack pages. By accounting into
303kernel memory, we prevent new processes from being created when the kernel
304memory usage is too high.
305
Glauber Costa92e79342012-12-18 14:23:08 -0800306* slab pages: pages allocated by the SLAB or SLUB allocator are tracked. A copy
Anatol Pomozovf884ab12013-05-08 16:56:16 -0700307of each kmem_cache is created every time the cache is touched by the first time
Glauber Costa92e79342012-12-18 14:23:08 -0800308from inside the memcg. The creation is done lazily, so some objects can still be
309skipped while the cache is being created. All objects in a slab page should
310belong to the same memcg. This only fails to hold when a task is migrated to a
311different memcg during the page allocation by the cache.
312
Glauber Costae1aab162011-12-11 21:47:03 +0000313* sockets memory pressure: some sockets protocols have memory pressure
314thresholds. The Memory Controller allows them to be controlled individually
315per cgroup, instead of globally.
Glauber Costae5671df2011-12-11 21:47:01 +0000316
Glauber Costad1a4c0b2011-12-11 21:47:04 +0000317* tcp memory pressure: sockets memory pressure for the tcp protocol.
318
Glauber Costad5bdae72012-12-18 14:22:22 -08003192.7.3 Common use cases
320
321Because the "kmem" counter is fed to the main user counter, kernel memory can
322never be limited completely independently of user memory. Say "U" is the user
323limit, and "K" the kernel limit. There are three possible ways limits can be
324set:
325
326 U != 0, K = unlimited:
327 This is the standard memcg limitation mechanism already present before kmem
328 accounting. Kernel memory is completely ignored.
329
330 U != 0, K < U:
331 Kernel memory is a subset of the user memory. This setup is useful in
332 deployments where the total amount of memory per-cgroup is overcommited.
333 Overcommiting kernel memory limits is definitely not recommended, since the
334 box can still run out of non-reclaimable memory.
335 In this case, the admin could set up K so that the sum of all groups is
336 never greater than the total memory, and freely set U at the cost of his
337 QoS.
338
339 U != 0, K >= U:
340 Since kmem charges will also be fed to the user counter and reclaim will be
341 triggered for the cgroup for both kinds of memory. This setup gives the
342 admin a unified view of memory, and it is also useful for people who just
343 want to track kernel memory usage.
344
Balbir Singh1b6df3a2008-02-07 00:13:46 -08003453. User Interface
346
3470. Configuration
348
349a. Enable CONFIG_CGROUPS
350b. Enable CONFIG_RESOURCE_COUNTERS
Andrew Mortonc255a452012-07-31 16:43:02 -0700351c. Enable CONFIG_MEMCG
352d. Enable CONFIG_MEMCG_SWAP (to use swap extension)
Glauber Costad5bdae72012-12-18 14:22:22 -0800353d. Enable CONFIG_MEMCG_KMEM (to use kmem extension)
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800354
Jörg Sommerf6e07d32011-06-15 12:59:45 -07003551. Prepare the cgroups (see cgroups.txt, Why are cgroups needed?)
356# mount -t tmpfs none /sys/fs/cgroup
357# mkdir /sys/fs/cgroup/memory
358# mount -t cgroup none /sys/fs/cgroup/memory -o memory
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800359
3602. Make the new group and move bash into it
Jörg Sommerf6e07d32011-06-15 12:59:45 -0700361# mkdir /sys/fs/cgroup/memory/0
362# echo $$ > /sys/fs/cgroup/memory/0/tasks
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800363
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700364Since now we're in the 0 cgroup, we can alter the memory limit:
Jörg Sommerf6e07d32011-06-15 12:59:45 -0700365# echo 4M > /sys/fs/cgroup/memory/0/memory.limit_in_bytes
Balbir Singh0eea1032008-02-07 00:13:57 -0800366
367NOTE: We can use a suffix (k, K, m, M, g or G) to indicate values in kilo,
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700368mega or gigabytes. (Here, Kilo, Mega, Giga are Kibibytes, Mebibytes, Gibibytes.)
369
Daisuke Nishimurac5b947b2009-06-17 16:27:20 -0700370NOTE: We can write "-1" to reset the *.limit_in_bytes(unlimited).
Balbir Singh4b3bde42009-09-23 15:56:32 -0700371NOTE: We cannot set limits on the root cgroup any more.
Balbir Singh0eea1032008-02-07 00:13:57 -0800372
Jörg Sommerf6e07d32011-06-15 12:59:45 -0700373# cat /sys/fs/cgroup/memory/0/memory.limit_in_bytes
Li Zefan2324c5d2008-02-23 15:24:12 -08003744194304
Balbir Singh0eea1032008-02-07 00:13:57 -0800375
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800376We can check the usage:
Jörg Sommerf6e07d32011-06-15 12:59:45 -0700377# cat /sys/fs/cgroup/memory/0/memory.usage_in_bytes
Li Zefan2324c5d2008-02-23 15:24:12 -08003781216512
Balbir Singh0eea1032008-02-07 00:13:57 -0800379
Michael Kerrisk1939c552012-10-08 16:33:09 -0700380A successful write to this file does not guarantee a successful setting of
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700381this limit to the value written into the file. This can be due to a
Balbir Singh0eea1032008-02-07 00:13:57 -0800382number of factors, such as rounding up to page boundaries or the total
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700383availability of memory on the system. The user is required to re-read
Balbir Singh0eea1032008-02-07 00:13:57 -0800384this file after a write to guarantee the value committed by the kernel.
385
Balbir Singhfb789222008-03-04 14:28:24 -0800386# echo 1 > memory.limit_in_bytes
Balbir Singh0eea1032008-02-07 00:13:57 -0800387# cat memory.limit_in_bytes
Li Zefan2324c5d2008-02-23 15:24:12 -08003884096
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800389
390The memory.failcnt field gives the number of times that the cgroup limit was
391exceeded.
392
KAMEZAWA Hiroyukidfc05c22008-02-07 00:14:41 -0800393The memory.stat file gives accounting information. Now, the number of
394caches, RSS and Active pages/Inactive pages are shown.
395
Balbir Singh1b6df3a2008-02-07 00:13:46 -08003964. Testing
397
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700398For testing features and implementation, see memcg_test.txt.
399
400Performance test is also important. To see pure memory controller's overhead,
401testing on tmpfs will give you good numbers of small overheads.
402Example: do kernel make on tmpfs.
403
404Page-fault scalability is also important. At measuring parallel
405page fault test, multi-process test may be better than multi-thread
406test because it has noise of shared objects/status.
407
408But the above two are testing extreme situations.
409Trying usual test under memory controller is always helpful.
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800410
4114.1 Troubleshooting
412
413Sometimes a user might find that the application under a cgroup is
Michael Kerrisk1939c552012-10-08 16:33:09 -0700414terminated by the OOM killer. There are several causes for this:
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800415
4161. The cgroup limit is too low (just too low to do anything useful)
4172. The user is using anonymous memory and swap is turned off or too low
418
419A sync followed by echo 1 > /proc/sys/vm/drop_caches will help get rid of
420some of the pages cached in the cgroup (page cache pages).
421
Michael Kerrisk1939c552012-10-08 16:33:09 -0700422To know what happens, disabling OOM_Kill as per "10. OOM Control" (below) and
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700423seeing what happens will be helpful.
424
Balbir Singh1b6df3a2008-02-07 00:13:46 -08004254.2 Task migration
426
Francis Galieguea33f3222010-04-23 00:08:02 +0200427When a task migrates from one cgroup to another, its charge is not
Daisuke Nishimura7dc74be2010-03-10 15:22:13 -0800428carried forward by default. The pages allocated from the original cgroup still
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800429remain charged to it, the charge is dropped when the page is freed or
430reclaimed.
431
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700432You can move charges of a task along with task migration.
433See 8. "Move charges at task migration"
Daisuke Nishimura7dc74be2010-03-10 15:22:13 -0800434
Balbir Singh1b6df3a2008-02-07 00:13:46 -08004354.3 Removing a cgroup
436
437A cgroup can be removed by rmdir, but as discussed in sections 4.1 and 4.2, a
438cgroup might have some charge associated with it, even though all
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700439tasks have migrated away from it. (because we charge against pages, not
440against tasks.)
441
KAMEZAWA Hiroyukicc926f72012-05-29 15:07:04 -0700442We move the stats to root (if use_hierarchy==0) or parent (if
443use_hierarchy==1), and no change on the charge except uncharging
444from the child.
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800445
KAMEZAWA Hiroyuki8c7c6e342009-01-07 18:08:00 -0800446Charges recorded in swap information is not updated at removal of cgroup.
447Recorded information is discarded and a cgroup which uses swap (swapcache)
448will be charged as a new owner of it.
449
KAMEZAWA Hiroyukicc926f72012-05-29 15:07:04 -0700450About use_hierarchy, see Section 6.
KAMEZAWA Hiroyuki8c7c6e342009-01-07 18:08:00 -0800451
KAMEZAWA Hiroyukic1e862c2009-01-07 18:07:55 -08004525. Misc. interfaces.
453
4545.1 force_empty
455 memory.force_empty interface is provided to make cgroup's memory usage empty.
456 You can use this interface only when the cgroup has no tasks.
457 When writing anything to this
458
459 # echo 0 > memory.force_empty
460
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700461 Almost all pages tracked by this memory cgroup will be unmapped and freed.
462 Some pages cannot be freed because they are locked or in-use. Such pages are
Michael Kerrisk1939c552012-10-08 16:33:09 -0700463 moved to parent (if use_hierarchy==1) or root (if use_hierarchy==0) and this
KAMEZAWA Hiroyukicc926f72012-05-29 15:07:04 -0700464 cgroup will be empty.
KAMEZAWA Hiroyukic1e862c2009-01-07 18:07:55 -0800465
Michael Kerrisk1939c552012-10-08 16:33:09 -0700466 The typical use case for this interface is before calling rmdir().
KAMEZAWA Hiroyukic1e862c2009-01-07 18:07:55 -0800467 Because rmdir() moves all pages to parent, some out-of-use page caches can be
468 moved to the parent. If you want to avoid that, force_empty will be useful.
469
Glauber Costad5bdae72012-12-18 14:22:22 -0800470 Also, note that when memory.kmem.limit_in_bytes is set the charges due to
471 kernel pages will still be seen. This is not considered a failure and the
472 write will still return success. In this case, it is expected that
473 memory.kmem.usage_in_bytes == memory.usage_in_bytes.
474
KAMEZAWA Hiroyukicc926f72012-05-29 15:07:04 -0700475 About use_hierarchy, see Section 6.
476
KOSAKI Motohiro7f016ee2009-01-07 18:08:22 -08004775.2 stat file
KOSAKI Motohiro7f016ee2009-01-07 18:08:22 -0800478
Johannes Weiner185efc02011-09-14 16:21:58 -0700479memory.stat file includes following statistics
KOSAKI Motohiro7f016ee2009-01-07 18:08:22 -0800480
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700481# per-memory cgroup local status
Bharata B Raoc863d832009-04-13 14:40:15 -0700482cache - # of bytes of page cache memory.
David Rientjesb070e652013-05-07 16:18:09 -0700483rss - # of bytes of anonymous and swap cache memory (includes
484 transparent hugepages).
485rss_huge - # of bytes of anonymous transparent hugepages.
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700486mapped_file - # of bytes of mapped file (includes tmpfs/shmem)
Ying Han0527b692012-01-12 17:18:27 -0800487pgpgin - # of charging events to the memory cgroup. The charging
488 event happens each time a page is accounted as either mapped
489 anon page(RSS) or cache page(Page Cache) to the cgroup.
490pgpgout - # of uncharging events to the memory cgroup. The uncharging
491 event happens each time a page is unaccounted from the cgroup.
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700492swap - # of bytes of swap usage
Sha Zhengju9cb2dc12013-09-12 15:13:54 -0700493writeback - # of bytes of file/anon cache that are queued for syncing to
494 disk.
Aaro Koskinena15e4192013-06-19 15:34:29 +0300495inactive_anon - # of bytes of anonymous and swap cache memory on inactive
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700496 LRU list.
497active_anon - # of bytes of anonymous and swap cache memory on active
Aaro Koskinena15e4192013-06-19 15:34:29 +0300498 LRU list.
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700499inactive_file - # of bytes of file-backed memory on inactive LRU list.
500active_file - # of bytes of file-backed memory on active LRU list.
Bharata B Raoc863d832009-04-13 14:40:15 -0700501unevictable - # of bytes of memory that cannot be reclaimed (mlocked etc).
502
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700503# status considering hierarchy (see memory.use_hierarchy settings)
504
505hierarchical_memory_limit - # of bytes of memory limit with regard to hierarchy
506 under which the memory cgroup is
507hierarchical_memsw_limit - # of bytes of memory+swap limit with regard to
508 hierarchy under which memory cgroup is.
509
Johannes Weinereb6332a2012-05-29 15:06:26 -0700510total_<counter> - # hierarchical version of <counter>, which in
511 addition to the cgroup's own value includes the
512 sum of all hierarchical children's values of
513 <counter>, i.e. total_cache
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700514
515# The following additional stats are dependent on CONFIG_DEBUG_VM.
Bharata B Raoc863d832009-04-13 14:40:15 -0700516
Bharata B Raoc863d832009-04-13 14:40:15 -0700517recent_rotated_anon - VM internal parameter. (see mm/vmscan.c)
518recent_rotated_file - VM internal parameter. (see mm/vmscan.c)
519recent_scanned_anon - VM internal parameter. (see mm/vmscan.c)
520recent_scanned_file - VM internal parameter. (see mm/vmscan.c)
521
522Memo:
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700523 recent_rotated means recent frequency of LRU rotation.
524 recent_scanned means recent # of scans to LRU.
KOSAKI Motohiro7f016ee2009-01-07 18:08:22 -0800525 showing for better debug please see the code for meanings.
526
Bharata B Raoc863d832009-04-13 14:40:15 -0700527Note:
528 Only anonymous and swap cache memory is listed as part of 'rss' stat.
529 This should not be confused with the true 'resident set size' or the
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700530 amount of physical memory used by the cgroup.
531 'rss + file_mapped" will give you resident set size of cgroup.
532 (Note: file and shmem may be shared among other cgroups. In that case,
533 file_mapped is accounted only when the memory cgroup is owner of page
534 cache.)
KOSAKI Motohiro7f016ee2009-01-07 18:08:22 -0800535
KOSAKI Motohiroa7885eb2009-01-07 18:08:24 -08005365.3 swappiness
KOSAKI Motohiroa7885eb2009-01-07 18:08:24 -0800537
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700538Similar to /proc/sys/vm/swappiness, but affecting a hierarchy of groups only.
Michal Hocko9a5a8f12012-11-16 14:14:49 -0800539Please note that unlike the global swappiness, memcg knob set to 0
540really prevents from any swapping even if there is a swap storage
541available. This might lead to memcg OOM killer if there are no file
542pages to reclaim.
KOSAKI Motohiroa7885eb2009-01-07 18:08:24 -0800543
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700544Following cgroups' swappiness can't be changed.
545- root cgroup (uses /proc/sys/vm/swappiness).
546- a cgroup which uses hierarchy and it has other cgroup(s) below it.
547- a cgroup which uses hierarchy and not the root of hierarchy.
548
5495.4 failcnt
550
551A memory cgroup provides memory.failcnt and memory.memsw.failcnt files.
552This failcnt(== failure count) shows the number of times that a usage counter
553hit its limit. When a memory cgroup hits a limit, failcnt increases and
554memory under it will be reclaimed.
555
556You can reset failcnt by writing 0 to failcnt file.
557# echo 0 > .../memory.failcnt
KOSAKI Motohiroa7885eb2009-01-07 18:08:24 -0800558
Daisuke Nishimuraa111c962011-04-27 15:26:48 -07005595.5 usage_in_bytes
560
561For efficiency, as other kernel components, memory cgroup uses some optimization
562to avoid unnecessary cacheline false sharing. usage_in_bytes is affected by the
Michael Kerrisk1939c552012-10-08 16:33:09 -0700563method and doesn't show 'exact' value of memory (and swap) usage, it's a fuzz
Daisuke Nishimuraa111c962011-04-27 15:26:48 -0700564value for efficient access. (Of course, when necessary, it's synchronized.)
565If you want to know more exact memory usage, you should use RSS+CACHE(+SWAP)
566value in memory.stat(see 5.2).
567
Ying Han50c35e52011-06-15 15:08:16 -07005685.6 numa_stat
569
570This is similar to numa_maps but operates on a per-memcg basis. This is
571useful for providing visibility into the numa locality information within
572an memcg since the pages are allowed to be allocated from any physical
Michael Kerrisk1939c552012-10-08 16:33:09 -0700573node. One of the use cases is evaluating application performance by
574combining this information with the application's CPU allocation.
Ying Han50c35e52011-06-15 15:08:16 -0700575
Ying Han071aee12013-11-12 15:07:41 -0800576Each memcg's numa_stat file includes "total", "file", "anon" and "unevictable"
577per-node page counts including "hierarchical_<counter>" which sums up all
578hierarchical children's values in addition to the memcg's own value.
579
Masanari Iida8173d5a2013-12-22 00:57:33 +0900580The output format of memory.numa_stat is:
Ying Han50c35e52011-06-15 15:08:16 -0700581
582total=<total pages> N0=<node 0 pages> N1=<node 1 pages> ...
583file=<total file pages> N0=<node 0 pages> N1=<node 1 pages> ...
584anon=<total anon pages> N0=<node 0 pages> N1=<node 1 pages> ...
585unevictable=<total anon pages> N0=<node 0 pages> N1=<node 1 pages> ...
Ying Han071aee12013-11-12 15:07:41 -0800586hierarchical_<counter>=<counter pages> N0=<node 0 pages> N1=<node 1 pages> ...
Ying Han50c35e52011-06-15 15:08:16 -0700587
Ying Han071aee12013-11-12 15:07:41 -0800588The "total" count is sum of file + anon + unevictable.
Ying Han50c35e52011-06-15 15:08:16 -0700589
Balbir Singh52bc0d82009-01-07 18:08:03 -08005906. Hierarchy support
KAMEZAWA Hiroyukic1e862c2009-01-07 18:07:55 -0800591
Balbir Singh52bc0d82009-01-07 18:08:03 -0800592The memory controller supports a deep hierarchy and hierarchical accounting.
593The hierarchy is created by creating the appropriate cgroups in the
594cgroup filesystem. Consider for example, the following cgroup filesystem
595hierarchy
596
Jörg Sommer67de0162011-06-15 13:00:47 -0700597 root
Balbir Singh52bc0d82009-01-07 18:08:03 -0800598 / | \
Jörg Sommer67de0162011-06-15 13:00:47 -0700599 / | \
600 a b c
601 | \
602 | \
603 d e
Balbir Singh52bc0d82009-01-07 18:08:03 -0800604
605In the diagram above, with hierarchical accounting enabled, all memory
606usage of e, is accounted to its ancestors up until the root (i.e, c and root),
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700607that has memory.use_hierarchy enabled. If one of the ancestors goes over its
Balbir Singh52bc0d82009-01-07 18:08:03 -0800608limit, the reclaim algorithm reclaims from the tasks in the ancestor and the
609children of the ancestor.
610
6116.1 Enabling hierarchical accounting and reclaim
612
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700613A memory cgroup by default disables the hierarchy feature. Support
Balbir Singh52bc0d82009-01-07 18:08:03 -0800614can be enabled by writing 1 to memory.use_hierarchy file of the root cgroup
615
616# echo 1 > memory.use_hierarchy
617
618The feature can be disabled by
619
620# echo 0 > memory.use_hierarchy
621
Greg Thelen689bca32011-02-16 17:51:23 -0800622NOTE1: Enabling/disabling will fail if either the cgroup already has other
623 cgroups created below it, or if the parent cgroup has use_hierarchy
624 enabled.
Balbir Singh52bc0d82009-01-07 18:08:03 -0800625
KAMEZAWA Hiroyukidaaf1e62010-03-10 15:22:32 -0800626NOTE2: When panic_on_oom is set to "2", the whole system will panic in
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700627 case of an OOM event in any cgroup.
Balbir Singh52bc0d82009-01-07 18:08:03 -0800628
Balbir Singha6df6362009-09-23 15:56:34 -07006297. Soft limits
630
631Soft limits allow for greater sharing of memory. The idea behind soft limits
632is to allow control groups to use as much of the memory as needed, provided
633
634a. There is no memory contention
635b. They do not exceed their hard limit
636
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700637When the system detects memory contention or low memory, control groups
Balbir Singha6df6362009-09-23 15:56:34 -0700638are pushed back to their soft limits. If the soft limit of each control
639group is very high, they are pushed back as much as possible to make
640sure that one control group does not starve the others of memory.
641
Michael Kerrisk1939c552012-10-08 16:33:09 -0700642Please note that soft limits is a best-effort feature; it comes with
Balbir Singha6df6362009-09-23 15:56:34 -0700643no guarantees, but it does its best to make sure that when memory is
644heavily contended for, memory is allocated based on the soft limit
Michael Kerrisk1939c552012-10-08 16:33:09 -0700645hints/setup. Currently soft limit based reclaim is set up such that
Balbir Singha6df6362009-09-23 15:56:34 -0700646it gets invoked from balance_pgdat (kswapd).
647
6487.1 Interface
649
650Soft limits can be setup by using the following commands (in this example we
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700651assume a soft limit of 256 MiB)
Balbir Singha6df6362009-09-23 15:56:34 -0700652
653# echo 256M > memory.soft_limit_in_bytes
654
655If we want to change this to 1G, we can at any time use
656
657# echo 1G > memory.soft_limit_in_bytes
658
659NOTE1: Soft limits take effect over a long period of time, since they involve
660 reclaiming memory for balancing between memory cgroups
661NOTE2: It is recommended to set the soft limit always below the hard limit,
662 otherwise the hard limit will take precedence.
663
Daisuke Nishimura7dc74be2010-03-10 15:22:13 -08006648. Move charges at task migration
665
666Users can move charges associated with a task along with task migration, that
667is, uncharge task's pages from the old cgroup and charge them to the new cgroup.
Daisuke Nishimura02491442010-03-10 15:22:17 -0800668This feature is not supported in !CONFIG_MMU environments because of lack of
669page tables.
Daisuke Nishimura7dc74be2010-03-10 15:22:13 -0800670
6718.1 Interface
672
Masanari Iida8173d5a2013-12-22 00:57:33 +0900673This feature is disabled by default. It can be enabled (and disabled again) by
Daisuke Nishimura7dc74be2010-03-10 15:22:13 -0800674writing to memory.move_charge_at_immigrate of the destination cgroup.
675
676If you want to enable it:
677
678# echo (some positive value) > memory.move_charge_at_immigrate
679
680Note: Each bits of move_charge_at_immigrate has its own meaning about what type
681 of charges should be moved. See 8.2 for details.
Michael Kerrisk1939c552012-10-08 16:33:09 -0700682Note: Charges are moved only when you move mm->owner, in other words,
683 a leader of a thread group.
Daisuke Nishimura7dc74be2010-03-10 15:22:13 -0800684Note: If we cannot find enough space for the task in the destination cgroup, we
685 try to make space by reclaiming memory. Task migration may fail if we
686 cannot make enough space.
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700687Note: It can take several seconds if you move charges much.
Daisuke Nishimura7dc74be2010-03-10 15:22:13 -0800688
689And if you want disable it again:
690
691# echo 0 > memory.move_charge_at_immigrate
692
Michael Kerrisk1939c552012-10-08 16:33:09 -07006938.2 Type of charges which can be moved
Daisuke Nishimura7dc74be2010-03-10 15:22:13 -0800694
Michael Kerrisk1939c552012-10-08 16:33:09 -0700695Each bit in move_charge_at_immigrate has its own meaning about what type of
696charges should be moved. But in any case, it must be noted that an account of
697a page or a swap can be moved only when it is charged to the task's current
698(old) memory cgroup.
Daisuke Nishimura7dc74be2010-03-10 15:22:13 -0800699
700 bit | what type of charges would be moved ?
701 -----+------------------------------------------------------------------------
Michael Kerrisk1939c552012-10-08 16:33:09 -0700702 0 | A charge of an anonymous page (or swap of it) used by the target task.
703 | You must enable Swap Extension (see 2.4) to enable move of swap charges.
Daisuke Nishimura87946a72010-05-26 14:42:39 -0700704 -----+------------------------------------------------------------------------
Michael Kerrisk1939c552012-10-08 16:33:09 -0700705 1 | A charge of file pages (normal file, tmpfs file (e.g. ipc shared memory)
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700706 | and swaps of tmpfs file) mmapped by the target task. Unlike the case of
Michael Kerrisk1939c552012-10-08 16:33:09 -0700707 | anonymous pages, file pages (and swaps) in the range mmapped by the task
Daisuke Nishimura87946a72010-05-26 14:42:39 -0700708 | will be moved even if the task hasn't done page fault, i.e. they might
709 | not be the task's "RSS", but other task's "RSS" that maps the same file.
Michael Kerrisk1939c552012-10-08 16:33:09 -0700710 | And mapcount of the page is ignored (the page can be moved even if
711 | page_mapcount(page) > 1). You must enable Swap Extension (see 2.4) to
Daisuke Nishimura87946a72010-05-26 14:42:39 -0700712 | enable move of swap charges.
Daisuke Nishimura7dc74be2010-03-10 15:22:13 -0800713
7148.3 TODO
715
Daisuke Nishimura7dc74be2010-03-10 15:22:13 -0800716- All of moving charge operations are done under cgroup_mutex. It's not good
717 behavior to hold the mutex too long, so we may need some trick.
718
Kirill A. Shutemov2e72b632010-03-10 15:22:24 -08007199. Memory thresholds
720
Michael Kerrisk1939c552012-10-08 16:33:09 -0700721Memory cgroup implements memory thresholds using the cgroups notification
Kirill A. Shutemov2e72b632010-03-10 15:22:24 -0800722API (see cgroups.txt). It allows to register multiple memory and memsw
723thresholds and gets notifications when it crosses.
724
Michael Kerrisk1939c552012-10-08 16:33:09 -0700725To register a threshold, an application must:
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700726- create an eventfd using eventfd(2);
727- open memory.usage_in_bytes or memory.memsw.usage_in_bytes;
728- write string like "<event_fd> <fd of memory.usage_in_bytes> <threshold>" to
729 cgroup.event_control.
Kirill A. Shutemov2e72b632010-03-10 15:22:24 -0800730
731Application will be notified through eventfd when memory usage crosses
732threshold in any direction.
733
734It's applicable for root and non-root cgroup.
735
KAMEZAWA Hiroyuki9490ff22010-05-26 14:42:36 -070073610. OOM Control
737
KAMEZAWA Hiroyuki3c11ecf2010-05-26 14:42:37 -0700738memory.oom_control file is for OOM notification and other controls.
739
Michael Kerrisk1939c552012-10-08 16:33:09 -0700740Memory cgroup implements OOM notifier using the cgroup notification
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700741API (See cgroups.txt). It allows to register multiple OOM notification
742delivery and gets notification when OOM happens.
KAMEZAWA Hiroyuki9490ff22010-05-26 14:42:36 -0700743
Michael Kerrisk1939c552012-10-08 16:33:09 -0700744To register a notifier, an application must:
KAMEZAWA Hiroyuki9490ff22010-05-26 14:42:36 -0700745 - create an eventfd using eventfd(2)
746 - open memory.oom_control file
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700747 - write string like "<event_fd> <fd of memory.oom_control>" to
748 cgroup.event_control
KAMEZAWA Hiroyuki9490ff22010-05-26 14:42:36 -0700749
Michael Kerrisk1939c552012-10-08 16:33:09 -0700750The application will be notified through eventfd when OOM happens.
751OOM notification doesn't work for the root cgroup.
KAMEZAWA Hiroyuki9490ff22010-05-26 14:42:36 -0700752
Michael Kerrisk1939c552012-10-08 16:33:09 -0700753You can disable the OOM-killer by writing "1" to memory.oom_control file, as:
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700754
KAMEZAWA Hiroyuki3c11ecf2010-05-26 14:42:37 -0700755 #echo 1 > memory.oom_control
756
Michael Kerrisk1939c552012-10-08 16:33:09 -0700757This operation is only allowed to the top cgroup of a sub-hierarchy.
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700758If OOM-killer is disabled, tasks under cgroup will hang/sleep
759in memory cgroup's OOM-waitqueue when they request accountable memory.
KAMEZAWA Hiroyuki3c11ecf2010-05-26 14:42:37 -0700760
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700761For running them, you have to relax the memory cgroup's OOM status by
KAMEZAWA Hiroyuki3c11ecf2010-05-26 14:42:37 -0700762 * enlarge limit or reduce usage.
763To reduce usage,
764 * kill some tasks.
765 * move some tasks to other group with account migration.
766 * remove some files (on tmpfs?)
767
768Then, stopped tasks will work again.
769
770At reading, current status of OOM is shown.
771 oom_kill_disable 0 or 1 (if 1, oom-killer is disabled)
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700772 under_oom 0 or 1 (if 1, the memory cgroup is under OOM, tasks may
KAMEZAWA Hiroyuki3c11ecf2010-05-26 14:42:37 -0700773 be stopped.)
KAMEZAWA Hiroyuki9490ff22010-05-26 14:42:36 -0700774
Anton Vorontsov70ddf632013-04-29 15:08:31 -070077511. Memory Pressure
776
777The pressure level notifications can be used to monitor the memory
778allocation cost; based on the pressure, applications can implement
779different strategies of managing their memory resources. The pressure
780levels are defined as following:
781
782The "low" level means that the system is reclaiming memory for new
783allocations. Monitoring this reclaiming activity might be useful for
784maintaining cache level. Upon notification, the program (typically
785"Activity Manager") might analyze vmstat and act in advance (i.e.
786prematurely shutdown unimportant services).
787
788The "medium" level means that the system is experiencing medium memory
789pressure, the system might be making swap, paging out active file caches,
790etc. Upon this event applications may decide to further analyze
791vmstat/zoneinfo/memcg or internal memory usage statistics and free any
792resources that can be easily reconstructed or re-read from a disk.
793
794The "critical" level means that the system is actively thrashing, it is
795about to out of memory (OOM) or even the in-kernel OOM killer is on its
796way to trigger. Applications should do whatever they can to help the
797system. It might be too late to consult with vmstat or any other
798statistics, so it's advisable to take an immediate action.
799
800The events are propagated upward until the event is handled, i.e. the
801events are not pass-through. Here is what this means: for example you have
802three cgroups: A->B->C. Now you set up an event listener on cgroups A, B
803and C, and suppose group C experiences some pressure. In this situation,
804only group C will receive the notification, i.e. groups A and B will not
805receive it. This is done to avoid excessive "broadcasting" of messages,
806which disturbs the system and which is especially bad if we are low on
807memory or thrashing. So, organize the cgroups wisely, or propagate the
808events manually (or, ask us to implement the pass-through events,
809explaining why would you need them.)
810
811The file memory.pressure_level is only used to setup an eventfd. To
812register a notification, an application must:
813
814- create an eventfd using eventfd(2);
815- open memory.pressure_level;
816- write string like "<event_fd> <fd of memory.pressure_level> <level>"
817 to cgroup.event_control.
818
819Application will be notified through eventfd when memory pressure is at
820the specific level (or higher). Read/write operations to
821memory.pressure_level are no implemented.
822
823Test:
824
825 Here is a small script example that makes a new cgroup, sets up a
826 memory limit, sets up a notification in the cgroup and then makes child
827 cgroup experience a critical pressure:
828
829 # cd /sys/fs/cgroup/memory/
830 # mkdir foo
831 # cd foo
832 # cgroup_event_listener memory.pressure_level low &
833 # echo 8000000 > memory.limit_in_bytes
834 # echo 8000000 > memory.memsw.limit_in_bytes
835 # echo $$ > tasks
836 # dd if=/dev/zero | read x
837
838 (Expect a bunch of notifications, and eventually, the oom-killer will
839 trigger.)
840
84112. TODO
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800842
Li Zefanf968ef12013-07-03 15:02:25 -07008431. Make per-cgroup scanner reclaim not-shared pages first
8442. Teach controller to account for shared-pages
8453. Start reclamation in the background when the limit is
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800846 not yet hit but the usage is getting closer
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800847
848Summary
849
850Overall, the memory controller has been a stable controller and has been
851commented and discussed quite extensively in the community.
852
853References
854
8551. Singh, Balbir. RFC: Memory Controller, http://lwn.net/Articles/206697/
8562. Singh, Balbir. Memory Controller (RSS Control),
857 http://lwn.net/Articles/222762/
8583. Emelianov, Pavel. Resource controllers based on process cgroups
859 http://lkml.org/lkml/2007/3/6/198
8604. Emelianov, Pavel. RSS controller based on process cgroups (v2)
Li Zefan2324c5d2008-02-23 15:24:12 -0800861 http://lkml.org/lkml/2007/4/9/78
Balbir Singh1b6df3a2008-02-07 00:13:46 -08008625. Emelianov, Pavel. RSS controller based on process cgroups (v3)
863 http://lkml.org/lkml/2007/5/30/244
8646. Menage, Paul. Control Groups v10, http://lwn.net/Articles/236032/
8657. Vaidyanathan, Srinivasan, Control Groups: Pagecache accounting and control
866 subsystem (v3), http://lwn.net/Articles/235534/
Li Zefan2324c5d2008-02-23 15:24:12 -08008678. Singh, Balbir. RSS controller v2 test results (lmbench),
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800868 http://lkml.org/lkml/2007/5/17/232
Li Zefan2324c5d2008-02-23 15:24:12 -08008699. Singh, Balbir. RSS controller v2 AIM9 results
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800870 http://lkml.org/lkml/2007/5/18/1
Li Zefan2324c5d2008-02-23 15:24:12 -080087110. Singh, Balbir. Memory controller v6 test results,
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800872 http://lkml.org/lkml/2007/8/19/36
Li Zefan2324c5d2008-02-23 15:24:12 -080087311. Singh, Balbir. Memory controller introduction (v6),
874 http://lkml.org/lkml/2007/8/17/69
Balbir Singh1b6df3a2008-02-07 00:13:46 -080087512. Corbet, Jonathan, Controlling memory use in cgroups,
876 http://lwn.net/Articles/243795/