blob: 98e4db5ba8d17efa9fdba836758a4ba3dc4b9c69 [file] [log] [blame]
Balbir Singh00f0b822008-03-04 14:28:39 -08001Memory Resource Controller
2
Jörg Sommer67de0162011-06-15 13:00:47 -07003NOTE: The Memory Resource Controller has generically been referred to as the
4 memory controller in this document. Do not confuse memory controller
5 used here with the memory controller that is used in hardware.
Balbir Singh1b6df3a2008-02-07 00:13:46 -08006
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -07007(For editors)
8In this document:
9 When we mention a cgroup (cgroupfs's directory) with memory controller,
10 we call it "memory cgroup". When you see git-log and source code, you'll
11 see patch's title and function names tend to use "memcg".
12 In this document, we avoid using it.
Balbir Singh1b6df3a2008-02-07 00:13:46 -080013
Balbir Singh1b6df3a2008-02-07 00:13:46 -080014Benefits and Purpose of the memory controller
15
16The memory controller isolates the memory behaviour of a group of tasks
17from the rest of the system. The article on LWN [12] mentions some probable
18uses of the memory controller. The memory controller can be used to
19
20a. Isolate an application or a group of applications
Michael Kerrisk1939c552012-10-08 16:33:09 -070021 Memory-hungry applications can be isolated and limited to a smaller
Balbir Singh1b6df3a2008-02-07 00:13:46 -080022 amount of memory.
Michael Kerrisk1939c552012-10-08 16:33:09 -070023b. Create a cgroup with a limited amount of memory; this can be used
Balbir Singh1b6df3a2008-02-07 00:13:46 -080024 as a good alternative to booting with mem=XXXX.
25c. Virtualization solutions can control the amount of memory they want
26 to assign to a virtual machine instance.
27d. A CD/DVD burner could control the amount of memory used by the
28 rest of the system to ensure that burning does not fail due to lack
29 of available memory.
Michael Kerrisk1939c552012-10-08 16:33:09 -070030e. There are several other use cases; find one or use the controller just
Balbir Singh1b6df3a2008-02-07 00:13:46 -080031 for fun (to learn and hack on the VM subsystem).
32
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -070033Current Status: linux-2.6.34-mmotm(development version of 2010/April)
34
35Features:
36 - accounting anonymous pages, file caches, swap caches usage and limiting them.
Ying Han6252efc2012-04-12 12:49:10 -070037 - pages are linked to per-memcg LRU exclusively, and there is no global LRU.
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -070038 - optionally, memory+swap usage can be accounted and limited.
39 - hierarchical accounting
40 - soft limit
Michael Kerrisk1939c552012-10-08 16:33:09 -070041 - moving (recharging) account at moving a task is selectable.
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -070042 - usage threshold notifier
Anton Vorontsov70ddf632013-04-29 15:08:31 -070043 - memory pressure notifier
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -070044 - oom-killer disable knob and oom-notifier
45 - Root cgroup has no limit controls.
46
Michael Kerrisk1939c552012-10-08 16:33:09 -070047 Kernel memory support is a work in progress, and the current version provides
Glauber Costa65c64ce2011-12-22 01:02:27 +000048 basically functionality. (See Section 2.7)
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -070049
50Brief summary of control files.
51
52 tasks # attach a task(thread) and show list of threads
53 cgroup.procs # show list of processes
54 cgroup.event_control # an interface for event_fd()
Daisuke Nishimuraa111c962011-04-27 15:26:48 -070055 memory.usage_in_bytes # show current res_counter usage for memory
56 (See 5.5 for details)
57 memory.memsw.usage_in_bytes # show current res_counter usage for memory+Swap
58 (See 5.5 for details)
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -070059 memory.limit_in_bytes # set/show limit of memory usage
60 memory.memsw.limit_in_bytes # set/show limit of memory+Swap usage
61 memory.failcnt # show the number of memory usage hits limits
62 memory.memsw.failcnt # show the number of memory+Swap hits limits
63 memory.max_usage_in_bytes # show max memory usage recorded
Zhu Yanhaid66c1ce2012-01-12 17:18:24 -080064 memory.memsw.max_usage_in_bytes # show max memory+Swap usage recorded
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -070065 memory.soft_limit_in_bytes # set/show soft limit of memory usage
66 memory.stat # show various statistics
67 memory.use_hierarchy # set/show hierarchical account enabled
68 memory.force_empty # trigger forced move charge to parent
Anton Vorontsov70ddf632013-04-29 15:08:31 -070069 memory.pressure_level # set memory pressure notifications
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -070070 memory.swappiness # set/show swappiness parameter of vmscan
71 (See sysctl's vm.swappiness)
72 memory.move_charge_at_immigrate # set/show controls of moving charges
73 memory.oom_control # set/show oom controls.
Ying Han50c35e52011-06-15 15:08:16 -070074 memory.numa_stat # show the number of memory usage per numa node
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -070075
Glauber Costad5bdae72012-12-18 14:22:22 -080076 memory.kmem.limit_in_bytes # set/show hard limit for kernel memory
77 memory.kmem.usage_in_bytes # show current kernel memory allocation
78 memory.kmem.failcnt # show the number of kernel memory usage hits limits
79 memory.kmem.max_usage_in_bytes # show max kernel memory usage recorded
80
Glauber Costa3aaabe22011-12-11 21:47:06 +000081 memory.kmem.tcp.limit_in_bytes # set/show hard limit for tcp buf memory
Glauber Costa5a6dd342011-12-11 21:47:07 +000082 memory.kmem.tcp.usage_in_bytes # show current tcp buf memory allocation
Wanpeng Li05a73ed2012-07-31 16:43:21 -070083 memory.kmem.tcp.failcnt # show the number of tcp buf memory usage hits limits
84 memory.kmem.tcp.max_usage_in_bytes # show max tcp buf memory usage recorded
Glauber Costae5671df2011-12-11 21:47:01 +000085
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800861. History
87
88The memory controller has a long history. A request for comments for the memory
89controller was posted by Balbir Singh [1]. At the time the RFC was posted
90there were several implementations for memory control. The goal of the
91RFC was to build consensus and agreement for the minimal features required
92for memory control. The first RSS controller was posted by Balbir Singh[2]
93in Feb 2007. Pavel Emelianov [3][4][5] has since posted three versions of the
94RSS controller. At OLS, at the resource management BoF, everyone suggested
95that we handle both page cache and RSS together. Another request was raised
96to allow user space handling of OOM. The current memory controller is
97at version 6; it combines both mapped (RSS) and unmapped Page
98Cache Control [11].
99
1002. Memory Control
101
102Memory is a unique resource in the sense that it is present in a limited
103amount. If a task requires a lot of CPU processing, the task can spread
104its processing over a period of hours, days, months or years, but with
105memory, the same physical memory needs to be reused to accomplish the task.
106
107The memory controller implementation has been divided into phases. These
108are:
109
1101. Memory controller
1112. mlock(2) controller
1123. Kernel user memory accounting and slab control
1134. user mappings length controller
114
115The memory controller is the first controller developed.
116
1172.1. Design
118
119The core of the design is a counter called the res_counter. The res_counter
120tracks the current memory usage and limit of the group of processes associated
121with the controller. Each cgroup has a memory controller specific data
122structure (mem_cgroup) associated with it.
123
1242.2. Accounting
125
126 +--------------------+
127 | mem_cgroup |
128 | (res_counter) |
129 +--------------------+
130 / ^ \
131 / | \
132 +---------------+ | +---------------+
133 | mm_struct | |.... | mm_struct |
134 | | | | |
135 +---------------+ | +---------------+
136 |
137 + --------------+
138 |
139 +---------------+ +------+--------+
140 | page +----------> page_cgroup|
141 | | | |
142 +---------------+ +---------------+
143
144 (Figure 1: Hierarchy of Accounting)
145
146
147Figure 1 shows the important aspects of the controller
148
1491. Accounting happens per cgroup
1502. Each mm_struct knows about which cgroup it belongs to
1513. Each page has a pointer to the page_cgroup, which in turn knows the
152 cgroup it belongs to
153
Jeff Liu348b4652012-12-11 16:01:28 -0800154The accounting is done as follows: mem_cgroup_charge_common() is invoked to
155set up the necessary data structures and check if the cgroup that is being
156charged is over its limit. If it is, then reclaim is invoked on the cgroup.
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800157More details can be found in the reclaim section of this document.
158If everything goes well, a page meta-data-structure called page_cgroup is
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700159updated. page_cgroup has its own LRU on cgroup.
160(*) page_cgroup structure is allocated at boot/memory-hotplug time.
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800161
1622.2.1 Accounting details
163
KAMEZAWA Hiroyuki5b4e6552008-10-18 20:28:10 -0700164All mapped anon pages (RSS) and cache pages (Page Cache) are accounted.
Ying Han6252efc2012-04-12 12:49:10 -0700165Some pages which are never reclaimable and will not be on the LRU
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700166are not accounted. We just account pages under usual VM management.
KAMEZAWA Hiroyuki5b4e6552008-10-18 20:28:10 -0700167
168RSS pages are accounted at page_fault unless they've already been accounted
169for earlier. A file page will be accounted for as Page Cache when it's
170inserted into inode (radix-tree). While it's mapped into the page tables of
171processes, duplicate accounting is carefully avoided.
172
Michael Kerrisk1939c552012-10-08 16:33:09 -0700173An RSS page is unaccounted when it's fully unmapped. A PageCache page is
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700174unaccounted when it's removed from radix-tree. Even if RSS pages are fully
175unmapped (by kswapd), they may exist as SwapCache in the system until they
Michael Kerrisk1939c552012-10-08 16:33:09 -0700176are really freed. Such SwapCaches are also accounted.
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700177A swapped-in page is not accounted until it's mapped.
178
Michael Kerrisk1939c552012-10-08 16:33:09 -0700179Note: The kernel does swapin-readahead and reads multiple swaps at once.
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700180This means swapped-in pages may contain pages for other tasks than a task
181causing page fault. So, we avoid accounting at swap-in I/O.
KAMEZAWA Hiroyuki5b4e6552008-10-18 20:28:10 -0700182
183At page migration, accounting information is kept.
184
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700185Note: we just account pages-on-LRU because our purpose is to control amount
186of used pages; not-on-LRU pages tend to be out-of-control from VM view.
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800187
1882.3 Shared Page Accounting
189
190Shared pages are accounted on the basis of the first touch approach. The
191cgroup that first touches a page is accounted for the page. The principle
192behind this approach is that a cgroup that aggressively uses a shared
193page will eventually get charged for it (once it is uncharged from
194the cgroup that brought it in -- this will happen on memory pressure).
195
KAMEZAWA Hiroyuki4b913552012-05-29 15:06:51 -0700196But see section 8.2: when moving a task to another cgroup, its pages may
197be recharged to the new cgroup, if move_charge_at_immigrate has been chosen.
198
Paul Bolledf7c6b92013-03-25 23:59:16 +0100199Exception: If CONFIG_MEMCG_SWAP is not used.
KAMEZAWA Hiroyuki8c7c6e342009-01-07 18:08:00 -0800200When you do swapoff and make swapped-out pages of shmem(tmpfs) to
KAMEZAWA Hiroyukid13d1442009-01-07 18:07:56 -0800201be backed into memory in force, charges for pages are accounted against the
202caller of swapoff rather than the users of shmem.
203
Andrew Mortonc255a452012-07-31 16:43:02 -07002042.4 Swap Extension (CONFIG_MEMCG_SWAP)
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700205
KAMEZAWA Hiroyuki8c7c6e342009-01-07 18:08:00 -0800206Swap Extension allows you to record charge for swap. A swapped-in page is
207charged back to original page allocator if possible.
208
209When swap is accounted, following files are added.
210 - memory.memsw.usage_in_bytes.
211 - memory.memsw.limit_in_bytes.
212
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700213memsw means memory+swap. Usage of memory+swap is limited by
214memsw.limit_in_bytes.
KAMEZAWA Hiroyuki8c7c6e342009-01-07 18:08:00 -0800215
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700216Example: Assume a system with 4G of swap. A task which allocates 6G of memory
217(by mistake) under 2G memory limitation will use all swap.
218In this case, setting memsw.limit_in_bytes=3G will prevent bad use of swap.
Michael Kerrisk1939c552012-10-08 16:33:09 -0700219By using the memsw limit, you can avoid system OOM which can be caused by swap
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700220shortage.
221
222* why 'memory+swap' rather than swap.
KAMEZAWA Hiroyuki8c7c6e342009-01-07 18:08:00 -0800223The global LRU(kswapd) can swap out arbitrary pages. Swap-out means
224to move account from memory to swap...there is no change in usage of
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700225memory+swap. In other words, when we want to limit the usage of swap without
226affecting global LRU, memory+swap limit is better than just limiting swap from
Michael Kerrisk1939c552012-10-08 16:33:09 -0700227an OS point of view.
KAMEZAWA Hiroyuki8c7c6e342009-01-07 18:08:00 -0800228
KAMEZAWA Hiroyuki22a668d2009-06-17 16:27:19 -0700229* What happens when a cgroup hits memory.memsw.limit_in_bytes
Jörg Sommer67de0162011-06-15 13:00:47 -0700230When a cgroup hits memory.memsw.limit_in_bytes, it's useless to do swap-out
KAMEZAWA Hiroyuki22a668d2009-06-17 16:27:19 -0700231in this cgroup. Then, swap-out will not be done by cgroup routine and file
232caches are dropped. But as mentioned above, global LRU can do swapout memory
233from it for sanity of the system's memory management state. You can't forbid
234it by cgroup.
KAMEZAWA Hiroyuki8c7c6e342009-01-07 18:08:00 -0800235
2362.5 Reclaim
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800237
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700238Each cgroup maintains a per cgroup LRU which has the same structure as
239global VM. When a cgroup goes over its limit, we first try
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800240to reclaim memory from the cgroup so as to make space for the new
241pages that the cgroup has touched. If the reclaim is unsuccessful,
242an OOM routine is invoked to select and kill the bulkiest task in the
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700243cgroup. (See 10. OOM Control below.)
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800244
245The reclaim algorithm has not been modified for cgroups, except that
Michael Kerrisk1939c552012-10-08 16:33:09 -0700246pages that are selected for reclaiming come from the per-cgroup LRU
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800247list.
248
Balbir Singh4b3bde42009-09-23 15:56:32 -0700249NOTE: Reclaim does not work for the root cgroup, since we cannot set any
250limits on the root cgroup.
251
KAMEZAWA Hiroyukidaaf1e62010-03-10 15:22:32 -0800252Note2: When panic_on_oom is set to "2", the whole system will panic.
253
KAMEZAWA Hiroyuki9490ff22010-05-26 14:42:36 -0700254When oom event notifier is registered, event will be delivered.
255(See oom_control section)
256
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -07002572.6 Locking
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800258
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700259 lock_page_cgroup()/unlock_page_cgroup() should not be called under
260 mapping->tree_lock.
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800261
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700262 Other lock order is following:
263 PG_locked.
264 mm->page_table_lock
265 zone->lru_lock
266 lock_page_cgroup.
267 In many cases, just lock_page_cgroup() is called.
268 per-zone-per-cgroup LRU (cgroup's private LRU) is just guarded by
269 zone->lru_lock, it has no lock of its own.
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800270
Andrew Mortonc255a452012-07-31 16:43:02 -07002712.7 Kernel Memory Extension (CONFIG_MEMCG_KMEM)
Glauber Costae5671df2011-12-11 21:47:01 +0000272
273With the Kernel memory extension, the Memory Controller is able to limit
274the amount of kernel memory used by the system. Kernel memory is fundamentally
275different than user memory, since it can't be swapped out, which makes it
276possible to DoS the system by consuming too much of this precious resource.
277
Glauber Costad5bdae72012-12-18 14:22:22 -0800278Kernel memory won't be accounted at all until limit on a group is set. This
279allows for existing setups to continue working without disruption. The limit
280cannot be set if the cgroup have children, or if there are already tasks in the
281cgroup. Attempting to set the limit under those conditions will return -EBUSY.
282When use_hierarchy == 1 and a group is accounted, its children will
283automatically be accounted regardless of their limit value.
284
285After a group is first limited, it will be kept being accounted until it
286is removed. The memory limitation itself, can of course be removed by writing
287-1 to memory.kmem.limit_in_bytes. In this case, kmem will be accounted, but not
288limited.
289
Glauber Costae5671df2011-12-11 21:47:01 +0000290Kernel memory limits are not imposed for the root cgroup. Usage for the root
Glauber Costad5bdae72012-12-18 14:22:22 -0800291cgroup may or may not be accounted. The memory used is accumulated into
292memory.kmem.usage_in_bytes, or in a separate counter when it makes sense.
293(currently only for tcp).
294The main "kmem" counter is fed into the main counter, so kmem charges will
295also be visible from the user counter.
Glauber Costae5671df2011-12-11 21:47:01 +0000296
Glauber Costae5671df2011-12-11 21:47:01 +0000297Currently no soft limit is implemented for kernel memory. It is future work
298to trigger slab reclaim when those limits are reached.
299
3002.7.1 Current Kernel Memory resources accounted
301
Glauber Costad5bdae72012-12-18 14:22:22 -0800302* stack pages: every process consumes some stack pages. By accounting into
303kernel memory, we prevent new processes from being created when the kernel
304memory usage is too high.
305
Glauber Costa92e79342012-12-18 14:23:08 -0800306* slab pages: pages allocated by the SLAB or SLUB allocator are tracked. A copy
Anatol Pomozovf884ab12013-05-08 16:56:16 -0700307of each kmem_cache is created every time the cache is touched by the first time
Glauber Costa92e79342012-12-18 14:23:08 -0800308from inside the memcg. The creation is done lazily, so some objects can still be
309skipped while the cache is being created. All objects in a slab page should
310belong to the same memcg. This only fails to hold when a task is migrated to a
311different memcg during the page allocation by the cache.
312
Glauber Costae1aab162011-12-11 21:47:03 +0000313* sockets memory pressure: some sockets protocols have memory pressure
314thresholds. The Memory Controller allows them to be controlled individually
315per cgroup, instead of globally.
Glauber Costae5671df2011-12-11 21:47:01 +0000316
Glauber Costad1a4c0b2011-12-11 21:47:04 +0000317* tcp memory pressure: sockets memory pressure for the tcp protocol.
318
Glauber Costad5bdae72012-12-18 14:22:22 -08003192.7.3 Common use cases
320
321Because the "kmem" counter is fed to the main user counter, kernel memory can
322never be limited completely independently of user memory. Say "U" is the user
323limit, and "K" the kernel limit. There are three possible ways limits can be
324set:
325
326 U != 0, K = unlimited:
327 This is the standard memcg limitation mechanism already present before kmem
328 accounting. Kernel memory is completely ignored.
329
330 U != 0, K < U:
331 Kernel memory is a subset of the user memory. This setup is useful in
332 deployments where the total amount of memory per-cgroup is overcommited.
333 Overcommiting kernel memory limits is definitely not recommended, since the
334 box can still run out of non-reclaimable memory.
335 In this case, the admin could set up K so that the sum of all groups is
336 never greater than the total memory, and freely set U at the cost of his
337 QoS.
338
339 U != 0, K >= U:
340 Since kmem charges will also be fed to the user counter and reclaim will be
341 triggered for the cgroup for both kinds of memory. This setup gives the
342 admin a unified view of memory, and it is also useful for people who just
343 want to track kernel memory usage.
344
Balbir Singh1b6df3a2008-02-07 00:13:46 -08003453. User Interface
346
3470. Configuration
348
349a. Enable CONFIG_CGROUPS
350b. Enable CONFIG_RESOURCE_COUNTERS
Andrew Mortonc255a452012-07-31 16:43:02 -0700351c. Enable CONFIG_MEMCG
352d. Enable CONFIG_MEMCG_SWAP (to use swap extension)
Glauber Costad5bdae72012-12-18 14:22:22 -0800353d. Enable CONFIG_MEMCG_KMEM (to use kmem extension)
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800354
Jörg Sommerf6e07d32011-06-15 12:59:45 -07003551. Prepare the cgroups (see cgroups.txt, Why are cgroups needed?)
356# mount -t tmpfs none /sys/fs/cgroup
357# mkdir /sys/fs/cgroup/memory
358# mount -t cgroup none /sys/fs/cgroup/memory -o memory
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800359
3602. Make the new group and move bash into it
Jörg Sommerf6e07d32011-06-15 12:59:45 -0700361# mkdir /sys/fs/cgroup/memory/0
362# echo $$ > /sys/fs/cgroup/memory/0/tasks
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800363
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700364Since now we're in the 0 cgroup, we can alter the memory limit:
Jörg Sommerf6e07d32011-06-15 12:59:45 -0700365# echo 4M > /sys/fs/cgroup/memory/0/memory.limit_in_bytes
Balbir Singh0eea1032008-02-07 00:13:57 -0800366
367NOTE: We can use a suffix (k, K, m, M, g or G) to indicate values in kilo,
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700368mega or gigabytes. (Here, Kilo, Mega, Giga are Kibibytes, Mebibytes, Gibibytes.)
369
Daisuke Nishimurac5b947b2009-06-17 16:27:20 -0700370NOTE: We can write "-1" to reset the *.limit_in_bytes(unlimited).
Balbir Singh4b3bde42009-09-23 15:56:32 -0700371NOTE: We cannot set limits on the root cgroup any more.
Balbir Singh0eea1032008-02-07 00:13:57 -0800372
Jörg Sommerf6e07d32011-06-15 12:59:45 -0700373# cat /sys/fs/cgroup/memory/0/memory.limit_in_bytes
Li Zefan2324c5d2008-02-23 15:24:12 -08003744194304
Balbir Singh0eea1032008-02-07 00:13:57 -0800375
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800376We can check the usage:
Jörg Sommerf6e07d32011-06-15 12:59:45 -0700377# cat /sys/fs/cgroup/memory/0/memory.usage_in_bytes
Li Zefan2324c5d2008-02-23 15:24:12 -08003781216512
Balbir Singh0eea1032008-02-07 00:13:57 -0800379
Michael Kerrisk1939c552012-10-08 16:33:09 -0700380A successful write to this file does not guarantee a successful setting of
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700381this limit to the value written into the file. This can be due to a
Balbir Singh0eea1032008-02-07 00:13:57 -0800382number of factors, such as rounding up to page boundaries or the total
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700383availability of memory on the system. The user is required to re-read
Balbir Singh0eea1032008-02-07 00:13:57 -0800384this file after a write to guarantee the value committed by the kernel.
385
Balbir Singhfb789222008-03-04 14:28:24 -0800386# echo 1 > memory.limit_in_bytes
Balbir Singh0eea1032008-02-07 00:13:57 -0800387# cat memory.limit_in_bytes
Li Zefan2324c5d2008-02-23 15:24:12 -08003884096
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800389
390The memory.failcnt field gives the number of times that the cgroup limit was
391exceeded.
392
KAMEZAWA Hiroyukidfc05c22008-02-07 00:14:41 -0800393The memory.stat file gives accounting information. Now, the number of
394caches, RSS and Active pages/Inactive pages are shown.
395
Balbir Singh1b6df3a2008-02-07 00:13:46 -08003964. Testing
397
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700398For testing features and implementation, see memcg_test.txt.
399
400Performance test is also important. To see pure memory controller's overhead,
401testing on tmpfs will give you good numbers of small overheads.
402Example: do kernel make on tmpfs.
403
404Page-fault scalability is also important. At measuring parallel
405page fault test, multi-process test may be better than multi-thread
406test because it has noise of shared objects/status.
407
408But the above two are testing extreme situations.
409Trying usual test under memory controller is always helpful.
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800410
4114.1 Troubleshooting
412
413Sometimes a user might find that the application under a cgroup is
Michael Kerrisk1939c552012-10-08 16:33:09 -0700414terminated by the OOM killer. There are several causes for this:
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800415
4161. The cgroup limit is too low (just too low to do anything useful)
4172. The user is using anonymous memory and swap is turned off or too low
418
419A sync followed by echo 1 > /proc/sys/vm/drop_caches will help get rid of
420some of the pages cached in the cgroup (page cache pages).
421
Michael Kerrisk1939c552012-10-08 16:33:09 -0700422To know what happens, disabling OOM_Kill as per "10. OOM Control" (below) and
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700423seeing what happens will be helpful.
424
Balbir Singh1b6df3a2008-02-07 00:13:46 -08004254.2 Task migration
426
Francis Galieguea33f3222010-04-23 00:08:02 +0200427When a task migrates from one cgroup to another, its charge is not
Daisuke Nishimura7dc74be2010-03-10 15:22:13 -0800428carried forward by default. The pages allocated from the original cgroup still
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800429remain charged to it, the charge is dropped when the page is freed or
430reclaimed.
431
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700432You can move charges of a task along with task migration.
433See 8. "Move charges at task migration"
Daisuke Nishimura7dc74be2010-03-10 15:22:13 -0800434
Balbir Singh1b6df3a2008-02-07 00:13:46 -08004354.3 Removing a cgroup
436
437A cgroup can be removed by rmdir, but as discussed in sections 4.1 and 4.2, a
438cgroup might have some charge associated with it, even though all
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700439tasks have migrated away from it. (because we charge against pages, not
440against tasks.)
441
KAMEZAWA Hiroyukicc926f72012-05-29 15:07:04 -0700442We move the stats to root (if use_hierarchy==0) or parent (if
443use_hierarchy==1), and no change on the charge except uncharging
444from the child.
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800445
KAMEZAWA Hiroyuki8c7c6e342009-01-07 18:08:00 -0800446Charges recorded in swap information is not updated at removal of cgroup.
447Recorded information is discarded and a cgroup which uses swap (swapcache)
448will be charged as a new owner of it.
449
KAMEZAWA Hiroyukicc926f72012-05-29 15:07:04 -0700450About use_hierarchy, see Section 6.
KAMEZAWA Hiroyuki8c7c6e342009-01-07 18:08:00 -0800451
KAMEZAWA Hiroyukic1e862c2009-01-07 18:07:55 -08004525. Misc. interfaces.
453
4545.1 force_empty
455 memory.force_empty interface is provided to make cgroup's memory usage empty.
456 You can use this interface only when the cgroup has no tasks.
457 When writing anything to this
458
459 # echo 0 > memory.force_empty
460
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700461 Almost all pages tracked by this memory cgroup will be unmapped and freed.
462 Some pages cannot be freed because they are locked or in-use. Such pages are
Michael Kerrisk1939c552012-10-08 16:33:09 -0700463 moved to parent (if use_hierarchy==1) or root (if use_hierarchy==0) and this
KAMEZAWA Hiroyukicc926f72012-05-29 15:07:04 -0700464 cgroup will be empty.
KAMEZAWA Hiroyukic1e862c2009-01-07 18:07:55 -0800465
Michael Kerrisk1939c552012-10-08 16:33:09 -0700466 The typical use case for this interface is before calling rmdir().
KAMEZAWA Hiroyukic1e862c2009-01-07 18:07:55 -0800467 Because rmdir() moves all pages to parent, some out-of-use page caches can be
468 moved to the parent. If you want to avoid that, force_empty will be useful.
469
Glauber Costad5bdae72012-12-18 14:22:22 -0800470 Also, note that when memory.kmem.limit_in_bytes is set the charges due to
471 kernel pages will still be seen. This is not considered a failure and the
472 write will still return success. In this case, it is expected that
473 memory.kmem.usage_in_bytes == memory.usage_in_bytes.
474
KAMEZAWA Hiroyukicc926f72012-05-29 15:07:04 -0700475 About use_hierarchy, see Section 6.
476
KOSAKI Motohiro7f016ee2009-01-07 18:08:22 -08004775.2 stat file
KOSAKI Motohiro7f016ee2009-01-07 18:08:22 -0800478
Johannes Weiner185efc02011-09-14 16:21:58 -0700479memory.stat file includes following statistics
KOSAKI Motohiro7f016ee2009-01-07 18:08:22 -0800480
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700481# per-memory cgroup local status
Bharata B Raoc863d832009-04-13 14:40:15 -0700482cache - # of bytes of page cache memory.
David Rientjesb070e652013-05-07 16:18:09 -0700483rss - # of bytes of anonymous and swap cache memory (includes
484 transparent hugepages).
485rss_huge - # of bytes of anonymous transparent hugepages.
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700486mapped_file - # of bytes of mapped file (includes tmpfs/shmem)
Ying Han0527b692012-01-12 17:18:27 -0800487pgpgin - # of charging events to the memory cgroup. The charging
488 event happens each time a page is accounted as either mapped
489 anon page(RSS) or cache page(Page Cache) to the cgroup.
490pgpgout - # of uncharging events to the memory cgroup. The uncharging
491 event happens each time a page is unaccounted from the cgroup.
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700492swap - # of bytes of swap usage
Aaro Koskinena15e4192013-06-19 15:34:29 +0300493inactive_anon - # of bytes of anonymous and swap cache memory on inactive
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700494 LRU list.
495active_anon - # of bytes of anonymous and swap cache memory on active
Aaro Koskinena15e4192013-06-19 15:34:29 +0300496 LRU list.
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700497inactive_file - # of bytes of file-backed memory on inactive LRU list.
498active_file - # of bytes of file-backed memory on active LRU list.
Bharata B Raoc863d832009-04-13 14:40:15 -0700499unevictable - # of bytes of memory that cannot be reclaimed (mlocked etc).
500
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700501# status considering hierarchy (see memory.use_hierarchy settings)
502
503hierarchical_memory_limit - # of bytes of memory limit with regard to hierarchy
504 under which the memory cgroup is
505hierarchical_memsw_limit - # of bytes of memory+swap limit with regard to
506 hierarchy under which memory cgroup is.
507
Johannes Weinereb6332a2012-05-29 15:06:26 -0700508total_<counter> - # hierarchical version of <counter>, which in
509 addition to the cgroup's own value includes the
510 sum of all hierarchical children's values of
511 <counter>, i.e. total_cache
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700512
513# The following additional stats are dependent on CONFIG_DEBUG_VM.
Bharata B Raoc863d832009-04-13 14:40:15 -0700514
Bharata B Raoc863d832009-04-13 14:40:15 -0700515recent_rotated_anon - VM internal parameter. (see mm/vmscan.c)
516recent_rotated_file - VM internal parameter. (see mm/vmscan.c)
517recent_scanned_anon - VM internal parameter. (see mm/vmscan.c)
518recent_scanned_file - VM internal parameter. (see mm/vmscan.c)
519
520Memo:
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700521 recent_rotated means recent frequency of LRU rotation.
522 recent_scanned means recent # of scans to LRU.
KOSAKI Motohiro7f016ee2009-01-07 18:08:22 -0800523 showing for better debug please see the code for meanings.
524
Bharata B Raoc863d832009-04-13 14:40:15 -0700525Note:
526 Only anonymous and swap cache memory is listed as part of 'rss' stat.
527 This should not be confused with the true 'resident set size' or the
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700528 amount of physical memory used by the cgroup.
529 'rss + file_mapped" will give you resident set size of cgroup.
530 (Note: file and shmem may be shared among other cgroups. In that case,
531 file_mapped is accounted only when the memory cgroup is owner of page
532 cache.)
KOSAKI Motohiro7f016ee2009-01-07 18:08:22 -0800533
KOSAKI Motohiroa7885eb2009-01-07 18:08:24 -08005345.3 swappiness
KOSAKI Motohiroa7885eb2009-01-07 18:08:24 -0800535
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700536Similar to /proc/sys/vm/swappiness, but affecting a hierarchy of groups only.
Michal Hocko9a5a8f12012-11-16 14:14:49 -0800537Please note that unlike the global swappiness, memcg knob set to 0
538really prevents from any swapping even if there is a swap storage
539available. This might lead to memcg OOM killer if there are no file
540pages to reclaim.
KOSAKI Motohiroa7885eb2009-01-07 18:08:24 -0800541
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700542Following cgroups' swappiness can't be changed.
543- root cgroup (uses /proc/sys/vm/swappiness).
544- a cgroup which uses hierarchy and it has other cgroup(s) below it.
545- a cgroup which uses hierarchy and not the root of hierarchy.
546
5475.4 failcnt
548
549A memory cgroup provides memory.failcnt and memory.memsw.failcnt files.
550This failcnt(== failure count) shows the number of times that a usage counter
551hit its limit. When a memory cgroup hits a limit, failcnt increases and
552memory under it will be reclaimed.
553
554You can reset failcnt by writing 0 to failcnt file.
555# echo 0 > .../memory.failcnt
KOSAKI Motohiroa7885eb2009-01-07 18:08:24 -0800556
Daisuke Nishimuraa111c962011-04-27 15:26:48 -07005575.5 usage_in_bytes
558
559For efficiency, as other kernel components, memory cgroup uses some optimization
560to avoid unnecessary cacheline false sharing. usage_in_bytes is affected by the
Michael Kerrisk1939c552012-10-08 16:33:09 -0700561method and doesn't show 'exact' value of memory (and swap) usage, it's a fuzz
Daisuke Nishimuraa111c962011-04-27 15:26:48 -0700562value for efficient access. (Of course, when necessary, it's synchronized.)
563If you want to know more exact memory usage, you should use RSS+CACHE(+SWAP)
564value in memory.stat(see 5.2).
565
Ying Han50c35e52011-06-15 15:08:16 -07005665.6 numa_stat
567
568This is similar to numa_maps but operates on a per-memcg basis. This is
569useful for providing visibility into the numa locality information within
570an memcg since the pages are allowed to be allocated from any physical
Michael Kerrisk1939c552012-10-08 16:33:09 -0700571node. One of the use cases is evaluating application performance by
572combining this information with the application's CPU allocation.
Ying Han50c35e52011-06-15 15:08:16 -0700573
574We export "total", "file", "anon" and "unevictable" pages per-node for
575each memcg. The ouput format of memory.numa_stat is:
576
577total=<total pages> N0=<node 0 pages> N1=<node 1 pages> ...
578file=<total file pages> N0=<node 0 pages> N1=<node 1 pages> ...
579anon=<total anon pages> N0=<node 0 pages> N1=<node 1 pages> ...
580unevictable=<total anon pages> N0=<node 0 pages> N1=<node 1 pages> ...
581
582And we have total = file + anon + unevictable.
583
Balbir Singh52bc0d82009-01-07 18:08:03 -08005846. Hierarchy support
KAMEZAWA Hiroyukic1e862c2009-01-07 18:07:55 -0800585
Balbir Singh52bc0d82009-01-07 18:08:03 -0800586The memory controller supports a deep hierarchy and hierarchical accounting.
587The hierarchy is created by creating the appropriate cgroups in the
588cgroup filesystem. Consider for example, the following cgroup filesystem
589hierarchy
590
Jörg Sommer67de0162011-06-15 13:00:47 -0700591 root
Balbir Singh52bc0d82009-01-07 18:08:03 -0800592 / | \
Jörg Sommer67de0162011-06-15 13:00:47 -0700593 / | \
594 a b c
595 | \
596 | \
597 d e
Balbir Singh52bc0d82009-01-07 18:08:03 -0800598
599In the diagram above, with hierarchical accounting enabled, all memory
600usage of e, is accounted to its ancestors up until the root (i.e, c and root),
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700601that has memory.use_hierarchy enabled. If one of the ancestors goes over its
Balbir Singh52bc0d82009-01-07 18:08:03 -0800602limit, the reclaim algorithm reclaims from the tasks in the ancestor and the
603children of the ancestor.
604
6056.1 Enabling hierarchical accounting and reclaim
606
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700607A memory cgroup by default disables the hierarchy feature. Support
Balbir Singh52bc0d82009-01-07 18:08:03 -0800608can be enabled by writing 1 to memory.use_hierarchy file of the root cgroup
609
610# echo 1 > memory.use_hierarchy
611
612The feature can be disabled by
613
614# echo 0 > memory.use_hierarchy
615
Greg Thelen689bca32011-02-16 17:51:23 -0800616NOTE1: Enabling/disabling will fail if either the cgroup already has other
617 cgroups created below it, or if the parent cgroup has use_hierarchy
618 enabled.
Balbir Singh52bc0d82009-01-07 18:08:03 -0800619
KAMEZAWA Hiroyukidaaf1e62010-03-10 15:22:32 -0800620NOTE2: When panic_on_oom is set to "2", the whole system will panic in
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700621 case of an OOM event in any cgroup.
Balbir Singh52bc0d82009-01-07 18:08:03 -0800622
Balbir Singha6df6362009-09-23 15:56:34 -07006237. Soft limits
624
625Soft limits allow for greater sharing of memory. The idea behind soft limits
626is to allow control groups to use as much of the memory as needed, provided
627
628a. There is no memory contention
629b. They do not exceed their hard limit
630
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700631When the system detects memory contention or low memory, control groups
Balbir Singha6df6362009-09-23 15:56:34 -0700632are pushed back to their soft limits. If the soft limit of each control
633group is very high, they are pushed back as much as possible to make
634sure that one control group does not starve the others of memory.
635
Michael Kerrisk1939c552012-10-08 16:33:09 -0700636Please note that soft limits is a best-effort feature; it comes with
Balbir Singha6df6362009-09-23 15:56:34 -0700637no guarantees, but it does its best to make sure that when memory is
638heavily contended for, memory is allocated based on the soft limit
Michael Kerrisk1939c552012-10-08 16:33:09 -0700639hints/setup. Currently soft limit based reclaim is set up such that
Balbir Singha6df6362009-09-23 15:56:34 -0700640it gets invoked from balance_pgdat (kswapd).
641
6427.1 Interface
643
644Soft limits can be setup by using the following commands (in this example we
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700645assume a soft limit of 256 MiB)
Balbir Singha6df6362009-09-23 15:56:34 -0700646
647# echo 256M > memory.soft_limit_in_bytes
648
649If we want to change this to 1G, we can at any time use
650
651# echo 1G > memory.soft_limit_in_bytes
652
653NOTE1: Soft limits take effect over a long period of time, since they involve
654 reclaiming memory for balancing between memory cgroups
655NOTE2: It is recommended to set the soft limit always below the hard limit,
656 otherwise the hard limit will take precedence.
657
Daisuke Nishimura7dc74be2010-03-10 15:22:13 -08006588. Move charges at task migration
659
660Users can move charges associated with a task along with task migration, that
661is, uncharge task's pages from the old cgroup and charge them to the new cgroup.
Daisuke Nishimura02491442010-03-10 15:22:17 -0800662This feature is not supported in !CONFIG_MMU environments because of lack of
663page tables.
Daisuke Nishimura7dc74be2010-03-10 15:22:13 -0800664
6658.1 Interface
666
Michael Kerrisk1939c552012-10-08 16:33:09 -0700667This feature is disabled by default. It can be enabledi (and disabled again) by
Daisuke Nishimura7dc74be2010-03-10 15:22:13 -0800668writing to memory.move_charge_at_immigrate of the destination cgroup.
669
670If you want to enable it:
671
672# echo (some positive value) > memory.move_charge_at_immigrate
673
674Note: Each bits of move_charge_at_immigrate has its own meaning about what type
675 of charges should be moved. See 8.2 for details.
Michael Kerrisk1939c552012-10-08 16:33:09 -0700676Note: Charges are moved only when you move mm->owner, in other words,
677 a leader of a thread group.
Daisuke Nishimura7dc74be2010-03-10 15:22:13 -0800678Note: If we cannot find enough space for the task in the destination cgroup, we
679 try to make space by reclaiming memory. Task migration may fail if we
680 cannot make enough space.
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700681Note: It can take several seconds if you move charges much.
Daisuke Nishimura7dc74be2010-03-10 15:22:13 -0800682
683And if you want disable it again:
684
685# echo 0 > memory.move_charge_at_immigrate
686
Michael Kerrisk1939c552012-10-08 16:33:09 -07006878.2 Type of charges which can be moved
Daisuke Nishimura7dc74be2010-03-10 15:22:13 -0800688
Michael Kerrisk1939c552012-10-08 16:33:09 -0700689Each bit in move_charge_at_immigrate has its own meaning about what type of
690charges should be moved. But in any case, it must be noted that an account of
691a page or a swap can be moved only when it is charged to the task's current
692(old) memory cgroup.
Daisuke Nishimura7dc74be2010-03-10 15:22:13 -0800693
694 bit | what type of charges would be moved ?
695 -----+------------------------------------------------------------------------
Michael Kerrisk1939c552012-10-08 16:33:09 -0700696 0 | A charge of an anonymous page (or swap of it) used by the target task.
697 | You must enable Swap Extension (see 2.4) to enable move of swap charges.
Daisuke Nishimura87946a72010-05-26 14:42:39 -0700698 -----+------------------------------------------------------------------------
Michael Kerrisk1939c552012-10-08 16:33:09 -0700699 1 | A charge of file pages (normal file, tmpfs file (e.g. ipc shared memory)
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700700 | and swaps of tmpfs file) mmapped by the target task. Unlike the case of
Michael Kerrisk1939c552012-10-08 16:33:09 -0700701 | anonymous pages, file pages (and swaps) in the range mmapped by the task
Daisuke Nishimura87946a72010-05-26 14:42:39 -0700702 | will be moved even if the task hasn't done page fault, i.e. they might
703 | not be the task's "RSS", but other task's "RSS" that maps the same file.
Michael Kerrisk1939c552012-10-08 16:33:09 -0700704 | And mapcount of the page is ignored (the page can be moved even if
705 | page_mapcount(page) > 1). You must enable Swap Extension (see 2.4) to
Daisuke Nishimura87946a72010-05-26 14:42:39 -0700706 | enable move of swap charges.
Daisuke Nishimura7dc74be2010-03-10 15:22:13 -0800707
7088.3 TODO
709
Daisuke Nishimura7dc74be2010-03-10 15:22:13 -0800710- All of moving charge operations are done under cgroup_mutex. It's not good
711 behavior to hold the mutex too long, so we may need some trick.
712
Kirill A. Shutemov2e72b632010-03-10 15:22:24 -08007139. Memory thresholds
714
Michael Kerrisk1939c552012-10-08 16:33:09 -0700715Memory cgroup implements memory thresholds using the cgroups notification
Kirill A. Shutemov2e72b632010-03-10 15:22:24 -0800716API (see cgroups.txt). It allows to register multiple memory and memsw
717thresholds and gets notifications when it crosses.
718
Michael Kerrisk1939c552012-10-08 16:33:09 -0700719To register a threshold, an application must:
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700720- create an eventfd using eventfd(2);
721- open memory.usage_in_bytes or memory.memsw.usage_in_bytes;
722- write string like "<event_fd> <fd of memory.usage_in_bytes> <threshold>" to
723 cgroup.event_control.
Kirill A. Shutemov2e72b632010-03-10 15:22:24 -0800724
725Application will be notified through eventfd when memory usage crosses
726threshold in any direction.
727
728It's applicable for root and non-root cgroup.
729
KAMEZAWA Hiroyuki9490ff22010-05-26 14:42:36 -070073010. OOM Control
731
KAMEZAWA Hiroyuki3c11ecf2010-05-26 14:42:37 -0700732memory.oom_control file is for OOM notification and other controls.
733
Michael Kerrisk1939c552012-10-08 16:33:09 -0700734Memory cgroup implements OOM notifier using the cgroup notification
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700735API (See cgroups.txt). It allows to register multiple OOM notification
736delivery and gets notification when OOM happens.
KAMEZAWA Hiroyuki9490ff22010-05-26 14:42:36 -0700737
Michael Kerrisk1939c552012-10-08 16:33:09 -0700738To register a notifier, an application must:
KAMEZAWA Hiroyuki9490ff22010-05-26 14:42:36 -0700739 - create an eventfd using eventfd(2)
740 - open memory.oom_control file
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700741 - write string like "<event_fd> <fd of memory.oom_control>" to
742 cgroup.event_control
KAMEZAWA Hiroyuki9490ff22010-05-26 14:42:36 -0700743
Michael Kerrisk1939c552012-10-08 16:33:09 -0700744The application will be notified through eventfd when OOM happens.
745OOM notification doesn't work for the root cgroup.
KAMEZAWA Hiroyuki9490ff22010-05-26 14:42:36 -0700746
Michael Kerrisk1939c552012-10-08 16:33:09 -0700747You can disable the OOM-killer by writing "1" to memory.oom_control file, as:
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700748
KAMEZAWA Hiroyuki3c11ecf2010-05-26 14:42:37 -0700749 #echo 1 > memory.oom_control
750
Michael Kerrisk1939c552012-10-08 16:33:09 -0700751This operation is only allowed to the top cgroup of a sub-hierarchy.
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700752If OOM-killer is disabled, tasks under cgroup will hang/sleep
753in memory cgroup's OOM-waitqueue when they request accountable memory.
KAMEZAWA Hiroyuki3c11ecf2010-05-26 14:42:37 -0700754
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700755For running them, you have to relax the memory cgroup's OOM status by
KAMEZAWA Hiroyuki3c11ecf2010-05-26 14:42:37 -0700756 * enlarge limit or reduce usage.
757To reduce usage,
758 * kill some tasks.
759 * move some tasks to other group with account migration.
760 * remove some files (on tmpfs?)
761
762Then, stopped tasks will work again.
763
764At reading, current status of OOM is shown.
765 oom_kill_disable 0 or 1 (if 1, oom-killer is disabled)
KAMEZAWA Hiroyukidc10e282010-05-26 14:42:40 -0700766 under_oom 0 or 1 (if 1, the memory cgroup is under OOM, tasks may
KAMEZAWA Hiroyuki3c11ecf2010-05-26 14:42:37 -0700767 be stopped.)
KAMEZAWA Hiroyuki9490ff22010-05-26 14:42:36 -0700768
Anton Vorontsov70ddf632013-04-29 15:08:31 -070076911. Memory Pressure
770
771The pressure level notifications can be used to monitor the memory
772allocation cost; based on the pressure, applications can implement
773different strategies of managing their memory resources. The pressure
774levels are defined as following:
775
776The "low" level means that the system is reclaiming memory for new
777allocations. Monitoring this reclaiming activity might be useful for
778maintaining cache level. Upon notification, the program (typically
779"Activity Manager") might analyze vmstat and act in advance (i.e.
780prematurely shutdown unimportant services).
781
782The "medium" level means that the system is experiencing medium memory
783pressure, the system might be making swap, paging out active file caches,
784etc. Upon this event applications may decide to further analyze
785vmstat/zoneinfo/memcg or internal memory usage statistics and free any
786resources that can be easily reconstructed or re-read from a disk.
787
788The "critical" level means that the system is actively thrashing, it is
789about to out of memory (OOM) or even the in-kernel OOM killer is on its
790way to trigger. Applications should do whatever they can to help the
791system. It might be too late to consult with vmstat or any other
792statistics, so it's advisable to take an immediate action.
793
794The events are propagated upward until the event is handled, i.e. the
795events are not pass-through. Here is what this means: for example you have
796three cgroups: A->B->C. Now you set up an event listener on cgroups A, B
797and C, and suppose group C experiences some pressure. In this situation,
798only group C will receive the notification, i.e. groups A and B will not
799receive it. This is done to avoid excessive "broadcasting" of messages,
800which disturbs the system and which is especially bad if we are low on
801memory or thrashing. So, organize the cgroups wisely, or propagate the
802events manually (or, ask us to implement the pass-through events,
803explaining why would you need them.)
804
805The file memory.pressure_level is only used to setup an eventfd. To
806register a notification, an application must:
807
808- create an eventfd using eventfd(2);
809- open memory.pressure_level;
810- write string like "<event_fd> <fd of memory.pressure_level> <level>"
811 to cgroup.event_control.
812
813Application will be notified through eventfd when memory pressure is at
814the specific level (or higher). Read/write operations to
815memory.pressure_level are no implemented.
816
817Test:
818
819 Here is a small script example that makes a new cgroup, sets up a
820 memory limit, sets up a notification in the cgroup and then makes child
821 cgroup experience a critical pressure:
822
823 # cd /sys/fs/cgroup/memory/
824 # mkdir foo
825 # cd foo
826 # cgroup_event_listener memory.pressure_level low &
827 # echo 8000000 > memory.limit_in_bytes
828 # echo 8000000 > memory.memsw.limit_in_bytes
829 # echo $$ > tasks
830 # dd if=/dev/zero | read x
831
832 (Expect a bunch of notifications, and eventually, the oom-killer will
833 trigger.)
834
83512. TODO
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800836
8371. Add support for accounting huge pages (as a separate controller)
KAMEZAWA Hiroyukidfc05c22008-02-07 00:14:41 -08008382. Make per-cgroup scanner reclaim not-shared pages first
8393. Teach controller to account for shared-pages
KAMEZAWA Hiroyuki628f4232008-07-25 01:47:20 -07008404. Start reclamation in the background when the limit is
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800841 not yet hit but the usage is getting closer
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800842
843Summary
844
845Overall, the memory controller has been a stable controller and has been
846commented and discussed quite extensively in the community.
847
848References
849
8501. Singh, Balbir. RFC: Memory Controller, http://lwn.net/Articles/206697/
8512. Singh, Balbir. Memory Controller (RSS Control),
852 http://lwn.net/Articles/222762/
8533. Emelianov, Pavel. Resource controllers based on process cgroups
854 http://lkml.org/lkml/2007/3/6/198
8554. Emelianov, Pavel. RSS controller based on process cgroups (v2)
Li Zefan2324c5d2008-02-23 15:24:12 -0800856 http://lkml.org/lkml/2007/4/9/78
Balbir Singh1b6df3a2008-02-07 00:13:46 -08008575. Emelianov, Pavel. RSS controller based on process cgroups (v3)
858 http://lkml.org/lkml/2007/5/30/244
8596. Menage, Paul. Control Groups v10, http://lwn.net/Articles/236032/
8607. Vaidyanathan, Srinivasan, Control Groups: Pagecache accounting and control
861 subsystem (v3), http://lwn.net/Articles/235534/
Li Zefan2324c5d2008-02-23 15:24:12 -08008628. Singh, Balbir. RSS controller v2 test results (lmbench),
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800863 http://lkml.org/lkml/2007/5/17/232
Li Zefan2324c5d2008-02-23 15:24:12 -08008649. Singh, Balbir. RSS controller v2 AIM9 results
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800865 http://lkml.org/lkml/2007/5/18/1
Li Zefan2324c5d2008-02-23 15:24:12 -080086610. Singh, Balbir. Memory controller v6 test results,
Balbir Singh1b6df3a2008-02-07 00:13:46 -0800867 http://lkml.org/lkml/2007/8/19/36
Li Zefan2324c5d2008-02-23 15:24:12 -080086811. Singh, Balbir. Memory controller introduction (v6),
869 http://lkml.org/lkml/2007/8/17/69
Balbir Singh1b6df3a2008-02-07 00:13:46 -080087012. Corbet, Jonathan, Controlling memory use in cgroups,
871 http://lwn.net/Articles/243795/