blob: dc44785dc0fa146a2de30aa11ef2c410f1b00622 [file] [log] [blame]
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001================
Tejun Heo6c292092015-11-16 11:13:34 -05002Control Group v2
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03003================
Tejun Heo6c292092015-11-16 11:13:34 -05004
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03005:Date: October, 2015
6:Author: Tejun Heo <tj@kernel.org>
Tejun Heo6c292092015-11-16 11:13:34 -05007
8This is the authoritative documentation on the design, interface and
9conventions of cgroup v2. It describes all userland-visible aspects
10of cgroup including core and specific controller behaviors. All
11future changes must be reflected in this document. Documentation for
W. Trevor King9a2ddda2016-01-27 13:01:52 -080012v1 is available under Documentation/cgroup-v1/.
Tejun Heo6c292092015-11-16 11:13:34 -050013
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -030014.. CONTENTS
Tejun Heo6c292092015-11-16 11:13:34 -050015
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -030016 1. Introduction
17 1-1. Terminology
18 1-2. What is cgroup?
19 2. Basic Operations
20 2-1. Mounting
Tejun Heo8cfd8142017-07-21 11:14:51 -040021 2-2. Organizing Processes and Threads
22 2-2-1. Processes
23 2-2-2. Threads
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -030024 2-3. [Un]populated Notification
25 2-4. Controlling Controllers
26 2-4-1. Enabling and Disabling
27 2-4-2. Top-down Constraint
28 2-4-3. No Internal Process Constraint
29 2-5. Delegation
30 2-5-1. Model of Delegation
31 2-5-2. Delegation Containment
32 2-6. Guidelines
33 2-6-1. Organize Once and Control
34 2-6-2. Avoid Name Collisions
35 3. Resource Distribution Models
36 3-1. Weights
37 3-2. Limits
38 3-3. Protections
39 3-4. Allocations
40 4. Interface Files
41 4-1. Format
42 4-2. Conventions
43 4-3. Core Interface Files
44 5. Controllers
45 5-1. CPU
46 5-1-1. CPU Interface Files
47 5-2. Memory
48 5-2-1. Memory Interface Files
49 5-2-2. Usage Guidelines
50 5-2-3. Memory Ownership
51 5-3. IO
52 5-3-1. IO Interface Files
53 5-3-2. Writeback
54 5-4. PID
55 5-4-1. PID Interface Files
56 5-5. RDMA
57 5-5-1. RDMA Interface Files
58 5-6. Misc
59 5-6-1. perf_event
60 6. Namespace
61 6-1. Basics
62 6-2. The Root and Views
63 6-3. Migration and setns(2)
64 6-4. Interaction with Other Namespaces
65 P. Information on Kernel Programming
66 P-1. Filesystem Support for Writeback
67 D. Deprecated v1 Core Features
68 R. Issues with v1 and Rationales for v2
69 R-1. Multiple Hierarchies
70 R-2. Thread Granularity
71 R-3. Competition Between Inner Nodes and Threads
72 R-4. Other Interface Issues
73 R-5. Controller Issues and Remedies
74 R-5-1. Memory
Tejun Heo6c292092015-11-16 11:13:34 -050075
76
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -030077Introduction
78============
Tejun Heo6c292092015-11-16 11:13:34 -050079
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -030080Terminology
81-----------
Tejun Heo6c292092015-11-16 11:13:34 -050082
83"cgroup" stands for "control group" and is never capitalized. The
84singular form is used to designate the whole feature and also as a
85qualifier as in "cgroup controllers". When explicitly referring to
86multiple individual control groups, the plural form "cgroups" is used.
87
88
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -030089What is cgroup?
90---------------
Tejun Heo6c292092015-11-16 11:13:34 -050091
92cgroup is a mechanism to organize processes hierarchically and
93distribute system resources along the hierarchy in a controlled and
94configurable manner.
95
96cgroup is largely composed of two parts - the core and controllers.
97cgroup core is primarily responsible for hierarchically organizing
98processes. A cgroup controller is usually responsible for
99distributing a specific type of system resource along the hierarchy
100although there are utility controllers which serve purposes other than
101resource distribution.
102
103cgroups form a tree structure and every process in the system belongs
104to one and only one cgroup. All threads of a process belong to the
105same cgroup. On creation, all processes are put in the cgroup that
106the parent process belongs to at the time. A process can be migrated
107to another cgroup. Migration of a process doesn't affect already
108existing descendant processes.
109
110Following certain structural constraints, controllers may be enabled or
111disabled selectively on a cgroup. All controller behaviors are
112hierarchical - if a controller is enabled on a cgroup, it affects all
113processes which belong to the cgroups consisting the inclusive
114sub-hierarchy of the cgroup. When a controller is enabled on a nested
115cgroup, it always restricts the resource distribution further. The
116restrictions set closer to the root in the hierarchy can not be
117overridden from further away.
118
119
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300120Basic Operations
121================
Tejun Heo6c292092015-11-16 11:13:34 -0500122
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300123Mounting
124--------
Tejun Heo6c292092015-11-16 11:13:34 -0500125
126Unlike v1, cgroup v2 has only single hierarchy. The cgroup v2
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300127hierarchy can be mounted with the following mount command::
Tejun Heo6c292092015-11-16 11:13:34 -0500128
129 # mount -t cgroup2 none $MOUNT_POINT
130
131cgroup2 filesystem has the magic number 0x63677270 ("cgrp"). All
132controllers which support v2 and are not bound to a v1 hierarchy are
133automatically bound to the v2 hierarchy and show up at the root.
134Controllers which are not in active use in the v2 hierarchy can be
135bound to other hierarchies. This allows mixing v2 hierarchy with the
136legacy v1 multiple hierarchies in a fully backward compatible way.
137
138A controller can be moved across hierarchies only after the controller
139is no longer referenced in its current hierarchy. Because per-cgroup
140controller states are destroyed asynchronously and controllers may
141have lingering references, a controller may not show up immediately on
142the v2 hierarchy after the final umount of the previous hierarchy.
143Similarly, a controller should be fully disabled to be moved out of
144the unified hierarchy and it may take some time for the disabled
145controller to become available for other hierarchies; furthermore, due
146to inter-controller dependencies, other controllers may need to be
147disabled too.
148
149While useful for development and manual configurations, moving
150controllers dynamically between the v2 and other hierarchies is
151strongly discouraged for production use. It is recommended to decide
152the hierarchies and controller associations before starting using the
153controllers after system boot.
154
Johannes Weiner1619b6d2016-02-16 13:21:14 -0500155During transition to v2, system management software might still
156automount the v1 cgroup filesystem and so hijack all controllers
157during boot, before manual intervention is possible. To make testing
158and experimenting easier, the kernel parameter cgroup_no_v1= allows
159disabling controllers in v1 and make them always available in v2.
160
Tejun Heo5136f632017-06-27 14:30:28 -0400161cgroup v2 currently supports the following mount options.
162
163 nsdelegate
164
165 Consider cgroup namespaces as delegation boundaries. This
166 option is system wide and can only be set on mount or modified
167 through remount from the init namespace. The mount option is
168 ignored on non-init namespace mounts. Please refer to the
169 Delegation section for details.
170
Tejun Heo6c292092015-11-16 11:13:34 -0500171
Tejun Heo8cfd8142017-07-21 11:14:51 -0400172Organizing Processes and Threads
173--------------------------------
174
175Processes
176~~~~~~~~~
Tejun Heo6c292092015-11-16 11:13:34 -0500177
178Initially, only the root cgroup exists to which all processes belong.
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300179A child cgroup can be created by creating a sub-directory::
Tejun Heo6c292092015-11-16 11:13:34 -0500180
181 # mkdir $CGROUP_NAME
182
183A given cgroup may have multiple child cgroups forming a tree
184structure. Each cgroup has a read-writable interface file
185"cgroup.procs". When read, it lists the PIDs of all processes which
186belong to the cgroup one-per-line. The PIDs are not ordered and the
187same PID may show up more than once if the process got moved to
188another cgroup and then back or the PID got recycled while reading.
189
190A process can be migrated into a cgroup by writing its PID to the
191target cgroup's "cgroup.procs" file. Only one process can be migrated
192on a single write(2) call. If a process is composed of multiple
193threads, writing the PID of any thread migrates all threads of the
194process.
195
196When a process forks a child process, the new process is born into the
197cgroup that the forking process belongs to at the time of the
198operation. After exit, a process stays associated with the cgroup
199that it belonged to at the time of exit until it's reaped; however, a
200zombie process does not appear in "cgroup.procs" and thus can't be
201moved to another cgroup.
202
203A cgroup which doesn't have any children or live processes can be
204destroyed by removing the directory. Note that a cgroup which doesn't
205have any children and is associated only with zombie processes is
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300206considered empty and can be removed::
Tejun Heo6c292092015-11-16 11:13:34 -0500207
208 # rmdir $CGROUP_NAME
209
210"/proc/$PID/cgroup" lists a process's cgroup membership. If legacy
211cgroup is in use in the system, this file may contain multiple lines,
212one for each hierarchy. The entry for cgroup v2 is always in the
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300213format "0::$PATH"::
Tejun Heo6c292092015-11-16 11:13:34 -0500214
215 # cat /proc/842/cgroup
216 ...
217 0::/test-cgroup/test-cgroup-nested
218
219If the process becomes a zombie and the cgroup it was associated with
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300220is removed subsequently, " (deleted)" is appended to the path::
Tejun Heo6c292092015-11-16 11:13:34 -0500221
222 # cat /proc/842/cgroup
223 ...
224 0::/test-cgroup/test-cgroup-nested (deleted)
225
226
Tejun Heo8cfd8142017-07-21 11:14:51 -0400227Threads
228~~~~~~~
229
230cgroup v2 supports thread granularity for a subset of controllers to
231support use cases requiring hierarchical resource distribution across
232the threads of a group of processes. By default, all threads of a
233process belong to the same cgroup, which also serves as the resource
234domain to host resource consumptions which are not specific to a
235process or thread. The thread mode allows threads to be spread across
236a subtree while still maintaining the common resource domain for them.
237
238Controllers which support thread mode are called threaded controllers.
239The ones which don't are called domain controllers.
240
241Marking a cgroup threaded makes it join the resource domain of its
242parent as a threaded cgroup. The parent may be another threaded
243cgroup whose resource domain is further up in the hierarchy. The root
244of a threaded subtree, that is, the nearest ancestor which is not
245threaded, is called threaded domain or thread root interchangeably and
246serves as the resource domain for the entire subtree.
247
248Inside a threaded subtree, threads of a process can be put in
249different cgroups and are not subject to the no internal process
250constraint - threaded controllers can be enabled on non-leaf cgroups
251whether they have threads in them or not.
252
253As the threaded domain cgroup hosts all the domain resource
254consumptions of the subtree, it is considered to have internal
255resource consumptions whether there are processes in it or not and
256can't have populated child cgroups which aren't threaded. Because the
257root cgroup is not subject to no internal process constraint, it can
258serve both as a threaded domain and a parent to domain cgroups.
259
260The current operation mode or type of the cgroup is shown in the
261"cgroup.type" file which indicates whether the cgroup is a normal
262domain, a domain which is serving as the domain of a threaded subtree,
263or a threaded cgroup.
264
265On creation, a cgroup is always a domain cgroup and can be made
266threaded by writing "threaded" to the "cgroup.type" file. The
267operation is single direction::
268
269 # echo threaded > cgroup.type
270
271Once threaded, the cgroup can't be made a domain again. To enable the
272thread mode, the following conditions must be met.
273
274- As the cgroup will join the parent's resource domain. The parent
275 must either be a valid (threaded) domain or a threaded cgroup.
276
Tejun Heo918a8c22017-07-23 08:18:26 -0400277- When the parent is an unthreaded domain, it must not have any domain
278 controllers enabled or populated domain children. The root is
279 exempt from this requirement.
Tejun Heo8cfd8142017-07-21 11:14:51 -0400280
281Topology-wise, a cgroup can be in an invalid state. Please consider
282the following toplogy::
283
284 A (threaded domain) - B (threaded) - C (domain, just created)
285
286C is created as a domain but isn't connected to a parent which can
287host child domains. C can't be used until it is turned into a
288threaded cgroup. "cgroup.type" file will report "domain (invalid)" in
289these cases. Operations which fail due to invalid topology use
290EOPNOTSUPP as the errno.
291
292A domain cgroup is turned into a threaded domain when one of its child
293cgroup becomes threaded or threaded controllers are enabled in the
294"cgroup.subtree_control" file while there are processes in the cgroup.
295A threaded domain reverts to a normal domain when the conditions
296clear.
297
298When read, "cgroup.threads" contains the list of the thread IDs of all
299threads in the cgroup. Except that the operations are per-thread
300instead of per-process, "cgroup.threads" has the same format and
301behaves the same way as "cgroup.procs". While "cgroup.threads" can be
302written to in any cgroup, as it can only move threads inside the same
303threaded domain, its operations are confined inside each threaded
304subtree.
305
306The threaded domain cgroup serves as the resource domain for the whole
307subtree, and, while the threads can be scattered across the subtree,
308all the processes are considered to be in the threaded domain cgroup.
309"cgroup.procs" in a threaded domain cgroup contains the PIDs of all
310processes in the subtree and is not readable in the subtree proper.
311However, "cgroup.procs" can be written to from anywhere in the subtree
312to migrate all threads of the matching process to the cgroup.
313
314Only threaded controllers can be enabled in a threaded subtree. When
315a threaded controller is enabled inside a threaded subtree, it only
316accounts for and controls resource consumptions associated with the
317threads in the cgroup and its descendants. All consumptions which
318aren't tied to a specific thread belong to the threaded domain cgroup.
319
320Because a threaded subtree is exempt from no internal process
321constraint, a threaded controller must be able to handle competition
322between threads in a non-leaf cgroup and its child cgroups. Each
323threaded controller defines how such competitions are handled.
324
325
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300326[Un]populated Notification
327--------------------------
Tejun Heo6c292092015-11-16 11:13:34 -0500328
329Each non-root cgroup has a "cgroup.events" file which contains
330"populated" field indicating whether the cgroup's sub-hierarchy has
331live processes in it. Its value is 0 if there is no live process in
332the cgroup and its descendants; otherwise, 1. poll and [id]notify
333events are triggered when the value changes. This can be used, for
334example, to start a clean-up operation after all processes of a given
335sub-hierarchy have exited. The populated state updates and
336notifications are recursive. Consider the following sub-hierarchy
337where the numbers in the parentheses represent the numbers of processes
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300338in each cgroup::
Tejun Heo6c292092015-11-16 11:13:34 -0500339
340 A(4) - B(0) - C(1)
341 \ D(0)
342
343A, B and C's "populated" fields would be 1 while D's 0. After the one
344process in C exits, B and C's "populated" fields would flip to "0" and
345file modified events will be generated on the "cgroup.events" files of
346both cgroups.
347
348
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300349Controlling Controllers
350-----------------------
Tejun Heo6c292092015-11-16 11:13:34 -0500351
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300352Enabling and Disabling
353~~~~~~~~~~~~~~~~~~~~~~
Tejun Heo6c292092015-11-16 11:13:34 -0500354
355Each cgroup has a "cgroup.controllers" file which lists all
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300356controllers available for the cgroup to enable::
Tejun Heo6c292092015-11-16 11:13:34 -0500357
358 # cat cgroup.controllers
359 cpu io memory
360
361No controller is enabled by default. Controllers can be enabled and
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300362disabled by writing to the "cgroup.subtree_control" file::
Tejun Heo6c292092015-11-16 11:13:34 -0500363
364 # echo "+cpu +memory -io" > cgroup.subtree_control
365
366Only controllers which are listed in "cgroup.controllers" can be
367enabled. When multiple operations are specified as above, either they
368all succeed or fail. If multiple operations on the same controller
369are specified, the last one is effective.
370
371Enabling a controller in a cgroup indicates that the distribution of
372the target resource across its immediate children will be controlled.
373Consider the following sub-hierarchy. The enabled controllers are
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300374listed in parentheses::
Tejun Heo6c292092015-11-16 11:13:34 -0500375
376 A(cpu,memory) - B(memory) - C()
377 \ D()
378
379As A has "cpu" and "memory" enabled, A will control the distribution
380of CPU cycles and memory to its children, in this case, B. As B has
381"memory" enabled but not "CPU", C and D will compete freely on CPU
382cycles but their division of memory available to B will be controlled.
383
384As a controller regulates the distribution of the target resource to
385the cgroup's children, enabling it creates the controller's interface
386files in the child cgroups. In the above example, enabling "cpu" on B
387would create the "cpu." prefixed controller interface files in C and
388D. Likewise, disabling "memory" from B would remove the "memory."
389prefixed controller interface files from C and D. This means that the
390controller interface files - anything which doesn't start with
391"cgroup." are owned by the parent rather than the cgroup itself.
392
393
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300394Top-down Constraint
395~~~~~~~~~~~~~~~~~~~
Tejun Heo6c292092015-11-16 11:13:34 -0500396
397Resources are distributed top-down and a cgroup can further distribute
398a resource only if the resource has been distributed to it from the
399parent. This means that all non-root "cgroup.subtree_control" files
400can only contain controllers which are enabled in the parent's
401"cgroup.subtree_control" file. A controller can be enabled only if
402the parent has the controller enabled and a controller can't be
403disabled if one or more children have it enabled.
404
405
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300406No Internal Process Constraint
407~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Tejun Heo6c292092015-11-16 11:13:34 -0500408
Tejun Heo8cfd8142017-07-21 11:14:51 -0400409Non-root cgroups can distribute domain resources to their children
410only when they don't have any processes of their own. In other words,
411only domain cgroups which don't contain any processes can have domain
412controllers enabled in their "cgroup.subtree_control" files.
Tejun Heo6c292092015-11-16 11:13:34 -0500413
Tejun Heo8cfd8142017-07-21 11:14:51 -0400414This guarantees that, when a domain controller is looking at the part
415of the hierarchy which has it enabled, processes are always only on
416the leaves. This rules out situations where child cgroups compete
417against internal processes of the parent.
Tejun Heo6c292092015-11-16 11:13:34 -0500418
419The root cgroup is exempt from this restriction. Root contains
420processes and anonymous resource consumption which can't be associated
421with any other cgroups and requires special treatment from most
422controllers. How resource consumption in the root cgroup is governed
423is up to each controller.
424
425Note that the restriction doesn't get in the way if there is no
426enabled controller in the cgroup's "cgroup.subtree_control". This is
427important as otherwise it wouldn't be possible to create children of a
428populated cgroup. To control resource distribution of a cgroup, the
429cgroup must create children and transfer all its processes to the
430children before enabling controllers in its "cgroup.subtree_control"
431file.
432
433
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300434Delegation
435----------
Tejun Heo6c292092015-11-16 11:13:34 -0500436
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300437Model of Delegation
438~~~~~~~~~~~~~~~~~~~
Tejun Heo6c292092015-11-16 11:13:34 -0500439
Tejun Heo5136f632017-06-27 14:30:28 -0400440A cgroup can be delegated in two ways. First, to a less privileged
Tejun Heo8cfd8142017-07-21 11:14:51 -0400441user by granting write access of the directory and its "cgroup.procs",
442"cgroup.threads" and "cgroup.subtree_control" files to the user.
443Second, if the "nsdelegate" mount option is set, automatically to a
444cgroup namespace on namespace creation.
Tejun Heo6c292092015-11-16 11:13:34 -0500445
Tejun Heo5136f632017-06-27 14:30:28 -0400446Because the resource control interface files in a given directory
447control the distribution of the parent's resources, the delegatee
448shouldn't be allowed to write to them. For the first method, this is
449achieved by not granting access to these files. For the second, the
450kernel rejects writes to all files other than "cgroup.procs" and
451"cgroup.subtree_control" on a namespace root from inside the
452namespace.
453
454The end results are equivalent for both delegation types. Once
455delegated, the user can build sub-hierarchy under the directory,
456organize processes inside it as it sees fit and further distribute the
457resources it received from the parent. The limits and other settings
458of all resource controllers are hierarchical and regardless of what
459happens in the delegated sub-hierarchy, nothing can escape the
460resource restrictions imposed by the parent.
Tejun Heo6c292092015-11-16 11:13:34 -0500461
462Currently, cgroup doesn't impose any restrictions on the number of
463cgroups in or nesting depth of a delegated sub-hierarchy; however,
464this may be limited explicitly in the future.
465
466
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300467Delegation Containment
468~~~~~~~~~~~~~~~~~~~~~~
Tejun Heo6c292092015-11-16 11:13:34 -0500469
470A delegated sub-hierarchy is contained in the sense that processes
Tejun Heo5136f632017-06-27 14:30:28 -0400471can't be moved into or out of the sub-hierarchy by the delegatee.
472
473For delegations to a less privileged user, this is achieved by
474requiring the following conditions for a process with a non-root euid
475to migrate a target process into a cgroup by writing its PID to the
476"cgroup.procs" file.
Tejun Heo6c292092015-11-16 11:13:34 -0500477
Tejun Heo6c292092015-11-16 11:13:34 -0500478- The writer must have write access to the "cgroup.procs" file.
479
480- The writer must have write access to the "cgroup.procs" file of the
481 common ancestor of the source and destination cgroups.
482
Tejun Heo576dd462017-01-20 11:29:54 -0500483The above two constraints ensure that while a delegatee may migrate
Tejun Heo6c292092015-11-16 11:13:34 -0500484processes around freely in the delegated sub-hierarchy it can't pull
485in from or push out to outside the sub-hierarchy.
486
487For an example, let's assume cgroups C0 and C1 have been delegated to
488user U0 who created C00, C01 under C0 and C10 under C1 as follows and
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300489all processes under C0 and C1 belong to U0::
Tejun Heo6c292092015-11-16 11:13:34 -0500490
491 ~~~~~~~~~~~~~ - C0 - C00
492 ~ cgroup ~ \ C01
493 ~ hierarchy ~
494 ~~~~~~~~~~~~~ - C1 - C10
495
496Let's also say U0 wants to write the PID of a process which is
497currently in C10 into "C00/cgroup.procs". U0 has write access to the
Tejun Heo576dd462017-01-20 11:29:54 -0500498file; however, the common ancestor of the source cgroup C10 and the
499destination cgroup C00 is above the points of delegation and U0 would
500not have write access to its "cgroup.procs" files and thus the write
501will be denied with -EACCES.
Tejun Heo6c292092015-11-16 11:13:34 -0500502
Tejun Heo5136f632017-06-27 14:30:28 -0400503For delegations to namespaces, containment is achieved by requiring
504that both the source and destination cgroups are reachable from the
505namespace of the process which is attempting the migration. If either
506is not reachable, the migration is rejected with -ENOENT.
507
Tejun Heo6c292092015-11-16 11:13:34 -0500508
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300509Guidelines
510----------
Tejun Heo6c292092015-11-16 11:13:34 -0500511
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300512Organize Once and Control
513~~~~~~~~~~~~~~~~~~~~~~~~~
Tejun Heo6c292092015-11-16 11:13:34 -0500514
515Migrating a process across cgroups is a relatively expensive operation
516and stateful resources such as memory are not moved together with the
517process. This is an explicit design decision as there often exist
518inherent trade-offs between migration and various hot paths in terms
519of synchronization cost.
520
521As such, migrating processes across cgroups frequently as a means to
522apply different resource restrictions is discouraged. A workload
523should be assigned to a cgroup according to the system's logical and
524resource structure once on start-up. Dynamic adjustments to resource
525distribution can be made by changing controller configuration through
526the interface files.
527
528
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300529Avoid Name Collisions
530~~~~~~~~~~~~~~~~~~~~~
Tejun Heo6c292092015-11-16 11:13:34 -0500531
532Interface files for a cgroup and its children cgroups occupy the same
533directory and it is possible to create children cgroups which collide
534with interface files.
535
536All cgroup core interface files are prefixed with "cgroup." and each
537controller's interface files are prefixed with the controller name and
538a dot. A controller's name is composed of lower case alphabets and
539'_'s but never begins with an '_' so it can be used as the prefix
540character for collision avoidance. Also, interface file names won't
541start or end with terms which are often used in categorizing workloads
542such as job, service, slice, unit or workload.
543
544cgroup doesn't do anything to prevent name collisions and it's the
545user's responsibility to avoid them.
546
547
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300548Resource Distribution Models
549============================
Tejun Heo6c292092015-11-16 11:13:34 -0500550
551cgroup controllers implement several resource distribution schemes
552depending on the resource type and expected use cases. This section
553describes major schemes in use along with their expected behaviors.
554
555
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300556Weights
557-------
Tejun Heo6c292092015-11-16 11:13:34 -0500558
559A parent's resource is distributed by adding up the weights of all
560active children and giving each the fraction matching the ratio of its
561weight against the sum. As only children which can make use of the
562resource at the moment participate in the distribution, this is
563work-conserving. Due to the dynamic nature, this model is usually
564used for stateless resources.
565
566All weights are in the range [1, 10000] with the default at 100. This
567allows symmetric multiplicative biases in both directions at fine
568enough granularity while staying in the intuitive range.
569
570As long as the weight is in range, all configuration combinations are
571valid and there is no reason to reject configuration changes or
572process migrations.
573
574"cpu.weight" proportionally distributes CPU cycles to active children
575and is an example of this type.
576
577
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300578Limits
579------
Tejun Heo6c292092015-11-16 11:13:34 -0500580
581A child can only consume upto the configured amount of the resource.
582Limits can be over-committed - the sum of the limits of children can
583exceed the amount of resource available to the parent.
584
585Limits are in the range [0, max] and defaults to "max", which is noop.
586
587As limits can be over-committed, all configuration combinations are
588valid and there is no reason to reject configuration changes or
589process migrations.
590
591"io.max" limits the maximum BPS and/or IOPS that a cgroup can consume
592on an IO device and is an example of this type.
593
594
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300595Protections
596-----------
Tejun Heo6c292092015-11-16 11:13:34 -0500597
598A cgroup is protected to be allocated upto the configured amount of
599the resource if the usages of all its ancestors are under their
600protected levels. Protections can be hard guarantees or best effort
601soft boundaries. Protections can also be over-committed in which case
602only upto the amount available to the parent is protected among
603children.
604
605Protections are in the range [0, max] and defaults to 0, which is
606noop.
607
608As protections can be over-committed, all configuration combinations
609are valid and there is no reason to reject configuration changes or
610process migrations.
611
612"memory.low" implements best-effort memory protection and is an
613example of this type.
614
615
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300616Allocations
617-----------
Tejun Heo6c292092015-11-16 11:13:34 -0500618
619A cgroup is exclusively allocated a certain amount of a finite
620resource. Allocations can't be over-committed - the sum of the
621allocations of children can not exceed the amount of resource
622available to the parent.
623
624Allocations are in the range [0, max] and defaults to 0, which is no
625resource.
626
627As allocations can't be over-committed, some configuration
628combinations are invalid and should be rejected. Also, if the
629resource is mandatory for execution of processes, process migrations
630may be rejected.
631
632"cpu.rt.max" hard-allocates realtime slices and is an example of this
633type.
634
635
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300636Interface Files
637===============
Tejun Heo6c292092015-11-16 11:13:34 -0500638
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300639Format
640------
Tejun Heo6c292092015-11-16 11:13:34 -0500641
642All interface files should be in one of the following formats whenever
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300643possible::
Tejun Heo6c292092015-11-16 11:13:34 -0500644
645 New-line separated values
646 (when only one value can be written at once)
647
648 VAL0\n
649 VAL1\n
650 ...
651
652 Space separated values
653 (when read-only or multiple values can be written at once)
654
655 VAL0 VAL1 ...\n
656
657 Flat keyed
658
659 KEY0 VAL0\n
660 KEY1 VAL1\n
661 ...
662
663 Nested keyed
664
665 KEY0 SUB_KEY0=VAL00 SUB_KEY1=VAL01...
666 KEY1 SUB_KEY0=VAL10 SUB_KEY1=VAL11...
667 ...
668
669For a writable file, the format for writing should generally match
670reading; however, controllers may allow omitting later fields or
671implement restricted shortcuts for most common use cases.
672
673For both flat and nested keyed files, only the values for a single key
674can be written at a time. For nested keyed files, the sub key pairs
675may be specified in any order and not all pairs have to be specified.
676
677
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300678Conventions
679-----------
Tejun Heo6c292092015-11-16 11:13:34 -0500680
681- Settings for a single feature should be contained in a single file.
682
683- The root cgroup should be exempt from resource control and thus
684 shouldn't have resource control interface files. Also,
685 informational files on the root cgroup which end up showing global
686 information available elsewhere shouldn't exist.
687
688- If a controller implements weight based resource distribution, its
689 interface file should be named "weight" and have the range [1,
690 10000] with 100 as the default. The values are chosen to allow
691 enough and symmetric bias in both directions while keeping it
692 intuitive (the default is 100%).
693
694- If a controller implements an absolute resource guarantee and/or
695 limit, the interface files should be named "min" and "max"
696 respectively. If a controller implements best effort resource
697 guarantee and/or limit, the interface files should be named "low"
698 and "high" respectively.
699
700 In the above four control files, the special token "max" should be
701 used to represent upward infinity for both reading and writing.
702
703- If a setting has a configurable default value and keyed specific
704 overrides, the default entry should be keyed with "default" and
705 appear as the first entry in the file.
706
707 The default value can be updated by writing either "default $VAL" or
708 "$VAL".
709
710 When writing to update a specific override, "default" can be used as
711 the value to indicate removal of the override. Override entries
712 with "default" as the value must not appear when read.
713
714 For example, a setting which is keyed by major:minor device numbers
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300715 with integer values may look like the following::
Tejun Heo6c292092015-11-16 11:13:34 -0500716
717 # cat cgroup-example-interface-file
718 default 150
719 8:0 300
720
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300721 The default value can be updated by::
Tejun Heo6c292092015-11-16 11:13:34 -0500722
723 # echo 125 > cgroup-example-interface-file
724
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300725 or::
Tejun Heo6c292092015-11-16 11:13:34 -0500726
727 # echo "default 125" > cgroup-example-interface-file
728
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300729 An override can be set by::
Tejun Heo6c292092015-11-16 11:13:34 -0500730
731 # echo "8:16 170" > cgroup-example-interface-file
732
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300733 and cleared by::
Tejun Heo6c292092015-11-16 11:13:34 -0500734
735 # echo "8:0 default" > cgroup-example-interface-file
736 # cat cgroup-example-interface-file
737 default 125
738 8:16 170
739
740- For events which are not very high frequency, an interface file
741 "events" should be created which lists event key value pairs.
742 Whenever a notifiable event happens, file modified event should be
743 generated on the file.
744
745
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300746Core Interface Files
747--------------------
Tejun Heo6c292092015-11-16 11:13:34 -0500748
749All cgroup core files are prefixed with "cgroup."
750
Tejun Heo8cfd8142017-07-21 11:14:51 -0400751 cgroup.type
752
753 A read-write single value file which exists on non-root
754 cgroups.
755
756 When read, it indicates the current type of the cgroup, which
757 can be one of the following values.
758
759 - "domain" : A normal valid domain cgroup.
760
761 - "domain threaded" : A threaded domain cgroup which is
762 serving as the root of a threaded subtree.
763
764 - "domain invalid" : A cgroup which is in an invalid state.
765 It can't be populated or have controllers enabled. It may
766 be allowed to become a threaded cgroup.
767
768 - "threaded" : A threaded cgroup which is a member of a
769 threaded subtree.
770
771 A cgroup can be turned into a threaded cgroup by writing
772 "threaded" to this file.
773
Tejun Heo6c292092015-11-16 11:13:34 -0500774 cgroup.procs
Tejun Heo6c292092015-11-16 11:13:34 -0500775 A read-write new-line separated values file which exists on
776 all cgroups.
777
778 When read, it lists the PIDs of all processes which belong to
779 the cgroup one-per-line. The PIDs are not ordered and the
780 same PID may show up more than once if the process got moved
781 to another cgroup and then back or the PID got recycled while
782 reading.
783
784 A PID can be written to migrate the process associated with
785 the PID to the cgroup. The writer should match all of the
786 following conditions.
787
Tejun Heo6c292092015-11-16 11:13:34 -0500788 - It must have write access to the "cgroup.procs" file.
789
790 - It must have write access to the "cgroup.procs" file of the
791 common ancestor of the source and destination cgroups.
792
793 When delegating a sub-hierarchy, write access to this file
794 should be granted along with the containing directory.
795
Tejun Heo8cfd8142017-07-21 11:14:51 -0400796 In a threaded cgroup, reading this file fails with EOPNOTSUPP
797 as all the processes belong to the thread root. Writing is
798 supported and moves every thread of the process to the cgroup.
799
800 cgroup.threads
801 A read-write new-line separated values file which exists on
802 all cgroups.
803
804 When read, it lists the TIDs of all threads which belong to
805 the cgroup one-per-line. The TIDs are not ordered and the
806 same TID may show up more than once if the thread got moved to
807 another cgroup and then back or the TID got recycled while
808 reading.
809
810 A TID can be written to migrate the thread associated with the
811 TID to the cgroup. The writer should match all of the
812 following conditions.
813
814 - It must have write access to the "cgroup.threads" file.
815
816 - The cgroup that the thread is currently in must be in the
817 same resource domain as the destination cgroup.
818
819 - It must have write access to the "cgroup.procs" file of the
820 common ancestor of the source and destination cgroups.
821
822 When delegating a sub-hierarchy, write access to this file
823 should be granted along with the containing directory.
824
Tejun Heo6c292092015-11-16 11:13:34 -0500825 cgroup.controllers
Tejun Heo6c292092015-11-16 11:13:34 -0500826 A read-only space separated values file which exists on all
827 cgroups.
828
829 It shows space separated list of all controllers available to
830 the cgroup. The controllers are not ordered.
831
832 cgroup.subtree_control
Tejun Heo6c292092015-11-16 11:13:34 -0500833 A read-write space separated values file which exists on all
834 cgroups. Starts out empty.
835
836 When read, it shows space separated list of the controllers
837 which are enabled to control resource distribution from the
838 cgroup to its children.
839
840 Space separated list of controllers prefixed with '+' or '-'
841 can be written to enable or disable controllers. A controller
842 name prefixed with '+' enables the controller and '-'
843 disables. If a controller appears more than once on the list,
844 the last one is effective. When multiple enable and disable
845 operations are specified, either all succeed or all fail.
846
847 cgroup.events
Tejun Heo6c292092015-11-16 11:13:34 -0500848 A read-only flat-keyed file which exists on non-root cgroups.
849 The following entries are defined. Unless specified
850 otherwise, a value change in this file generates a file
851 modified event.
852
853 populated
Tejun Heo6c292092015-11-16 11:13:34 -0500854 1 if the cgroup or its descendants contains any live
855 processes; otherwise, 0.
856
Roman Gushchin1a926e02017-07-28 18:28:44 +0100857 cgroup.max.descendants
858 A read-write single value files. The default is "max".
859
860 Maximum allowed number of descent cgroups.
861 If the actual number of descendants is equal or larger,
862 an attempt to create a new cgroup in the hierarchy will fail.
863
864 cgroup.max.depth
865 A read-write single value files. The default is "max".
866
867 Maximum allowed descent depth below the current cgroup.
868 If the actual descent depth is equal or larger,
869 an attempt to create a new child cgroup will fail.
870
Roman Gushchinec392252017-08-02 17:55:31 +0100871 cgroup.stat
872 A read-only flat-keyed file with the following entries:
873
874 nr_descendants
875 Total number of visible descendant cgroups.
876
877 nr_dying_descendants
878 Total number of dying descendant cgroups. A cgroup becomes
879 dying after being deleted by a user. The cgroup will remain
880 in dying state for some time undefined time (which can depend
881 on system load) before being completely destroyed.
882
883 A process can't enter a dying cgroup under any circumstances,
884 a dying cgroup can't revive.
885
886 A dying cgroup can consume system resources not exceeding
887 limits, which were active at the moment of cgroup deletion.
888
Tejun Heo6c292092015-11-16 11:13:34 -0500889
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300890Controllers
891===========
Tejun Heo6c292092015-11-16 11:13:34 -0500892
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300893CPU
894---
Tejun Heo6c292092015-11-16 11:13:34 -0500895
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300896.. note::
897
898 The interface for the cpu controller hasn't been merged yet
Tejun Heo6c292092015-11-16 11:13:34 -0500899
900The "cpu" controllers regulates distribution of CPU cycles. This
901controller implements weight and absolute bandwidth limit models for
902normal scheduling policy and absolute bandwidth allocation model for
903realtime scheduling policy.
904
905
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300906CPU Interface Files
907~~~~~~~~~~~~~~~~~~~
Tejun Heo6c292092015-11-16 11:13:34 -0500908
909All time durations are in microseconds.
910
911 cpu.stat
Tejun Heo6c292092015-11-16 11:13:34 -0500912 A read-only flat-keyed file which exists on non-root cgroups.
913
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300914 It reports the following six stats:
Tejun Heo6c292092015-11-16 11:13:34 -0500915
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300916 - usage_usec
917 - user_usec
918 - system_usec
919 - nr_periods
920 - nr_throttled
921 - throttled_usec
Tejun Heo6c292092015-11-16 11:13:34 -0500922
923 cpu.weight
Tejun Heo6c292092015-11-16 11:13:34 -0500924 A read-write single value file which exists on non-root
925 cgroups. The default is "100".
926
927 The weight in the range [1, 10000].
928
929 cpu.max
Tejun Heo6c292092015-11-16 11:13:34 -0500930 A read-write two value file which exists on non-root cgroups.
931 The default is "max 100000".
932
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300933 The maximum bandwidth limit. It's in the following format::
Tejun Heo6c292092015-11-16 11:13:34 -0500934
935 $MAX $PERIOD
936
937 which indicates that the group may consume upto $MAX in each
938 $PERIOD duration. "max" for $MAX indicates no limit. If only
939 one number is written, $MAX is updated.
940
941 cpu.rt.max
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300942 .. note::
Tejun Heo6c292092015-11-16 11:13:34 -0500943
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300944 The semantics of this file is still under discussion and the
945 interface hasn't been merged yet
Tejun Heo6c292092015-11-16 11:13:34 -0500946
947 A read-write two value file which exists on all cgroups.
948 The default is "0 100000".
949
950 The maximum realtime runtime allocation. Over-committing
951 configurations are disallowed and process migrations are
952 rejected if not enough bandwidth is available. It's in the
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300953 following format::
Tejun Heo6c292092015-11-16 11:13:34 -0500954
955 $MAX $PERIOD
956
957 which indicates that the group may consume upto $MAX in each
958 $PERIOD duration. If only one number is written, $MAX is
959 updated.
960
961
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300962Memory
963------
Tejun Heo6c292092015-11-16 11:13:34 -0500964
965The "memory" controller regulates distribution of memory. Memory is
966stateful and implements both limit and protection models. Due to the
967intertwining between memory usage and reclaim pressure and the
968stateful nature of memory, the distribution model is relatively
969complex.
970
971While not completely water-tight, all major memory usages by a given
972cgroup are tracked so that the total memory consumption can be
973accounted and controlled to a reasonable extent. Currently, the
974following types of memory usages are tracked.
975
976- Userland memory - page cache and anonymous memory.
977
978- Kernel data structures such as dentries and inodes.
979
980- TCP socket buffers.
981
982The above list may expand in the future for better coverage.
983
984
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -0300985Memory Interface Files
986~~~~~~~~~~~~~~~~~~~~~~
Tejun Heo6c292092015-11-16 11:13:34 -0500987
988All memory amounts are in bytes. If a value which is not aligned to
989PAGE_SIZE is written, the value may be rounded up to the closest
990PAGE_SIZE multiple when read back.
991
992 memory.current
Tejun Heo6c292092015-11-16 11:13:34 -0500993 A read-only single value file which exists on non-root
994 cgroups.
995
996 The total amount of memory currently being used by the cgroup
997 and its descendants.
998
999 memory.low
Tejun Heo6c292092015-11-16 11:13:34 -05001000 A read-write single value file which exists on non-root
1001 cgroups. The default is "0".
1002
1003 Best-effort memory protection. If the memory usages of a
1004 cgroup and all its ancestors are below their low boundaries,
1005 the cgroup's memory won't be reclaimed unless memory can be
1006 reclaimed from unprotected cgroups.
1007
1008 Putting more memory than generally available under this
1009 protection is discouraged.
1010
1011 memory.high
Tejun Heo6c292092015-11-16 11:13:34 -05001012 A read-write single value file which exists on non-root
1013 cgroups. The default is "max".
1014
1015 Memory usage throttle limit. This is the main mechanism to
1016 control memory usage of a cgroup. If a cgroup's usage goes
1017 over the high boundary, the processes of the cgroup are
1018 throttled and put under heavy reclaim pressure.
1019
1020 Going over the high limit never invokes the OOM killer and
1021 under extreme conditions the limit may be breached.
1022
1023 memory.max
Tejun Heo6c292092015-11-16 11:13:34 -05001024 A read-write single value file which exists on non-root
1025 cgroups. The default is "max".
1026
1027 Memory usage hard limit. This is the final protection
1028 mechanism. If a cgroup's memory usage reaches this limit and
1029 can't be reduced, the OOM killer is invoked in the cgroup.
1030 Under certain circumstances, the usage may go over the limit
1031 temporarily.
1032
1033 This is the ultimate protection mechanism. As long as the
1034 high limit is used and monitored properly, this limit's
1035 utility is limited to providing the final safety net.
1036
1037 memory.events
Tejun Heo6c292092015-11-16 11:13:34 -05001038 A read-only flat-keyed file which exists on non-root cgroups.
1039 The following entries are defined. Unless specified
1040 otherwise, a value change in this file generates a file
1041 modified event.
1042
1043 low
Tejun Heo6c292092015-11-16 11:13:34 -05001044 The number of times the cgroup is reclaimed due to
1045 high memory pressure even though its usage is under
1046 the low boundary. This usually indicates that the low
1047 boundary is over-committed.
1048
1049 high
Tejun Heo6c292092015-11-16 11:13:34 -05001050 The number of times processes of the cgroup are
1051 throttled and routed to perform direct memory reclaim
1052 because the high memory boundary was exceeded. For a
1053 cgroup whose memory usage is capped by the high limit
1054 rather than global memory pressure, this event's
1055 occurrences are expected.
1056
1057 max
Tejun Heo6c292092015-11-16 11:13:34 -05001058 The number of times the cgroup's memory usage was
1059 about to go over the max boundary. If direct reclaim
Konstantin Khlebnikov8e675f72017-07-06 15:40:28 -07001060 fails to bring it down, the cgroup goes to OOM state.
Tejun Heo6c292092015-11-16 11:13:34 -05001061
1062 oom
Konstantin Khlebnikov8e675f72017-07-06 15:40:28 -07001063 The number of time the cgroup's memory usage was
1064 reached the limit and allocation was about to fail.
1065
1066 Depending on context result could be invocation of OOM
1067 killer and retrying allocation or failing alloction.
1068
1069 Failed allocation in its turn could be returned into
1070 userspace as -ENOMEM or siletly ignored in cases like
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001071 disk readahead. For now OOM in memory cgroup kills
Konstantin Khlebnikov8e675f72017-07-06 15:40:28 -07001072 tasks iff shortage has happened inside page fault.
1073
1074 oom_kill
Konstantin Khlebnikov8e675f72017-07-06 15:40:28 -07001075 The number of processes belonging to this cgroup
1076 killed by any kind of OOM killer.
Tejun Heo6c292092015-11-16 11:13:34 -05001077
Johannes Weiner587d9f72016-01-20 15:03:19 -08001078 memory.stat
Johannes Weiner587d9f72016-01-20 15:03:19 -08001079 A read-only flat-keyed file which exists on non-root cgroups.
1080
1081 This breaks down the cgroup's memory footprint into different
1082 types of memory, type-specific details, and other information
1083 on the state and past events of the memory management system.
1084
1085 All memory amounts are in bytes.
1086
1087 The entries are ordered to be human readable, and new entries
1088 can show up in the middle. Don't rely on items remaining in a
1089 fixed position; use the keys to look up specific values!
1090
1091 anon
Johannes Weiner587d9f72016-01-20 15:03:19 -08001092 Amount of memory used in anonymous mappings such as
1093 brk(), sbrk(), and mmap(MAP_ANONYMOUS)
1094
1095 file
Johannes Weiner587d9f72016-01-20 15:03:19 -08001096 Amount of memory used to cache filesystem data,
1097 including tmpfs and shared memory.
1098
Vladimir Davydov12580e42016-03-17 14:17:38 -07001099 kernel_stack
Vladimir Davydov12580e42016-03-17 14:17:38 -07001100 Amount of memory allocated to kernel stacks.
1101
Vladimir Davydov27ee57c2016-03-17 14:17:35 -07001102 slab
Vladimir Davydov27ee57c2016-03-17 14:17:35 -07001103 Amount of memory used for storing in-kernel data
1104 structures.
1105
Johannes Weiner4758e192016-02-02 16:57:41 -08001106 sock
Johannes Weiner4758e192016-02-02 16:57:41 -08001107 Amount of memory used in network transmission buffers
1108
Johannes Weiner9a4caf12017-05-03 14:52:45 -07001109 shmem
Johannes Weiner9a4caf12017-05-03 14:52:45 -07001110 Amount of cached filesystem data that is swap-backed,
1111 such as tmpfs, shm segments, shared anonymous mmap()s
1112
Johannes Weiner587d9f72016-01-20 15:03:19 -08001113 file_mapped
Johannes Weiner587d9f72016-01-20 15:03:19 -08001114 Amount of cached filesystem data mapped with mmap()
1115
1116 file_dirty
Johannes Weiner587d9f72016-01-20 15:03:19 -08001117 Amount of cached filesystem data that was modified but
1118 not yet written back to disk
1119
1120 file_writeback
Johannes Weiner587d9f72016-01-20 15:03:19 -08001121 Amount of cached filesystem data that was modified and
1122 is currently being written back to disk
1123
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001124 inactive_anon, active_anon, inactive_file, active_file, unevictable
Johannes Weiner587d9f72016-01-20 15:03:19 -08001125 Amount of memory, swap-backed and filesystem-backed,
1126 on the internal memory management lists used by the
1127 page reclaim algorithm
1128
Vladimir Davydov27ee57c2016-03-17 14:17:35 -07001129 slab_reclaimable
Vladimir Davydov27ee57c2016-03-17 14:17:35 -07001130 Part of "slab" that might be reclaimed, such as
1131 dentries and inodes.
1132
1133 slab_unreclaimable
Vladimir Davydov27ee57c2016-03-17 14:17:35 -07001134 Part of "slab" that cannot be reclaimed on memory
1135 pressure.
1136
Johannes Weiner587d9f72016-01-20 15:03:19 -08001137 pgfault
Johannes Weiner587d9f72016-01-20 15:03:19 -08001138 Total number of page faults incurred
1139
1140 pgmajfault
Johannes Weiner587d9f72016-01-20 15:03:19 -08001141 Number of major page faults incurred
1142
Roman Gushchinb3409592017-05-12 15:47:09 -07001143 workingset_refault
1144
1145 Number of refaults of previously evicted pages
1146
1147 workingset_activate
1148
1149 Number of refaulted pages that were immediately activated
1150
1151 workingset_nodereclaim
1152
1153 Number of times a shadow node has been reclaimed
1154
Roman Gushchin22621852017-07-06 15:40:25 -07001155 pgrefill
1156
1157 Amount of scanned pages (in an active LRU list)
1158
1159 pgscan
1160
1161 Amount of scanned pages (in an inactive LRU list)
1162
1163 pgsteal
1164
1165 Amount of reclaimed pages
1166
1167 pgactivate
1168
1169 Amount of pages moved to the active LRU list
1170
1171 pgdeactivate
1172
1173 Amount of pages moved to the inactive LRU lis
1174
1175 pglazyfree
1176
1177 Amount of pages postponed to be freed under memory pressure
1178
1179 pglazyfreed
1180
1181 Amount of reclaimed lazyfree pages
1182
Vladimir Davydov3e24b192016-01-20 15:03:13 -08001183 memory.swap.current
Vladimir Davydov3e24b192016-01-20 15:03:13 -08001184 A read-only single value file which exists on non-root
1185 cgroups.
1186
1187 The total amount of swap currently being used by the cgroup
1188 and its descendants.
1189
1190 memory.swap.max
Vladimir Davydov3e24b192016-01-20 15:03:13 -08001191 A read-write single value file which exists on non-root
1192 cgroups. The default is "max".
1193
1194 Swap usage hard limit. If a cgroup's swap usage reaches this
1195 limit, anonymous meomry of the cgroup will not be swapped out.
1196
Tejun Heo6c292092015-11-16 11:13:34 -05001197
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001198Usage Guidelines
1199~~~~~~~~~~~~~~~~
Tejun Heo6c292092015-11-16 11:13:34 -05001200
1201"memory.high" is the main mechanism to control memory usage.
1202Over-committing on high limit (sum of high limits > available memory)
1203and letting global memory pressure to distribute memory according to
1204usage is a viable strategy.
1205
1206Because breach of the high limit doesn't trigger the OOM killer but
1207throttles the offending cgroup, a management agent has ample
1208opportunities to monitor and take appropriate actions such as granting
1209more memory or terminating the workload.
1210
1211Determining whether a cgroup has enough memory is not trivial as
1212memory usage doesn't indicate whether the workload can benefit from
1213more memory. For example, a workload which writes data received from
1214network to a file can use all available memory but can also operate as
1215performant with a small amount of memory. A measure of memory
1216pressure - how much the workload is being impacted due to lack of
1217memory - is necessary to determine whether a workload needs more
1218memory; unfortunately, memory pressure monitoring mechanism isn't
1219implemented yet.
1220
1221
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001222Memory Ownership
1223~~~~~~~~~~~~~~~~
Tejun Heo6c292092015-11-16 11:13:34 -05001224
1225A memory area is charged to the cgroup which instantiated it and stays
1226charged to the cgroup until the area is released. Migrating a process
1227to a different cgroup doesn't move the memory usages that it
1228instantiated while in the previous cgroup to the new cgroup.
1229
1230A memory area may be used by processes belonging to different cgroups.
1231To which cgroup the area will be charged is in-deterministic; however,
1232over time, the memory area is likely to end up in a cgroup which has
1233enough memory allowance to avoid high reclaim pressure.
1234
1235If a cgroup sweeps a considerable amount of memory which is expected
1236to be accessed repeatedly by other cgroups, it may make sense to use
1237POSIX_FADV_DONTNEED to relinquish the ownership of memory areas
1238belonging to the affected files to ensure correct memory ownership.
1239
1240
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001241IO
1242--
Tejun Heo6c292092015-11-16 11:13:34 -05001243
1244The "io" controller regulates the distribution of IO resources. This
1245controller implements both weight based and absolute bandwidth or IOPS
1246limit distribution; however, weight based distribution is available
1247only if cfq-iosched is in use and neither scheme is available for
1248blk-mq devices.
1249
1250
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001251IO Interface Files
1252~~~~~~~~~~~~~~~~~~
Tejun Heo6c292092015-11-16 11:13:34 -05001253
1254 io.stat
Tejun Heo6c292092015-11-16 11:13:34 -05001255 A read-only nested-keyed file which exists on non-root
1256 cgroups.
1257
1258 Lines are keyed by $MAJ:$MIN device numbers and not ordered.
1259 The following nested keys are defined.
1260
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001261 ====== ===================
Tejun Heo6c292092015-11-16 11:13:34 -05001262 rbytes Bytes read
1263 wbytes Bytes written
1264 rios Number of read IOs
1265 wios Number of write IOs
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001266 ====== ===================
Tejun Heo6c292092015-11-16 11:13:34 -05001267
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001268 An example read output follows:
Tejun Heo6c292092015-11-16 11:13:34 -05001269
1270 8:16 rbytes=1459200 wbytes=314773504 rios=192 wios=353
1271 8:0 rbytes=90430464 wbytes=299008000 rios=8950 wios=1252
1272
1273 io.weight
Tejun Heo6c292092015-11-16 11:13:34 -05001274 A read-write flat-keyed file which exists on non-root cgroups.
1275 The default is "default 100".
1276
1277 The first line is the default weight applied to devices
1278 without specific override. The rest are overrides keyed by
1279 $MAJ:$MIN device numbers and not ordered. The weights are in
1280 the range [1, 10000] and specifies the relative amount IO time
1281 the cgroup can use in relation to its siblings.
1282
1283 The default weight can be updated by writing either "default
1284 $WEIGHT" or simply "$WEIGHT". Overrides can be set by writing
1285 "$MAJ:$MIN $WEIGHT" and unset by writing "$MAJ:$MIN default".
1286
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001287 An example read output follows::
Tejun Heo6c292092015-11-16 11:13:34 -05001288
1289 default 100
1290 8:16 200
1291 8:0 50
1292
1293 io.max
Tejun Heo6c292092015-11-16 11:13:34 -05001294 A read-write nested-keyed file which exists on non-root
1295 cgroups.
1296
1297 BPS and IOPS based IO limit. Lines are keyed by $MAJ:$MIN
1298 device numbers and not ordered. The following nested keys are
1299 defined.
1300
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001301 ===== ==================================
Tejun Heo6c292092015-11-16 11:13:34 -05001302 rbps Max read bytes per second
1303 wbps Max write bytes per second
1304 riops Max read IO operations per second
1305 wiops Max write IO operations per second
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001306 ===== ==================================
Tejun Heo6c292092015-11-16 11:13:34 -05001307
1308 When writing, any number of nested key-value pairs can be
1309 specified in any order. "max" can be specified as the value
1310 to remove a specific limit. If the same key is specified
1311 multiple times, the outcome is undefined.
1312
1313 BPS and IOPS are measured in each IO direction and IOs are
1314 delayed if limit is reached. Temporary bursts are allowed.
1315
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001316 Setting read limit at 2M BPS and write at 120 IOPS for 8:16::
Tejun Heo6c292092015-11-16 11:13:34 -05001317
1318 echo "8:16 rbps=2097152 wiops=120" > io.max
1319
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001320 Reading returns the following::
Tejun Heo6c292092015-11-16 11:13:34 -05001321
1322 8:16 rbps=2097152 wbps=max riops=max wiops=120
1323
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001324 Write IOPS limit can be removed by writing the following::
Tejun Heo6c292092015-11-16 11:13:34 -05001325
1326 echo "8:16 wiops=max" > io.max
1327
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001328 Reading now returns the following::
Tejun Heo6c292092015-11-16 11:13:34 -05001329
1330 8:16 rbps=2097152 wbps=max riops=max wiops=max
1331
1332
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001333Writeback
1334~~~~~~~~~
Tejun Heo6c292092015-11-16 11:13:34 -05001335
1336Page cache is dirtied through buffered writes and shared mmaps and
1337written asynchronously to the backing filesystem by the writeback
1338mechanism. Writeback sits between the memory and IO domains and
1339regulates the proportion of dirty memory by balancing dirtying and
1340write IOs.
1341
1342The io controller, in conjunction with the memory controller,
1343implements control of page cache writeback IOs. The memory controller
1344defines the memory domain that dirty memory ratio is calculated and
1345maintained for and the io controller defines the io domain which
1346writes out dirty pages for the memory domain. Both system-wide and
1347per-cgroup dirty memory states are examined and the more restrictive
1348of the two is enforced.
1349
1350cgroup writeback requires explicit support from the underlying
1351filesystem. Currently, cgroup writeback is implemented on ext2, ext4
1352and btrfs. On other filesystems, all writeback IOs are attributed to
1353the root cgroup.
1354
1355There are inherent differences in memory and writeback management
1356which affects how cgroup ownership is tracked. Memory is tracked per
1357page while writeback per inode. For the purpose of writeback, an
1358inode is assigned to a cgroup and all IO requests to write dirty pages
1359from the inode are attributed to that cgroup.
1360
1361As cgroup ownership for memory is tracked per page, there can be pages
1362which are associated with different cgroups than the one the inode is
1363associated with. These are called foreign pages. The writeback
1364constantly keeps track of foreign pages and, if a particular foreign
1365cgroup becomes the majority over a certain period of time, switches
1366the ownership of the inode to that cgroup.
1367
1368While this model is enough for most use cases where a given inode is
1369mostly dirtied by a single cgroup even when the main writing cgroup
1370changes over time, use cases where multiple cgroups write to a single
1371inode simultaneously are not supported well. In such circumstances, a
1372significant portion of IOs are likely to be attributed incorrectly.
1373As memory controller assigns page ownership on the first use and
1374doesn't update it until the page is released, even if writeback
1375strictly follows page ownership, multiple cgroups dirtying overlapping
1376areas wouldn't work as expected. It's recommended to avoid such usage
1377patterns.
1378
1379The sysctl knobs which affect writeback behavior are applied to cgroup
1380writeback as follows.
1381
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001382 vm.dirty_background_ratio, vm.dirty_ratio
Tejun Heo6c292092015-11-16 11:13:34 -05001383 These ratios apply the same to cgroup writeback with the
1384 amount of available memory capped by limits imposed by the
1385 memory controller and system-wide clean memory.
1386
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001387 vm.dirty_background_bytes, vm.dirty_bytes
Tejun Heo6c292092015-11-16 11:13:34 -05001388 For cgroup writeback, this is calculated into ratio against
1389 total available memory and applied the same way as
1390 vm.dirty[_background]_ratio.
1391
1392
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001393PID
1394---
Hans Ragas20c56e52017-01-10 17:42:34 +00001395
1396The process number controller is used to allow a cgroup to stop any
1397new tasks from being fork()'d or clone()'d after a specified limit is
1398reached.
1399
1400The number of tasks in a cgroup can be exhausted in ways which other
1401controllers cannot prevent, thus warranting its own controller. For
1402example, a fork bomb is likely to exhaust the number of tasks before
1403hitting memory restrictions.
1404
1405Note that PIDs used in this controller refer to TIDs, process IDs as
1406used by the kernel.
1407
1408
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001409PID Interface Files
1410~~~~~~~~~~~~~~~~~~~
Hans Ragas20c56e52017-01-10 17:42:34 +00001411
1412 pids.max
Tobias Klauser312eb712017-02-17 18:44:11 +01001413 A read-write single value file which exists on non-root
1414 cgroups. The default is "max".
Hans Ragas20c56e52017-01-10 17:42:34 +00001415
Tobias Klauser312eb712017-02-17 18:44:11 +01001416 Hard limit of number of processes.
Hans Ragas20c56e52017-01-10 17:42:34 +00001417
1418 pids.current
Tobias Klauser312eb712017-02-17 18:44:11 +01001419 A read-only single value file which exists on all cgroups.
Hans Ragas20c56e52017-01-10 17:42:34 +00001420
Tobias Klauser312eb712017-02-17 18:44:11 +01001421 The number of processes currently in the cgroup and its
1422 descendants.
Hans Ragas20c56e52017-01-10 17:42:34 +00001423
1424Organisational operations are not blocked by cgroup policies, so it is
1425possible to have pids.current > pids.max. This can be done by either
1426setting the limit to be smaller than pids.current, or attaching enough
1427processes to the cgroup such that pids.current is larger than
1428pids.max. However, it is not possible to violate a cgroup PID policy
1429through fork() or clone(). These will return -EAGAIN if the creation
1430of a new process would cause a cgroup policy to be violated.
1431
1432
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001433RDMA
1434----
Tejun Heo968ebff2017-01-29 14:35:20 -05001435
Parav Pandit9c1e67f2017-01-10 00:02:15 +00001436The "rdma" controller regulates the distribution and accounting of
1437of RDMA resources.
1438
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001439RDMA Interface Files
1440~~~~~~~~~~~~~~~~~~~~
Parav Pandit9c1e67f2017-01-10 00:02:15 +00001441
1442 rdma.max
1443 A readwrite nested-keyed file that exists for all the cgroups
1444 except root that describes current configured resource limit
1445 for a RDMA/IB device.
1446
1447 Lines are keyed by device name and are not ordered.
1448 Each line contains space separated resource name and its configured
1449 limit that can be distributed.
1450
1451 The following nested keys are defined.
1452
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001453 ========== =============================
Parav Pandit9c1e67f2017-01-10 00:02:15 +00001454 hca_handle Maximum number of HCA Handles
1455 hca_object Maximum number of HCA Objects
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001456 ========== =============================
Parav Pandit9c1e67f2017-01-10 00:02:15 +00001457
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001458 An example for mlx4 and ocrdma device follows::
Parav Pandit9c1e67f2017-01-10 00:02:15 +00001459
1460 mlx4_0 hca_handle=2 hca_object=2000
1461 ocrdma1 hca_handle=3 hca_object=max
1462
1463 rdma.current
1464 A read-only file that describes current resource usage.
1465 It exists for all the cgroup except root.
1466
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001467 An example for mlx4 and ocrdma device follows::
Parav Pandit9c1e67f2017-01-10 00:02:15 +00001468
1469 mlx4_0 hca_handle=1 hca_object=20
1470 ocrdma1 hca_handle=1 hca_object=23
1471
1472
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001473Misc
1474----
Tejun Heo63f1ca52017-02-02 13:50:35 -05001475
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001476perf_event
1477~~~~~~~~~~
Tejun Heo968ebff2017-01-29 14:35:20 -05001478
1479perf_event controller, if not mounted on a legacy hierarchy, is
1480automatically enabled on the v2 hierarchy so that perf events can
1481always be filtered by cgroup v2 path. The controller can still be
1482moved to a legacy hierarchy after v2 hierarchy is populated.
1483
1484
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001485Namespace
1486=========
Serge Hallynd4021f62016-01-29 02:54:10 -06001487
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001488Basics
1489------
Serge Hallynd4021f62016-01-29 02:54:10 -06001490
1491cgroup namespace provides a mechanism to virtualize the view of the
1492"/proc/$PID/cgroup" file and cgroup mounts. The CLONE_NEWCGROUP clone
1493flag can be used with clone(2) and unshare(2) to create a new cgroup
1494namespace. The process running inside the cgroup namespace will have
1495its "/proc/$PID/cgroup" output restricted to cgroupns root. The
1496cgroupns root is the cgroup of the process at the time of creation of
1497the cgroup namespace.
1498
1499Without cgroup namespace, the "/proc/$PID/cgroup" file shows the
1500complete path of the cgroup of a process. In a container setup where
1501a set of cgroups and namespaces are intended to isolate processes the
1502"/proc/$PID/cgroup" file may leak potential system level information
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001503to the isolated processes. For Example::
Serge Hallynd4021f62016-01-29 02:54:10 -06001504
1505 # cat /proc/self/cgroup
1506 0::/batchjobs/container_id1
1507
1508The path '/batchjobs/container_id1' can be considered as system-data
1509and undesirable to expose to the isolated processes. cgroup namespace
1510can be used to restrict visibility of this path. For example, before
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001511creating a cgroup namespace, one would see::
Serge Hallynd4021f62016-01-29 02:54:10 -06001512
1513 # ls -l /proc/self/ns/cgroup
1514 lrwxrwxrwx 1 root root 0 2014-07-15 10:37 /proc/self/ns/cgroup -> cgroup:[4026531835]
1515 # cat /proc/self/cgroup
1516 0::/batchjobs/container_id1
1517
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001518After unsharing a new namespace, the view changes::
Serge Hallynd4021f62016-01-29 02:54:10 -06001519
1520 # ls -l /proc/self/ns/cgroup
1521 lrwxrwxrwx 1 root root 0 2014-07-15 10:35 /proc/self/ns/cgroup -> cgroup:[4026532183]
1522 # cat /proc/self/cgroup
1523 0::/
1524
1525When some thread from a multi-threaded process unshares its cgroup
1526namespace, the new cgroupns gets applied to the entire process (all
1527the threads). This is natural for the v2 hierarchy; however, for the
1528legacy hierarchies, this may be unexpected.
1529
1530A cgroup namespace is alive as long as there are processes inside or
1531mounts pinning it. When the last usage goes away, the cgroup
1532namespace is destroyed. The cgroupns root and the actual cgroups
1533remain.
1534
1535
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001536The Root and Views
1537------------------
Serge Hallynd4021f62016-01-29 02:54:10 -06001538
1539The 'cgroupns root' for a cgroup namespace is the cgroup in which the
1540process calling unshare(2) is running. For example, if a process in
1541/batchjobs/container_id1 cgroup calls unshare, cgroup
1542/batchjobs/container_id1 becomes the cgroupns root. For the
1543init_cgroup_ns, this is the real root ('/') cgroup.
1544
1545The cgroupns root cgroup does not change even if the namespace creator
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001546process later moves to a different cgroup::
Serge Hallynd4021f62016-01-29 02:54:10 -06001547
1548 # ~/unshare -c # unshare cgroupns in some cgroup
1549 # cat /proc/self/cgroup
1550 0::/
1551 # mkdir sub_cgrp_1
1552 # echo 0 > sub_cgrp_1/cgroup.procs
1553 # cat /proc/self/cgroup
1554 0::/sub_cgrp_1
1555
1556Each process gets its namespace-specific view of "/proc/$PID/cgroup"
1557
1558Processes running inside the cgroup namespace will be able to see
1559cgroup paths (in /proc/self/cgroup) only inside their root cgroup.
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001560From within an unshared cgroupns::
Serge Hallynd4021f62016-01-29 02:54:10 -06001561
1562 # sleep 100000 &
1563 [1] 7353
1564 # echo 7353 > sub_cgrp_1/cgroup.procs
1565 # cat /proc/7353/cgroup
1566 0::/sub_cgrp_1
1567
1568From the initial cgroup namespace, the real cgroup path will be
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001569visible::
Serge Hallynd4021f62016-01-29 02:54:10 -06001570
1571 $ cat /proc/7353/cgroup
1572 0::/batchjobs/container_id1/sub_cgrp_1
1573
1574From a sibling cgroup namespace (that is, a namespace rooted at a
1575different cgroup), the cgroup path relative to its own cgroup
1576namespace root will be shown. For instance, if PID 7353's cgroup
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001577namespace root is at '/batchjobs/container_id2', then it will see::
Serge Hallynd4021f62016-01-29 02:54:10 -06001578
1579 # cat /proc/7353/cgroup
1580 0::/../container_id2/sub_cgrp_1
1581
1582Note that the relative path always starts with '/' to indicate that
1583its relative to the cgroup namespace root of the caller.
1584
1585
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001586Migration and setns(2)
1587----------------------
Serge Hallynd4021f62016-01-29 02:54:10 -06001588
1589Processes inside a cgroup namespace can move into and out of the
1590namespace root if they have proper access to external cgroups. For
1591example, from inside a namespace with cgroupns root at
1592/batchjobs/container_id1, and assuming that the global hierarchy is
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001593still accessible inside cgroupns::
Serge Hallynd4021f62016-01-29 02:54:10 -06001594
1595 # cat /proc/7353/cgroup
1596 0::/sub_cgrp_1
1597 # echo 7353 > batchjobs/container_id2/cgroup.procs
1598 # cat /proc/7353/cgroup
1599 0::/../container_id2
1600
1601Note that this kind of setup is not encouraged. A task inside cgroup
1602namespace should only be exposed to its own cgroupns hierarchy.
1603
1604setns(2) to another cgroup namespace is allowed when:
1605
1606(a) the process has CAP_SYS_ADMIN against its current user namespace
1607(b) the process has CAP_SYS_ADMIN against the target cgroup
1608 namespace's userns
1609
1610No implicit cgroup changes happen with attaching to another cgroup
1611namespace. It is expected that the someone moves the attaching
1612process under the target cgroup namespace root.
1613
1614
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001615Interaction with Other Namespaces
1616---------------------------------
Serge Hallynd4021f62016-01-29 02:54:10 -06001617
1618Namespace specific cgroup hierarchy can be mounted by a process
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001619running inside a non-init cgroup namespace::
Serge Hallynd4021f62016-01-29 02:54:10 -06001620
1621 # mount -t cgroup2 none $MOUNT_POINT
1622
1623This will mount the unified cgroup hierarchy with cgroupns root as the
1624filesystem root. The process needs CAP_SYS_ADMIN against its user and
1625mount namespaces.
1626
1627The virtualization of /proc/self/cgroup file combined with restricting
1628the view of cgroup hierarchy by namespace-private cgroupfs mount
1629provides a properly isolated cgroup view inside the container.
1630
1631
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001632Information on Kernel Programming
1633=================================
Tejun Heo6c292092015-11-16 11:13:34 -05001634
1635This section contains kernel programming information in the areas
1636where interacting with cgroup is necessary. cgroup core and
1637controllers are not covered.
1638
1639
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001640Filesystem Support for Writeback
1641--------------------------------
Tejun Heo6c292092015-11-16 11:13:34 -05001642
1643A filesystem can support cgroup writeback by updating
1644address_space_operations->writepage[s]() to annotate bio's using the
1645following two functions.
1646
1647 wbc_init_bio(@wbc, @bio)
Tejun Heo6c292092015-11-16 11:13:34 -05001648 Should be called for each bio carrying writeback data and
1649 associates the bio with the inode's owner cgroup. Can be
1650 called anytime between bio allocation and submission.
1651
1652 wbc_account_io(@wbc, @page, @bytes)
Tejun Heo6c292092015-11-16 11:13:34 -05001653 Should be called for each data segment being written out.
1654 While this function doesn't care exactly when it's called
1655 during the writeback session, it's the easiest and most
1656 natural to call it as data segments are added to a bio.
1657
1658With writeback bio's annotated, cgroup support can be enabled per
1659super_block by setting SB_I_CGROUPWB in ->s_iflags. This allows for
1660selective disabling of cgroup writeback support which is helpful when
1661certain filesystem features, e.g. journaled data mode, are
1662incompatible.
1663
1664wbc_init_bio() binds the specified bio to its cgroup. Depending on
1665the configuration, the bio may be executed at a lower priority and if
1666the writeback session is holding shared resources, e.g. a journal
1667entry, may lead to priority inversion. There is no one easy solution
1668for the problem. Filesystems can try to work around specific problem
1669cases by skipping wbc_init_bio() or using bio_associate_blkcg()
1670directly.
1671
1672
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001673Deprecated v1 Core Features
1674===========================
Tejun Heo6c292092015-11-16 11:13:34 -05001675
1676- Multiple hierarchies including named ones are not supported.
1677
Tejun Heo5136f632017-06-27 14:30:28 -04001678- All v1 mount options are not supported.
Tejun Heo6c292092015-11-16 11:13:34 -05001679
1680- The "tasks" file is removed and "cgroup.procs" is not sorted.
1681
1682- "cgroup.clone_children" is removed.
1683
1684- /proc/cgroups is meaningless for v2. Use "cgroup.controllers" file
1685 at the root instead.
1686
1687
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001688Issues with v1 and Rationales for v2
1689====================================
Tejun Heo6c292092015-11-16 11:13:34 -05001690
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001691Multiple Hierarchies
1692--------------------
Tejun Heo6c292092015-11-16 11:13:34 -05001693
1694cgroup v1 allowed an arbitrary number of hierarchies and each
1695hierarchy could host any number of controllers. While this seemed to
1696provide a high level of flexibility, it wasn't useful in practice.
1697
1698For example, as there is only one instance of each controller, utility
1699type controllers such as freezer which can be useful in all
1700hierarchies could only be used in one. The issue is exacerbated by
1701the fact that controllers couldn't be moved to another hierarchy once
1702hierarchies were populated. Another issue was that all controllers
1703bound to a hierarchy were forced to have exactly the same view of the
1704hierarchy. It wasn't possible to vary the granularity depending on
1705the specific controller.
1706
1707In practice, these issues heavily limited which controllers could be
1708put on the same hierarchy and most configurations resorted to putting
1709each controller on its own hierarchy. Only closely related ones, such
1710as the cpu and cpuacct controllers, made sense to be put on the same
1711hierarchy. This often meant that userland ended up managing multiple
1712similar hierarchies repeating the same steps on each hierarchy
1713whenever a hierarchy management operation was necessary.
1714
1715Furthermore, support for multiple hierarchies came at a steep cost.
1716It greatly complicated cgroup core implementation but more importantly
1717the support for multiple hierarchies restricted how cgroup could be
1718used in general and what controllers was able to do.
1719
1720There was no limit on how many hierarchies there might be, which meant
1721that a thread's cgroup membership couldn't be described in finite
1722length. The key might contain any number of entries and was unlimited
1723in length, which made it highly awkward to manipulate and led to
1724addition of controllers which existed only to identify membership,
1725which in turn exacerbated the original problem of proliferating number
1726of hierarchies.
1727
1728Also, as a controller couldn't have any expectation regarding the
1729topologies of hierarchies other controllers might be on, each
1730controller had to assume that all other controllers were attached to
1731completely orthogonal hierarchies. This made it impossible, or at
1732least very cumbersome, for controllers to cooperate with each other.
1733
1734In most use cases, putting controllers on hierarchies which are
1735completely orthogonal to each other isn't necessary. What usually is
1736called for is the ability to have differing levels of granularity
1737depending on the specific controller. In other words, hierarchy may
1738be collapsed from leaf towards root when viewed from specific
1739controllers. For example, a given configuration might not care about
1740how memory is distributed beyond a certain level while still wanting
1741to control how CPU cycles are distributed.
1742
1743
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001744Thread Granularity
1745------------------
Tejun Heo6c292092015-11-16 11:13:34 -05001746
1747cgroup v1 allowed threads of a process to belong to different cgroups.
1748This didn't make sense for some controllers and those controllers
1749ended up implementing different ways to ignore such situations but
1750much more importantly it blurred the line between API exposed to
1751individual applications and system management interface.
1752
1753Generally, in-process knowledge is available only to the process
1754itself; thus, unlike service-level organization of processes,
1755categorizing threads of a process requires active participation from
1756the application which owns the target process.
1757
1758cgroup v1 had an ambiguously defined delegation model which got abused
1759in combination with thread granularity. cgroups were delegated to
1760individual applications so that they can create and manage their own
1761sub-hierarchies and control resource distributions along them. This
1762effectively raised cgroup to the status of a syscall-like API exposed
1763to lay programs.
1764
1765First of all, cgroup has a fundamentally inadequate interface to be
1766exposed this way. For a process to access its own knobs, it has to
1767extract the path on the target hierarchy from /proc/self/cgroup,
1768construct the path by appending the name of the knob to the path, open
1769and then read and/or write to it. This is not only extremely clunky
1770and unusual but also inherently racy. There is no conventional way to
1771define transaction across the required steps and nothing can guarantee
1772that the process would actually be operating on its own sub-hierarchy.
1773
1774cgroup controllers implemented a number of knobs which would never be
1775accepted as public APIs because they were just adding control knobs to
1776system-management pseudo filesystem. cgroup ended up with interface
1777knobs which were not properly abstracted or refined and directly
1778revealed kernel internal details. These knobs got exposed to
1779individual applications through the ill-defined delegation mechanism
1780effectively abusing cgroup as a shortcut to implementing public APIs
1781without going through the required scrutiny.
1782
1783This was painful for both userland and kernel. Userland ended up with
1784misbehaving and poorly abstracted interfaces and kernel exposing and
1785locked into constructs inadvertently.
1786
1787
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001788Competition Between Inner Nodes and Threads
1789-------------------------------------------
Tejun Heo6c292092015-11-16 11:13:34 -05001790
1791cgroup v1 allowed threads to be in any cgroups which created an
1792interesting problem where threads belonging to a parent cgroup and its
1793children cgroups competed for resources. This was nasty as two
1794different types of entities competed and there was no obvious way to
1795settle it. Different controllers did different things.
1796
1797The cpu controller considered threads and cgroups as equivalents and
1798mapped nice levels to cgroup weights. This worked for some cases but
1799fell flat when children wanted to be allocated specific ratios of CPU
1800cycles and the number of internal threads fluctuated - the ratios
1801constantly changed as the number of competing entities fluctuated.
1802There also were other issues. The mapping from nice level to weight
1803wasn't obvious or universal, and there were various other knobs which
1804simply weren't available for threads.
1805
1806The io controller implicitly created a hidden leaf node for each
1807cgroup to host the threads. The hidden leaf had its own copies of all
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001808the knobs with ``leaf_`` prefixed. While this allowed equivalent
Tejun Heo6c292092015-11-16 11:13:34 -05001809control over internal threads, it was with serious drawbacks. It
1810always added an extra layer of nesting which wouldn't be necessary
1811otherwise, made the interface messy and significantly complicated the
1812implementation.
1813
1814The memory controller didn't have a way to control what happened
1815between internal tasks and child cgroups and the behavior was not
1816clearly defined. There were attempts to add ad-hoc behaviors and
1817knobs to tailor the behavior to specific workloads which would have
1818led to problems extremely difficult to resolve in the long term.
1819
1820Multiple controllers struggled with internal tasks and came up with
1821different ways to deal with it; unfortunately, all the approaches were
1822severely flawed and, furthermore, the widely different behaviors
1823made cgroup as a whole highly inconsistent.
1824
1825This clearly is a problem which needs to be addressed from cgroup core
1826in a uniform way.
1827
1828
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001829Other Interface Issues
1830----------------------
Tejun Heo6c292092015-11-16 11:13:34 -05001831
1832cgroup v1 grew without oversight and developed a large number of
1833idiosyncrasies and inconsistencies. One issue on the cgroup core side
1834was how an empty cgroup was notified - a userland helper binary was
1835forked and executed for each event. The event delivery wasn't
1836recursive or delegatable. The limitations of the mechanism also led
1837to in-kernel event delivery filtering mechanism further complicating
1838the interface.
1839
1840Controller interfaces were problematic too. An extreme example is
1841controllers completely ignoring hierarchical organization and treating
1842all cgroups as if they were all located directly under the root
1843cgroup. Some controllers exposed a large amount of inconsistent
1844implementation details to userland.
1845
1846There also was no consistency across controllers. When a new cgroup
1847was created, some controllers defaulted to not imposing extra
1848restrictions while others disallowed any resource usage until
1849explicitly configured. Configuration knobs for the same type of
1850control used widely differing naming schemes and formats. Statistics
1851and information knobs were named arbitrarily and used different
1852formats and units even in the same controller.
1853
1854cgroup v2 establishes common conventions where appropriate and updates
1855controllers so that they expose minimal and consistent interfaces.
1856
1857
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001858Controller Issues and Remedies
1859------------------------------
Tejun Heo6c292092015-11-16 11:13:34 -05001860
Mauro Carvalho Chehab633b11b2017-05-14 08:48:40 -03001861Memory
1862~~~~~~
Tejun Heo6c292092015-11-16 11:13:34 -05001863
1864The original lower boundary, the soft limit, is defined as a limit
1865that is per default unset. As a result, the set of cgroups that
1866global reclaim prefers is opt-in, rather than opt-out. The costs for
1867optimizing these mostly negative lookups are so high that the
1868implementation, despite its enormous size, does not even provide the
1869basic desirable behavior. First off, the soft limit has no
1870hierarchical meaning. All configured groups are organized in a global
1871rbtree and treated like equal peers, regardless where they are located
1872in the hierarchy. This makes subtree delegation impossible. Second,
1873the soft limit reclaim pass is so aggressive that it not just
1874introduces high allocation latencies into the system, but also impacts
1875system performance due to overreclaim, to the point where the feature
1876becomes self-defeating.
1877
1878The memory.low boundary on the other hand is a top-down allocated
1879reserve. A cgroup enjoys reclaim protection when it and all its
1880ancestors are below their low boundaries, which makes delegation of
1881subtrees possible. Secondly, new cgroups have no reserve per default
1882and in the common case most cgroups are eligible for the preferred
1883reclaim pass. This allows the new low boundary to be efficiently
1884implemented with just a minor addition to the generic reclaim code,
1885without the need for out-of-band data structures and reclaim passes.
1886Because the generic reclaim code considers all cgroups except for the
1887ones running low in the preferred first reclaim pass, overreclaim of
1888individual groups is eliminated as well, resulting in much better
1889overall workload performance.
1890
1891The original high boundary, the hard limit, is defined as a strict
1892limit that can not budge, even if the OOM killer has to be called.
1893But this generally goes against the goal of making the most out of the
1894available memory. The memory consumption of workloads varies during
1895runtime, and that requires users to overcommit. But doing that with a
1896strict upper limit requires either a fairly accurate prediction of the
1897working set size or adding slack to the limit. Since working set size
1898estimation is hard and error prone, and getting it wrong results in
1899OOM kills, most users tend to err on the side of a looser limit and
1900end up wasting precious resources.
1901
1902The memory.high boundary on the other hand can be set much more
1903conservatively. When hit, it throttles allocations by forcing them
1904into direct reclaim to work off the excess, but it never invokes the
1905OOM killer. As a result, a high boundary that is chosen too
1906aggressively will not terminate the processes, but instead it will
1907lead to gradual performance degradation. The user can monitor this
1908and make corrections until the minimal memory footprint that still
1909gives acceptable performance is found.
1910
1911In extreme cases, with many concurrent allocations and a complete
1912breakdown of reclaim progress within the group, the high boundary can
1913be exceeded. But even then it's mostly better to satisfy the
1914allocation from the slack available in other groups or the rest of the
1915system than killing the group. Otherwise, memory.max is there to
1916limit this type of spillover and ultimately contain buggy or even
1917malicious applications.
Vladimir Davydov3e24b192016-01-20 15:03:13 -08001918
Johannes Weinerb6e6edc2016-03-17 14:20:28 -07001919Setting the original memory.limit_in_bytes below the current usage was
1920subject to a race condition, where concurrent charges could cause the
1921limit setting to fail. memory.max on the other hand will first set the
1922limit to prevent new charges, and then reclaim and OOM kill until the
1923new limit is met - or the task writing to memory.max is killed.
1924
Vladimir Davydov3e24b192016-01-20 15:03:13 -08001925The combined memory+swap accounting and limiting is replaced by real
1926control over swap space.
1927
1928The main argument for a combined memory+swap facility in the original
1929cgroup design was that global or parental pressure would always be
1930able to swap all anonymous memory of a child group, regardless of the
1931child's own (possibly untrusted) configuration. However, untrusted
1932groups can sabotage swapping by other means - such as referencing its
1933anonymous memory in a tight loop - and an admin can not assume full
1934swappability when overcommitting untrusted jobs.
1935
1936For trusted jobs, on the other hand, a combined counter is not an
1937intuitive userspace interface, and it flies in the face of the idea
1938that cgroup controllers should account and limit specific physical
1939resources. Swap space is a resource like all others in the system,
1940and that's why unified hierarchy allows distributing it separately.