David Rientjes | 2028019 | 2007-05-02 19:27:09 +0200 | [diff] [blame] | 1 | Using numa=fake and CPUSets for Resource Management |
| 2 | Written by David Rientjes <rientjes@cs.washington.edu> |
| 3 | |
| 4 | This document describes how the numa=fake x86_64 command-line option can be used |
| 5 | in conjunction with cpusets for coarse memory management. Using this feature, |
| 6 | you can create fake NUMA nodes that represent contiguous chunks of memory and |
| 7 | assign them to cpusets and their attached tasks. This is a way of limiting the |
| 8 | amount of system memory that are available to a certain class of tasks. |
| 9 | |
Thadeu Lima de Souza Cascardo | 21acb9c | 2009-02-04 10:12:08 +0100 | [diff] [blame] | 10 | For more information on the features of cpusets, see |
seokhoon.yoon | 09c3bcc | 2016-08-02 23:23:57 +0900 | [diff] [blame] | 11 | Documentation/cgroup-v1/cpusets.txt. |
David Rientjes | 2028019 | 2007-05-02 19:27:09 +0200 | [diff] [blame] | 12 | There are a number of different configurations you can use for your needs. For |
| 13 | more information on the numa=fake command line option and its various ways of |
Uwe Hermann | 71cced6 | 2008-10-20 09:32:21 -0700 | [diff] [blame] | 14 | configuring fake nodes, see Documentation/x86/x86_64/boot-options.txt. |
David Rientjes | 2028019 | 2007-05-02 19:27:09 +0200 | [diff] [blame] | 15 | |
| 16 | For the purposes of this introduction, we'll assume a very primitive NUMA |
| 17 | emulation setup of "numa=fake=4*512,". This will split our system memory into |
| 18 | four equal chunks of 512M each that we can now use to assign to cpusets. As |
| 19 | you become more familiar with using this combination for resource control, |
| 20 | you'll determine a better setup to minimize the number of nodes you have to deal |
| 21 | with. |
| 22 | |
| 23 | A machine may be split as follows with "numa=fake=4*512," as reported by dmesg: |
| 24 | |
| 25 | Faking node 0 at 0000000000000000-0000000020000000 (512MB) |
| 26 | Faking node 1 at 0000000020000000-0000000040000000 (512MB) |
| 27 | Faking node 2 at 0000000040000000-0000000060000000 (512MB) |
| 28 | Faking node 3 at 0000000060000000-0000000080000000 (512MB) |
| 29 | ... |
| 30 | On node 0 totalpages: 130975 |
| 31 | On node 1 totalpages: 131072 |
| 32 | On node 2 totalpages: 131072 |
| 33 | On node 3 totalpages: 131072 |
| 34 | |
| 35 | Now following the instructions for mounting the cpusets filesystem from |
seokhoon.yoon | 09c3bcc | 2016-08-02 23:23:57 +0900 | [diff] [blame] | 36 | Documentation/cgroup-v1/cpusets.txt, you can assign fake nodes (i.e. contiguous memory |
David Rientjes | 2028019 | 2007-05-02 19:27:09 +0200 | [diff] [blame] | 37 | address spaces) to individual cpusets: |
| 38 | |
| 39 | [root@xroads /]# mkdir exampleset |
| 40 | [root@xroads /]# mount -t cpuset none exampleset |
| 41 | [root@xroads /]# mkdir exampleset/ddset |
| 42 | [root@xroads /]# cd exampleset/ddset |
| 43 | [root@xroads /exampleset/ddset]# echo 0-1 > cpus |
| 44 | [root@xroads /exampleset/ddset]# echo 0-1 > mems |
| 45 | |
| 46 | Now this cpuset, 'ddset', will only allowed access to fake nodes 0 and 1 for |
| 47 | memory allocations (1G). |
| 48 | |
| 49 | You can now assign tasks to these cpusets to limit the memory resources |
| 50 | available to them according to the fake nodes assigned as mems: |
| 51 | |
| 52 | [root@xroads /exampleset/ddset]# echo $$ > tasks |
| 53 | [root@xroads /exampleset/ddset]# dd if=/dev/zero of=tmp bs=1024 count=1G |
| 54 | [1] 13425 |
| 55 | |
| 56 | Notice the difference between the system memory usage as reported by |
| 57 | /proc/meminfo between the restricted cpuset case above and the unrestricted |
| 58 | case (i.e. running the same 'dd' command without assigning it to a fake NUMA |
| 59 | cpuset): |
| 60 | Unrestricted Restricted |
| 61 | MemTotal: 3091900 kB 3091900 kB |
| 62 | MemFree: 42113 kB 1513236 kB |
| 63 | |
| 64 | This allows for coarse memory management for the tasks you assign to particular |
| 65 | cpusets. Since cpusets can form a hierarchy, you can create some pretty |
| 66 | interesting combinations of use-cases for various classes of tasks for your |
| 67 | memory management needs. |