blob: 7c871d6beb63c33eaf1c6daeb9d7ad1c260bed4e [file] [log] [blame]
Andrea Arcangeli1c9bf222011-01-13 15:46:30 -08001= Transparent Hugepage Support =
2
3== Objective ==
4
5Performance critical computing applications dealing with large memory
6working sets are already running on top of libhugetlbfs and in turn
7hugetlbfs. Transparent Hugepage Support is an alternative means of
8using huge pages for the backing of virtual memory with huge pages
9that supports the automatic promotion and demotion of page sizes and
10without the shortcomings of hugetlbfs.
11
12Currently it only works for anonymous memory mappings but in the
13future it can expand over the pagecache layer starting with tmpfs.
14
15The reason applications are running faster is because of two
16factors. The first factor is almost completely irrelevant and it's not
17of significant interest because it'll also have the downside of
18requiring larger clear-page copy-page in page faults which is a
19potentially negative effect. The first factor consists in taking a
20single page fault for each 2M virtual region touched by userland (so
21reducing the enter/exit kernel frequency by a 512 times factor). This
22only matters the first time the memory is accessed for the lifetime of
23a memory mapping. The second long lasting and much more important
24factor will affect all subsequent accesses to the memory for the whole
25runtime of the application. The second factor consist of two
26components: 1) the TLB miss will run faster (especially with
27virtualization using nested pagetables but almost always also on bare
28metal without virtualization) and 2) a single TLB entry will be
29mapping a much larger amount of virtual memory in turn reducing the
30number of TLB misses. With virtualization and nested pagetables the
31TLB can be mapped of larger size only if both KVM and the Linux guest
32are using hugepages but a significant speedup already happens if only
33one of the two is using hugepages just because of the fact the TLB
34miss is going to run faster.
35
36== Design ==
37
Kirill A. Shutemova46e6372016-01-15 16:54:30 -080038- "graceful fallback": mm components which don't have transparent hugepage
39 knowledge fall back to breaking huge pmd mapping into table of ptes and,
40 if necessary, split a transparent hugepage. Therefore these components
41 can continue working on the regular pages or regular pte mappings.
Andrea Arcangeli1c9bf222011-01-13 15:46:30 -080042
43- if a hugepage allocation fails because of memory fragmentation,
44 regular pages should be gracefully allocated instead and mixed in
45 the same vma without any failure or significant delay and without
46 userland noticing
47
48- if some task quits and more hugepages become available (either
49 immediately in the buddy or through the VM), guest physical memory
50 backed by regular pages should be relocated on hugepages
51 automatically (with khugepaged)
52
53- it doesn't require memory reservation and in turn it uses hugepages
54 whenever possible (the only possible reservation here is kernelcore=
55 to avoid unmovable pages to fragment all the memory but such a tweak
56 is not specific to transparent hugepage support and it's a generic
57 feature that applies to all dynamic high order allocations in the
58 kernel)
59
60- this initial support only offers the feature in the anonymous memory
61 regions but it'd be ideal to move it to tmpfs and the pagecache
62 later
63
64Transparent Hugepage Support maximizes the usefulness of free memory
65if compared to the reservation approach of hugetlbfs by allowing all
66unused memory to be used as cache or other movable (or even unmovable
67entities). It doesn't require reservation to prevent hugepage
68allocation failures to be noticeable from userland. It allows paging
69and all other advanced VM features to be available on the
70hugepages. It requires no modifications for applications to take
71advantage of it.
72
73Applications however can be further optimized to take advantage of
74this feature, like for example they've been optimized before to avoid
75a flood of mmap system calls for every malloc(4k). Optimizing userland
76is by far not mandatory and khugepaged already can take care of long
77lived page allocations even for hugepage unaware applications that
78deals with large amounts of memory.
79
80In certain cases when hugepages are enabled system wide, application
81may end up allocating more memory resources. An application may mmap a
82large region but only touch 1 byte of it, in that case a 2M page might
83be allocated instead of a 4k page for no good. This is why it's
84possible to disable hugepages system-wide and to only have them inside
85MADV_HUGEPAGE madvise regions.
86
87Embedded systems should enable hugepages only inside madvise regions
88to eliminate any risk of wasting any precious byte of memory and to
89only run faster.
90
91Applications that gets a lot of benefit from hugepages and that don't
92risk to lose memory by using hugepages, should use
93madvise(MADV_HUGEPAGE) on their critical mmapped regions.
94
95== sysfs ==
96
97Transparent Hugepage Support can be entirely disabled (mostly for
98debugging purposes) or only enabled inside MADV_HUGEPAGE regions (to
99avoid the risk of consuming more memory resources) or enabled system
100wide. This can be achieved with one of:
101
102echo always >/sys/kernel/mm/transparent_hugepage/enabled
103echo madvise >/sys/kernel/mm/transparent_hugepage/enabled
104echo never >/sys/kernel/mm/transparent_hugepage/enabled
105
106It's also possible to limit defrag efforts in the VM to generate
107hugepages in case they're not immediately free to madvise regions or
108to never try to defrag memory and simply fallback to regular pages
109unless hugepages are immediately available. Clearly if we spend CPU
110time to defrag memory, we would expect to gain even more by the fact
111we use hugepages later instead of regular pages. This isn't always
112guaranteed, but it may be more likely in case the allocation is for a
113MADV_HUGEPAGE region.
114
115echo always >/sys/kernel/mm/transparent_hugepage/defrag
Mel Gorman444eb2a42016-03-17 14:19:23 -0700116echo defer >/sys/kernel/mm/transparent_hugepage/defrag
Andrea Arcangeli1c9bf222011-01-13 15:46:30 -0800117echo madvise >/sys/kernel/mm/transparent_hugepage/defrag
118echo never >/sys/kernel/mm/transparent_hugepage/defrag
119
Mel Gorman444eb2a42016-03-17 14:19:23 -0700120"always" means that an application requesting THP will stall on allocation
121failure and directly reclaim pages and compact memory in an effort to
122allocate a THP immediately. This may be desirable for virtual machines
123that benefit heavily from THP use and are willing to delay the VM start
124to utilise them.
125
126"defer" means that an application will wake kswapd in the background
127to reclaim pages and wake kcompact to compact memory so that THP is
128available in the near future. It's the responsibility of khugepaged
129to then install the THP pages later.
130
131"madvise" will enter direct reclaim like "always" but only for regions
132that are have used madvise(MADV_HUGEPAGE). This is the default behaviour.
133
134"never" should be self-explanatory.
135
Kirill A. Shutemov79da5402012-12-12 13:51:12 -0800136By default kernel tries to use huge zero page on read page fault.
137It's possible to disable huge zero page by writing 0 or enable it
138back by writing 1:
139
Wanpeng Lif49cbdd2013-07-08 16:00:16 -0700140echo 0 >/sys/kernel/mm/transparent_hugepage/use_zero_page
141echo 1 >/sys/kernel/mm/transparent_hugepage/use_zero_page
Kirill A. Shutemov79da5402012-12-12 13:51:12 -0800142
Andrea Arcangeli1c9bf222011-01-13 15:46:30 -0800143khugepaged will be automatically started when
144transparent_hugepage/enabled is set to "always" or "madvise, and it'll
145be automatically shutdown if it's set to "never".
146
147khugepaged runs usually at low frequency so while one may not want to
148invoke defrag algorithms synchronously during the page faults, it
149should be worth invoking defrag at least in khugepaged. However it's
David Rientjese369fde2011-09-22 14:11:38 -0700150also possible to disable defrag in khugepaged by writing 0 or enable
151defrag in khugepaged by writing 1:
Andrea Arcangeli1c9bf222011-01-13 15:46:30 -0800152
David Rientjese369fde2011-09-22 14:11:38 -0700153echo 0 >/sys/kernel/mm/transparent_hugepage/khugepaged/defrag
154echo 1 >/sys/kernel/mm/transparent_hugepage/khugepaged/defrag
Andrea Arcangeli1c9bf222011-01-13 15:46:30 -0800155
156You can also control how many pages khugepaged should scan at each
157pass:
158
159/sys/kernel/mm/transparent_hugepage/khugepaged/pages_to_scan
160
161and how many milliseconds to wait in khugepaged between each pass (you
162can set this to 0 to run khugepaged at 100% utilization of one core):
163
164/sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs
165
166and how many milliseconds to wait in khugepaged if there's an hugepage
167allocation failure to throttle the next allocation attempt.
168
169/sys/kernel/mm/transparent_hugepage/khugepaged/alloc_sleep_millisecs
170
171The khugepaged progress can be seen in the number of pages collapsed:
172
173/sys/kernel/mm/transparent_hugepage/khugepaged/pages_collapsed
174
175for each pass:
176
177/sys/kernel/mm/transparent_hugepage/khugepaged/full_scans
178
Ebru Akagunduz9ddfa692015-02-26 23:34:36 +0200179max_ptes_none specifies how many extra small pages (that are
180not already mapped) can be allocated when collapsing a group
181of small pages into one large page.
182
183/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none
184
185A higher value leads to use additional memory for programs.
186A lower value leads to gain less thp performance. Value of
187max_ptes_none can waste cpu time very little, you can
188ignore it.
189
Ebru Akagunduz80f73b42015-11-05 18:47:32 -0800190max_ptes_swap specifies how many pages can be brought in from
191swap when collapsing a group of pages into a transparent huge page.
192
193/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_swap
194
195A higher value can cause excessive swap IO and waste
196memory. A lower value can prevent THPs from being
197collapsed, resulting fewer pages being collapsed into
198THPs, and lower memory access performance.
199
Andrea Arcangeli1c9bf222011-01-13 15:46:30 -0800200== Boot parameter ==
201
202You can change the sysfs boot time defaults of Transparent Hugepage
203Support by passing the parameter "transparent_hugepage=always" or
204"transparent_hugepage=madvise" or "transparent_hugepage=never"
205(without "") to the kernel command line.
206
207== Need of application restart ==
208
209The transparent_hugepage/enabled values only affect future
210behavior. So to make them effective you need to restart any
211application that could have been using hugepages. This also applies to
212the regions registered in khugepaged.
213
Mel Gorman69256992012-05-29 15:06:45 -0700214== Monitoring usage ==
215
216The number of transparent huge pages currently used by the system is
217available by reading the AnonHugePages field in /proc/meminfo. To
218identify what applications are using transparent huge pages, it is
219necessary to read /proc/PID/smaps and count the AnonHugePages fields
220for each mapping. Note that reading the smaps file is expensive and
221reading it frequently will incur overhead.
222
223There are a number of counters in /proc/vmstat that may be used to
224monitor how successfully the system is providing huge pages for use.
225
226thp_fault_alloc is incremented every time a huge page is successfully
227 allocated to handle a page fault. This applies to both the
228 first time a page is faulted and for COW faults.
229
230thp_collapse_alloc is incremented by khugepaged when it has found
231 a range of pages to collapse into one huge page and has
232 successfully allocated a new huge page to store the data.
233
234thp_fault_fallback is incremented if a page fault fails to allocate
235 a huge page and instead falls back to using small pages.
236
237thp_collapse_alloc_failed is incremented if khugepaged found a range
238 of pages that should be collapsed into one huge page but failed
239 the allocation.
240
Kirill A. Shutemova46e6372016-01-15 16:54:30 -0800241thp_split_page is incremented every time a huge page is split into base
Mel Gorman69256992012-05-29 15:06:45 -0700242 pages. This can happen for a variety of reasons but a common
243 reason is that a huge page is old and is being reclaimed.
Kirill A. Shutemova46e6372016-01-15 16:54:30 -0800244 This action implies splitting all PMD the page mapped with.
245
246thp_split_page_failed is is incremented if kernel fails to split huge
247 page. This can happen if the page was pinned by somebody.
248
Kirill A. Shutemovf9719a02016-03-17 14:18:45 -0700249thp_deferred_split_page is incremented when a huge page is put onto split
250 queue. This happens when a huge page is partially unmapped and
251 splitting it would free up some memory. Pages on split queue are
252 going to be split under memory pressure.
253
Kirill A. Shutemova46e6372016-01-15 16:54:30 -0800254thp_split_pmd is incremented every time a PMD split into table of PTEs.
255 This can happen, for instance, when application calls mprotect() or
256 munmap() on part of huge page. It doesn't split huge page, only
257 page table entry.
Mel Gorman69256992012-05-29 15:06:45 -0700258
Kirill A. Shutemovd8a8e1f2012-12-12 13:51:09 -0800259thp_zero_page_alloc is incremented every time a huge zero page is
260 successfully allocated. It includes allocations which where
261 dropped due race with other allocation. Note, it doesn't count
262 every map of the huge zero page, only its allocation.
263
264thp_zero_page_alloc_failed is incremented if kernel fails to allocate
265 huge zero page and falls back to using small pages.
266
Mel Gorman69256992012-05-29 15:06:45 -0700267As the system ages, allocating huge pages may be expensive as the
268system uses memory compaction to copy data around memory to free a
269huge page for use. There are some counters in /proc/vmstat to help
270monitor this overhead.
271
272compact_stall is incremented every time a process stalls to run
273 memory compaction so that a huge page is free for use.
274
275compact_success is incremented if the system compacted memory and
276 freed a huge page for use.
277
278compact_fail is incremented if the system tries to compact memory
279 but failed.
280
281compact_pages_moved is incremented each time a page is moved. If
282 this value is increasing rapidly, it implies that the system
283 is copying a lot of data to satisfy the huge page allocation.
284 It is possible that the cost of copying exceeds any savings
285 from reduced TLB misses.
286
287compact_pagemigrate_failed is incremented when the underlying mechanism
288 for moving a page failed.
289
290compact_blocks_moved is incremented each time memory compaction examines
291 a huge page aligned range of pages.
292
293It is possible to establish how long the stalls were using the function
294tracer to record how long was spent in __alloc_pages_nodemask and
295using the mm_page_alloc tracepoint to identify which allocations were
296for huge pages.
297
Andrea Arcangeli1c9bf222011-01-13 15:46:30 -0800298== get_user_pages and follow_page ==
299
300get_user_pages and follow_page if run on a hugepage, will return the
301head or tail pages as usual (exactly as they would do on
302hugetlbfs). Most gup users will only care about the actual physical
303address of the page and its temporary pinning to release after the I/O
304is complete, so they won't ever notice the fact the page is huge. But
305if any driver is going to mangle over the page structure of the tail
306page (like for checking page->mapping or other bits that are relevant
307for the head page and not the tail page), it should be updated to jump
Kirill A. Shutemova46e6372016-01-15 16:54:30 -0800308to check head page instead. Taking reference on any head/tail page would
309prevent page from being split by anyone.
Andrea Arcangeli1c9bf222011-01-13 15:46:30 -0800310
311NOTE: these aren't new constraints to the GUP API, and they match the
312same constrains that applies to hugetlbfs too, so any driver capable
313of handling GUP on hugetlbfs will also work fine on transparent
314hugepage backed mappings.
315
316In case you can't handle compound pages if they're returned by
317follow_page, the FOLL_SPLIT bit can be specified as parameter to
318follow_page, so that it will split the hugepages before returning
319them. Migration for example passes FOLL_SPLIT as parameter to
320follow_page because it's not hugepage aware and in fact it can't work
321at all on hugetlbfs (but it instead works fine on transparent
322hugepages thanks to FOLL_SPLIT). migration simply can't deal with
323hugepages being returned (as it's not only checking the pfn of the
324page and pinning it during the copy but it pretends to migrate the
325memory in regular page sizes and with regular pte/pmd mappings).
326
327== Optimizing the applications ==
328
329To be guaranteed that the kernel will map a 2M page immediately in any
330memory region, the mmap region has to be hugepage naturally
331aligned. posix_memalign() can provide that guarantee.
332
333== Hugetlbfs ==
334
335You can use hugetlbfs on a kernel that has transparent hugepage
336support enabled just fine as always. No difference can be noted in
337hugetlbfs other than there will be less overall fragmentation. All
338usual features belonging to hugetlbfs are preserved and
339unaffected. libhugetlbfs will also work fine as usual.
340
341== Graceful fallback ==
342
Eric Engestrom89474d52016-05-20 16:58:07 -0700343Code walking pagetables but unaware about huge pmds can simply call
Kirill A. Shutemova46e6372016-01-15 16:54:30 -0800344split_huge_pmd(vma, pmd, addr) where the pmd is the one returned by
Andrea Arcangeli1c9bf222011-01-13 15:46:30 -0800345pmd_offset. It's trivial to make the code transparent hugepage aware
Kirill A. Shutemova46e6372016-01-15 16:54:30 -0800346by just grepping for "pmd_offset" and adding split_huge_pmd where
Andrea Arcangeli1c9bf222011-01-13 15:46:30 -0800347missing after pmd_offset returns the pmd. Thanks to the graceful
348fallback design, with a one liner change, you can avoid to write
349hundred if not thousand of lines of complex code to make your code
350hugepage aware.
351
352If you're not walking pagetables but you run into a physical hugepage
353but you can't handle it natively in your code, you can split it by
354calling split_huge_page(page). This is what the Linux VM does before
Kirill A. Shutemova46e6372016-01-15 16:54:30 -0800355it tries to swapout the hugepage for example. split_huge_page() can fail
356if the page is pinned and you must handle this correctly.
Andrea Arcangeli1c9bf222011-01-13 15:46:30 -0800357
358Example to make mremap.c transparent hugepage aware with a one liner
359change:
360
361diff --git a/mm/mremap.c b/mm/mremap.c
362--- a/mm/mremap.c
363+++ b/mm/mremap.c
364@@ -41,6 +41,7 @@ static pmd_t *get_old_pmd(struct mm_stru
365 return NULL;
366
367 pmd = pmd_offset(pud, addr);
Kirill A. Shutemova46e6372016-01-15 16:54:30 -0800368+ split_huge_pmd(vma, pmd, addr);
Andrea Arcangeli1c9bf222011-01-13 15:46:30 -0800369 if (pmd_none_or_clear_bad(pmd))
370 return NULL;
371
372== Locking in hugepage aware code ==
373
374We want as much code as possible hugepage aware, as calling
Kirill A. Shutemova46e6372016-01-15 16:54:30 -0800375split_huge_page() or split_huge_pmd() has a cost.
Andrea Arcangeli1c9bf222011-01-13 15:46:30 -0800376
377To make pagetable walks huge pmd aware, all you need to do is to call
378pmd_trans_huge() on the pmd returned by pmd_offset. You must hold the
379mmap_sem in read (or write) mode to be sure an huge pmd cannot be
380created from under you by khugepaged (khugepaged collapse_huge_page
381takes the mmap_sem in write mode in addition to the anon_vma lock). If
382pmd_trans_huge returns false, you just fallback in the old code
383paths. If instead pmd_trans_huge returns true, you have to take the
Kirill A. Shutemova46e6372016-01-15 16:54:30 -0800384page table lock (pmd_lock()) and re-run pmd_trans_huge. Taking the
385page table lock will prevent the huge pmd to be converted into a
386regular pmd from under you (split_huge_pmd can run in parallel to the
Andrea Arcangeli1c9bf222011-01-13 15:46:30 -0800387pagetable walk). If the second pmd_trans_huge returns false, you
Kirill A. Shutemova46e6372016-01-15 16:54:30 -0800388should just drop the page table lock and fallback to the old code as
389before. Otherwise you can proceed to process the huge pmd and the
390hugepage natively. Once finished you can drop the page table lock.
Andrea Arcangeli1c9bf222011-01-13 15:46:30 -0800391
Kirill A. Shutemova46e6372016-01-15 16:54:30 -0800392== Refcounts and transparent huge pages ==
393
394Refcounting on THP is mostly consistent with refcounting on other compound
395pages:
396
Joonsoo Kim0139aa72016-05-19 17:10:49 -0700397 - get_page()/put_page() and GUP operate in head page's ->_refcount.
Kirill A. Shutemova46e6372016-01-15 16:54:30 -0800398
Joonsoo Kim0139aa72016-05-19 17:10:49 -0700399 - ->_refcount in tail pages is always zero: get_page_unless_zero() never
Kirill A. Shutemova46e6372016-01-15 16:54:30 -0800400 succeed on tail pages.
401
402 - map/unmap of the pages with PTE entry increment/decrement ->_mapcount
403 on relevant sub-page of the compound page.
404
405 - map/unmap of the whole compound page accounted in compound_mapcount
406 (stored in first tail page).
407
408PageDoubleMap() indicates that ->_mapcount in all subpages is offset up by one.
409This additional reference is required to get race-free detection of unmap of
410subpages when we have them mapped with both PMDs and PTEs.
411
412This is optimization required to lower overhead of per-subpage mapcount
413tracking. The alternative is alter ->_mapcount in all subpages on each
414map/unmap of the whole compound page.
415
416We set PG_double_map when a PMD of the page got split for the first time,
Eric Engestrom89474d52016-05-20 16:58:07 -0700417but still have PMD mapping. The additional references go away with last
Kirill A. Shutemova46e6372016-01-15 16:54:30 -0800418compound_mapcount.
Andrea Arcangeli1c9bf222011-01-13 15:46:30 -0800419
420split_huge_page internally has to distribute the refcounts in the head
Kirill A. Shutemova46e6372016-01-15 16:54:30 -0800421page to the tail pages before clearing all PG_head/tail bits from the page
422structures. It can be done easily for refcounts taken by page table
423entries. But we don't have enough information on how to distribute any
424additional pins (i.e. from get_user_pages). split_huge_page() fails any
425requests to split pinned huge page: it expects page count to be equal to
426sum of mapcount of all sub-pages plus one (split_huge_page caller must
427have reference for head page).
428
Joonsoo Kim0139aa72016-05-19 17:10:49 -0700429split_huge_page uses migration entries to stabilize page->_refcount and
Kirill A. Shutemova46e6372016-01-15 16:54:30 -0800430page->_mapcount.
431
432We safe against physical memory scanners too: the only legitimate way
433scanner can get reference to a page is get_page_unless_zero().
434
Eric Engestrom89474d52016-05-20 16:58:07 -0700435All tail pages have zero ->_refcount until atomic_add(). This prevents the
436scanner from getting a reference to the tail page up to that point. After the
437atomic_add() we don't care about the ->_refcount value. We already known how
438many references should be uncharged from the head page.
Kirill A. Shutemova46e6372016-01-15 16:54:30 -0800439
440For head page get_page_unless_zero() will succeed and we don't mind. It's
441clear where reference should go after split: it will stay on head page.
442
443Note that split_huge_pmd() doesn't have any limitation on refcounting:
444pmd can be split at any point and never fails.
445
446== Partial unmap and deferred_split_huge_page() ==
447
448Unmapping part of THP (with munmap() or other way) is not going to free
449memory immediately. Instead, we detect that a subpage of THP is not in use
450in page_remove_rmap() and queue the THP for splitting if memory pressure
451comes. Splitting will free up unused subpages.
452
453Splitting the page right away is not an option due to locking context in
454the place where we can detect partial unmap. It's also might be
455counterproductive since in many cases partial unmap unmap happens during
456exit(2) if an THP crosses VMA boundary.
457
458Function deferred_split_huge_page() is used to queue page for splitting.
459The splitting itself will happen when we get memory pressure via shrinker
460interface.