David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 1 | ============================== |
| 2 | UNEVICTABLE LRU INFRASTRUCTURE |
| 3 | ============================== |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 4 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 5 | ======== |
| 6 | CONTENTS |
| 7 | ======== |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 8 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 9 | (*) The Unevictable LRU |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 10 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 11 | - The unevictable page list. |
| 12 | - Memory control group interaction. |
| 13 | - Marking address spaces unevictable. |
| 14 | - Detecting Unevictable Pages. |
| 15 | - vmscan's handling of unevictable pages. |
| 16 | |
| 17 | (*) mlock()'d pages. |
| 18 | |
| 19 | - History. |
| 20 | - Basic management. |
| 21 | - mlock()/mlockall() system call handling. |
| 22 | - Filtering special vmas. |
| 23 | - munlock()/munlockall() system call handling. |
| 24 | - Migrating mlocked pages. |
| 25 | - mmap(MAP_LOCKED) system call handling. |
| 26 | - munmap()/exit()/exec() system call handling. |
| 27 | - try_to_unmap(). |
| 28 | - try_to_munlock() reverse map scan. |
| 29 | - Page reclaim in shrink_*_list(). |
| 30 | |
| 31 | |
| 32 | ============ |
| 33 | INTRODUCTION |
| 34 | ============ |
| 35 | |
| 36 | This document describes the Linux memory manager's "Unevictable LRU" |
| 37 | infrastructure and the use of this to manage several types of "unevictable" |
| 38 | pages. |
| 39 | |
| 40 | The document attempts to provide the overall rationale behind this mechanism |
| 41 | and the rationale for some of the design decisions that drove the |
| 42 | implementation. The latter design rationale is discussed in the context of an |
| 43 | implementation description. Admittedly, one can obtain the implementation |
| 44 | details - the "what does it do?" - by reading the code. One hopes that the |
| 45 | descriptions below add value by provide the answer to "why does it do that?". |
| 46 | |
| 47 | |
| 48 | =================== |
| 49 | THE UNEVICTABLE LRU |
| 50 | =================== |
| 51 | |
| 52 | The Unevictable LRU facility adds an additional LRU list to track unevictable |
| 53 | pages and to hide these pages from vmscan. This mechanism is based on a patch |
| 54 | by Larry Woodman of Red Hat to address several scalability problems with page |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 55 | reclaim in Linux. The problems have been observed at customer sites on large |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 56 | memory x86_64 systems. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 57 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 58 | To illustrate this with an example, a non-NUMA x86_64 platform with 128GB of |
| 59 | main memory will have over 32 million 4k pages in a single zone. When a large |
| 60 | fraction of these pages are not evictable for any reason [see below], vmscan |
| 61 | will spend a lot of time scanning the LRU lists looking for the small fraction |
| 62 | of pages that are evictable. This can result in a situation where all CPUs are |
| 63 | spending 100% of their time in vmscan for hours or days on end, with the system |
| 64 | completely unresponsive. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 65 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 66 | The unevictable list addresses the following classes of unevictable pages: |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 67 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 68 | (*) Those owned by ramfs. |
| 69 | |
| 70 | (*) Those mapped into SHM_LOCK'd shared memory regions. |
| 71 | |
| 72 | (*) Those mapped into VM_LOCKED [mlock()ed] VMAs. |
| 73 | |
| 74 | The infrastructure may also be able to handle other conditions that make pages |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 75 | unevictable, either by definition or by circumstance, in the future. |
| 76 | |
| 77 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 78 | THE UNEVICTABLE PAGE LIST |
| 79 | ------------------------- |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 80 | |
| 81 | The Unevictable LRU infrastructure consists of an additional, per-zone, LRU list |
| 82 | called the "unevictable" list and an associated page flag, PG_unevictable, to |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 83 | indicate that the page is being managed on the unevictable list. |
| 84 | |
| 85 | The PG_unevictable flag is analogous to, and mutually exclusive with, the |
| 86 | PG_active flag in that it indicates on which LRU list a page resides when |
Michal Hocko | e6e8dd5 | 2011-03-16 15:01:37 +0100 | [diff] [blame] | 87 | PG_lru is set. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 88 | |
| 89 | The Unevictable LRU infrastructure maintains unevictable pages on an additional |
| 90 | LRU list for a few reasons: |
| 91 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 92 | (1) We get to "treat unevictable pages just like we treat other pages in the |
| 93 | system - which means we get to use the same code to manipulate them, the |
| 94 | same code to isolate them (for migrate, etc.), the same code to keep track |
| 95 | of the statistics, etc..." [Rik van Riel] |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 96 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 97 | (2) We want to be able to migrate unevictable pages between nodes for memory |
| 98 | defragmentation, workload management and memory hotplug. The linux kernel |
| 99 | can only migrate pages that it can successfully isolate from the LRU |
| 100 | lists. If we were to maintain pages elsewhere than on an LRU-like list, |
| 101 | where they can be found by isolate_lru_page(), we would prevent their |
| 102 | migration, unless we reworked migration code to find the unevictable pages |
| 103 | itself. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 104 | |
| 105 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 106 | The unevictable list does not differentiate between file-backed and anonymous, |
| 107 | swap-backed pages. This differentiation is only important while the pages are, |
| 108 | in fact, evictable. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 109 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 110 | The unevictable list benefits from the "arrayification" of the per-zone LRU |
| 111 | lists and statistics originally proposed and posted by Christoph Lameter. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 112 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 113 | The unevictable list does not use the LRU pagevec mechanism. Rather, |
| 114 | unevictable pages are placed directly on the page's zone's unevictable list |
| 115 | under the zone lru_lock. This allows us to prevent the stranding of pages on |
| 116 | the unevictable list when one task has the page isolated from the LRU and other |
| 117 | tasks are changing the "evictability" state of the page. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 118 | |
| 119 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 120 | MEMORY CONTROL GROUP INTERACTION |
| 121 | -------------------------------- |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 122 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 123 | The unevictable LRU facility interacts with the memory control group [aka |
| 124 | memory controller; see Documentation/cgroups/memory.txt] by extending the |
| 125 | lru_list enum. |
| 126 | |
| 127 | The memory controller data structure automatically gets a per-zone unevictable |
| 128 | list as a result of the "arrayification" of the per-zone LRU lists (one per |
| 129 | lru_list enum element). The memory controller tracks the movement of pages to |
| 130 | and from the unevictable list. |
| 131 | |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 132 | When a memory control group comes under memory pressure, the controller will |
| 133 | not attempt to reclaim pages on the unevictable list. This has a couple of |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 134 | effects: |
| 135 | |
| 136 | (1) Because the pages are "hidden" from reclaim on the unevictable list, the |
| 137 | reclaim process can be more efficient, dealing only with pages that have a |
| 138 | chance of being reclaimed. |
| 139 | |
| 140 | (2) On the other hand, if too many of the pages charged to the control group |
| 141 | are unevictable, the evictable portion of the working set of the tasks in |
| 142 | the control group may not fit into the available memory. This can cause |
| 143 | the control group to thrash or to OOM-kill tasks. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 144 | |
| 145 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 146 | MARKING ADDRESS SPACES UNEVICTABLE |
| 147 | ---------------------------------- |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 148 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 149 | For facilities such as ramfs none of the pages attached to the address space |
| 150 | may be evicted. To prevent eviction of any such pages, the AS_UNEVICTABLE |
| 151 | address space flag is provided, and this can be manipulated by a filesystem |
| 152 | using a number of wrapper functions: |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 153 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 154 | (*) void mapping_set_unevictable(struct address_space *mapping); |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 155 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 156 | Mark the address space as being completely unevictable. |
| 157 | |
| 158 | (*) void mapping_clear_unevictable(struct address_space *mapping); |
| 159 | |
| 160 | Mark the address space as being evictable. |
| 161 | |
| 162 | (*) int mapping_unevictable(struct address_space *mapping); |
| 163 | |
| 164 | Query the address space, and return true if it is completely |
| 165 | unevictable. |
| 166 | |
| 167 | These are currently used in two places in the kernel: |
| 168 | |
| 169 | (1) By ramfs to mark the address spaces of its inodes when they are created, |
| 170 | and this mark remains for the life of the inode. |
| 171 | |
| 172 | (2) By SYSV SHM to mark SHM_LOCK'd address spaces until SHM_UNLOCK is called. |
| 173 | |
| 174 | Note that SHM_LOCK is not required to page in the locked pages if they're |
| 175 | swapped out; the application must touch the pages manually if it wants to |
| 176 | ensure they're in memory. |
| 177 | |
| 178 | |
| 179 | DETECTING UNEVICTABLE PAGES |
| 180 | --------------------------- |
| 181 | |
| 182 | The function page_evictable() in vmscan.c determines whether a page is |
| 183 | evictable or not using the query function outlined above [see section "Marking |
| 184 | address spaces unevictable"] to check the AS_UNEVICTABLE flag. |
| 185 | |
| 186 | For address spaces that are so marked after being populated (as SHM regions |
| 187 | might be), the lock action (eg: SHM_LOCK) can be lazy, and need not populate |
| 188 | the page tables for the region as does, for example, mlock(), nor need it make |
| 189 | any special effort to push any pages in the SHM_LOCK'd area to the unevictable |
| 190 | list. Instead, vmscan will do this if and when it encounters the pages during |
| 191 | a reclamation scan. |
| 192 | |
| 193 | On an unlock action (such as SHM_UNLOCK), the unlocker (eg: shmctl()) must scan |
| 194 | the pages in the region and "rescue" them from the unevictable list if no other |
| 195 | condition is keeping them unevictable. If an unevictable region is destroyed, |
| 196 | the pages are also "rescued" from the unevictable list in the process of |
| 197 | freeing them. |
| 198 | |
| 199 | page_evictable() also checks for mlocked pages by testing an additional page |
Hugh Dickins | 39b5f29 | 2012-10-08 16:33:18 -0700 | [diff] [blame] | 200 | flag, PG_mlocked (as wrapped by PageMlocked()), which is set when a page is |
| 201 | faulted into a VM_LOCKED vma, or found in a vma being VM_LOCKED. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 202 | |
| 203 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 204 | VMSCAN'S HANDLING OF UNEVICTABLE PAGES |
| 205 | -------------------------------------- |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 206 | |
| 207 | If unevictable pages are culled in the fault path, or moved to the unevictable |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 208 | list at mlock() or mmap() time, vmscan will not encounter the pages until they |
| 209 | have become evictable again (via munlock() for example) and have been "rescued" |
| 210 | from the unevictable list. However, there may be situations where we decide, |
| 211 | for the sake of expediency, to leave a unevictable page on one of the regular |
| 212 | active/inactive LRU lists for vmscan to deal with. vmscan checks for such |
| 213 | pages in all of the shrink_{active|inactive|page}_list() functions and will |
| 214 | "cull" such pages that it encounters: that is, it diverts those pages to the |
| 215 | unevictable list for the zone being scanned. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 216 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 217 | There may be situations where a page is mapped into a VM_LOCKED VMA, but the |
| 218 | page is not marked as PG_mlocked. Such pages will make it all the way to |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 219 | shrink_page_list() where they will be detected when vmscan walks the reverse |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 220 | map in try_to_unmap(). If try_to_unmap() returns SWAP_MLOCK, |
| 221 | shrink_page_list() will cull the page at that point. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 222 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 223 | To "cull" an unevictable page, vmscan simply puts the page back on the LRU list |
| 224 | using putback_lru_page() - the inverse operation to isolate_lru_page() - after |
| 225 | dropping the page lock. Because the condition which makes the page unevictable |
| 226 | may change once the page is unlocked, putback_lru_page() will recheck the |
| 227 | unevictable state of a page that it places on the unevictable list. If the |
| 228 | page has become unevictable, putback_lru_page() removes it from the list and |
| 229 | retries, including the page_unevictable() test. Because such a race is a rare |
| 230 | event and movement of pages onto the unevictable list should be rare, these |
| 231 | extra evictabilty checks should not occur in the majority of calls to |
| 232 | putback_lru_page(). |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 233 | |
| 234 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 235 | ============= |
| 236 | MLOCKED PAGES |
| 237 | ============= |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 238 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 239 | The unevictable page list is also useful for mlock(), in addition to ramfs and |
| 240 | SYSV SHM. Note that mlock() is only available in CONFIG_MMU=y situations; in |
| 241 | NOMMU situations, all mappings are effectively mlocked. |
| 242 | |
| 243 | |
| 244 | HISTORY |
| 245 | ------- |
| 246 | |
| 247 | The "Unevictable mlocked Pages" infrastructure is based on work originally |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 248 | posted by Nick Piggin in an RFC patch entitled "mm: mlocked pages off LRU". |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 249 | Nick posted his patch as an alternative to a patch posted by Christoph Lameter |
| 250 | to achieve the same objective: hiding mlocked pages from vmscan. |
| 251 | |
| 252 | In Nick's patch, he used one of the struct page LRU list link fields as a count |
| 253 | of VM_LOCKED VMAs that map the page. This use of the link field for a count |
| 254 | prevented the management of the pages on an LRU list, and thus mlocked pages |
| 255 | were not migratable as isolate_lru_page() could not find them, and the LRU list |
| 256 | link field was not available to the migration subsystem. |
| 257 | |
| 258 | Nick resolved this by putting mlocked pages back on the lru list before |
| 259 | attempting to isolate them, thus abandoning the count of VM_LOCKED VMAs. When |
| 260 | Nick's patch was integrated with the Unevictable LRU work, the count was |
| 261 | replaced by walking the reverse map to determine whether any VM_LOCKED VMAs |
| 262 | mapped the page. More on this below. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 263 | |
| 264 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 265 | BASIC MANAGEMENT |
| 266 | ---------------- |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 267 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 268 | mlocked pages - pages mapped into a VM_LOCKED VMA - are a class of unevictable |
| 269 | pages. When such a page has been "noticed" by the memory management subsystem, |
| 270 | the page is marked with the PG_mlocked flag. This can be manipulated using the |
| 271 | PageMlocked() functions. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 272 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 273 | A PG_mlocked page will be placed on the unevictable list when it is added to |
| 274 | the LRU. Such pages can be "noticed" by memory management in several places: |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 275 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 276 | (1) in the mlock()/mlockall() system call handlers; |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 277 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 278 | (2) in the mmap() system call handler when mmapping a region with the |
| 279 | MAP_LOCKED flag; |
| 280 | |
| 281 | (3) mmapping a region in a task that has called mlockall() with the MCL_FUTURE |
| 282 | flag |
| 283 | |
| 284 | (4) in the fault path, if mlocked pages are "culled" in the fault path, |
| 285 | and when a VM_LOCKED stack segment is expanded; or |
| 286 | |
| 287 | (5) as mentioned above, in vmscan:shrink_page_list() when attempting to |
| 288 | reclaim a page in a VM_LOCKED VMA via try_to_unmap() |
| 289 | |
| 290 | all of which result in the VM_LOCKED flag being set for the VMA if it doesn't |
| 291 | already have it set. |
| 292 | |
| 293 | mlocked pages become unlocked and rescued from the unevictable list when: |
| 294 | |
| 295 | (1) mapped in a range unlocked via the munlock()/munlockall() system calls; |
| 296 | |
| 297 | (2) munmap()'d out of the last VM_LOCKED VMA that maps the page, including |
| 298 | unmapping at task exit; |
| 299 | |
| 300 | (3) when the page is truncated from the last VM_LOCKED VMA of an mmapped file; |
| 301 | or |
| 302 | |
| 303 | (4) before a page is COW'd in a VM_LOCKED VMA. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 304 | |
| 305 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 306 | mlock()/mlockall() SYSTEM CALL HANDLING |
| 307 | --------------------------------------- |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 308 | |
| 309 | Both [do_]mlock() and [do_]mlockall() system call handlers call mlock_fixup() |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 310 | for each VMA in the range specified by the call. In the case of mlockall(), |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 311 | this is the entire active address space of the task. Note that mlock_fixup() |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 312 | is used for both mlocking and munlocking a range of memory. A call to mlock() |
| 313 | an already VM_LOCKED VMA, or to munlock() a VMA that is not VM_LOCKED is |
| 314 | treated as a no-op, and mlock_fixup() simply returns. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 315 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 316 | If the VMA passes some filtering as described in "Filtering Special Vmas" |
| 317 | below, mlock_fixup() will attempt to merge the VMA with its neighbors or split |
| 318 | off a subset of the VMA if the range does not cover the entire VMA. Once the |
| 319 | VMA has been merged or split or neither, mlock_fixup() will call |
| 320 | __mlock_vma_pages_range() to fault in the pages via get_user_pages() and to |
| 321 | mark the pages as mlocked via mlock_vma_page(). |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 322 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 323 | Note that the VMA being mlocked might be mapped with PROT_NONE. In this case, |
| 324 | get_user_pages() will be unable to fault in the pages. That's okay. If pages |
| 325 | do end up getting faulted into this VM_LOCKED VMA, we'll handle them in the |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 326 | fault path or in vmscan. |
| 327 | |
| 328 | Also note that a page returned by get_user_pages() could be truncated or |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 329 | migrated out from under us, while we're trying to mlock it. To detect this, |
| 330 | __mlock_vma_pages_range() checks page_mapping() after acquiring the page lock. |
| 331 | If the page is still associated with its mapping, we'll go ahead and call |
| 332 | mlock_vma_page(). If the mapping is gone, we just unlock the page and move on. |
| 333 | In the worst case, this will result in a page mapped in a VM_LOCKED VMA |
| 334 | remaining on a normal LRU list without being PageMlocked(). Again, vmscan will |
| 335 | detect and cull such pages. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 336 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 337 | mlock_vma_page() will call TestSetPageMlocked() for each page returned by |
| 338 | get_user_pages(). We use TestSetPageMlocked() because the page might already |
| 339 | be mlocked by another task/VMA and we don't want to do extra work. We |
| 340 | especially do not want to count an mlocked page more than once in the |
| 341 | statistics. If the page was already mlocked, mlock_vma_page() need do nothing |
| 342 | more. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 343 | |
| 344 | If the page was NOT already mlocked, mlock_vma_page() attempts to isolate the |
| 345 | page from the LRU, as it is likely on the appropriate active or inactive list |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 346 | at that time. If the isolate_lru_page() succeeds, mlock_vma_page() will put |
| 347 | back the page - by calling putback_lru_page() - which will notice that the page |
| 348 | is now mlocked and divert the page to the zone's unevictable list. If |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 349 | mlock_vma_page() is unable to isolate the page from the LRU, vmscan will handle |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 350 | it later if and when it attempts to reclaim the page. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 351 | |
| 352 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 353 | FILTERING SPECIAL VMAS |
| 354 | ---------------------- |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 355 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 356 | mlock_fixup() filters several classes of "special" VMAs: |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 357 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 358 | 1) VMAs with VM_IO or VM_PFNMAP set are skipped entirely. The pages behind |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 359 | these mappings are inherently pinned, so we don't need to mark them as |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 360 | mlocked. In any case, most of the pages have no struct page in which to so |
| 361 | mark the page. Because of this, get_user_pages() will fail for these VMAs, |
| 362 | so there is no sense in attempting to visit them. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 363 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 364 | 2) VMAs mapping hugetlbfs page are already effectively pinned into memory. We |
| 365 | neither need nor want to mlock() these pages. However, to preserve the |
| 366 | prior behavior of mlock() - before the unevictable/mlock changes - |
| 367 | mlock_fixup() will call make_pages_present() in the hugetlbfs VMA range to |
| 368 | allocate the huge pages and populate the ptes. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 369 | |
Konstantin Khlebnikov | 314e51b | 2012-10-08 16:29:02 -0700 | [diff] [blame] | 370 | 3) VMAs with VM_DONTEXPAND are generally userspace mappings of kernel pages, |
| 371 | such as the VDSO page, relay channel pages, etc. These pages |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 372 | are inherently unevictable and are not managed on the LRU lists. |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 373 | mlock_fixup() treats these VMAs the same as hugetlbfs VMAs. It calls |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 374 | make_pages_present() to populate the ptes. |
| 375 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 376 | Note that for all of these special VMAs, mlock_fixup() does not set the |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 377 | VM_LOCKED flag. Therefore, we won't have to deal with them later during |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 378 | munlock(), munmap() or task exit. Neither does mlock_fixup() account these |
| 379 | VMAs against the task's "locked_vm". |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 380 | |
| 381 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 382 | munlock()/munlockall() SYSTEM CALL HANDLING |
| 383 | ------------------------------------------- |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 384 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 385 | The munlock() and munlockall() system calls are handled by the same functions - |
| 386 | do_mlock[all]() - as the mlock() and mlockall() system calls with the unlock vs |
| 387 | lock operation indicated by an argument. So, these system calls are also |
| 388 | handled by mlock_fixup(). Again, if called for an already munlocked VMA, |
| 389 | mlock_fixup() simply returns. Because of the VMA filtering discussed above, |
| 390 | VM_LOCKED will not be set in any "special" VMAs. So, these VMAs will be |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 391 | ignored for munlock. |
| 392 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 393 | If the VMA is VM_LOCKED, mlock_fixup() again attempts to merge or split off the |
| 394 | specified range. The range is then munlocked via the function |
| 395 | __mlock_vma_pages_range() - the same function used to mlock a VMA range - |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 396 | passing a flag to indicate that munlock() is being performed. |
| 397 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 398 | Because the VMA access protections could have been changed to PROT_NONE after |
Hugh Dickins | 63d6c5a | 2009-01-06 14:39:38 -0800 | [diff] [blame] | 399 | faulting in and mlocking pages, get_user_pages() was unreliable for visiting |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 400 | these pages for munlocking. Because we don't want to leave pages mlocked, |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 401 | get_user_pages() was enhanced to accept a flag to ignore the permissions when |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 402 | fetching the pages - all of which should be resident as a result of previous |
| 403 | mlocking. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 404 | |
| 405 | For munlock(), __mlock_vma_pages_range() unlocks individual pages by calling |
| 406 | munlock_vma_page(). munlock_vma_page() unconditionally clears the PG_mlocked |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 407 | flag using TestClearPageMlocked(). As with mlock_vma_page(), |
| 408 | munlock_vma_page() use the Test*PageMlocked() function to handle the case where |
| 409 | the page might have already been unlocked by another task. If the page was |
| 410 | mlocked, munlock_vma_page() updates that zone statistics for the number of |
| 411 | mlocked pages. Note, however, that at this point we haven't checked whether |
| 412 | the page is mapped by other VM_LOCKED VMAs. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 413 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 414 | We can't call try_to_munlock(), the function that walks the reverse map to |
| 415 | check for other VM_LOCKED VMAs, without first isolating the page from the LRU. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 416 | try_to_munlock() is a variant of try_to_unmap() and thus requires that the page |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 417 | not be on an LRU list [more on these below]. However, the call to |
| 418 | isolate_lru_page() could fail, in which case we couldn't try_to_munlock(). So, |
| 419 | we go ahead and clear PG_mlocked up front, as this might be the only chance we |
| 420 | have. If we can successfully isolate the page, we go ahead and |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 421 | try_to_munlock(), which will restore the PG_mlocked flag and update the zone |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 422 | page statistics if it finds another VMA holding the page mlocked. If we fail |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 423 | to isolate the page, we'll have left a potentially mlocked page on the LRU. |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 424 | This is fine, because we'll catch it later if and if vmscan tries to reclaim |
| 425 | the page. This should be relatively rare. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 426 | |
| 427 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 428 | MIGRATING MLOCKED PAGES |
| 429 | ----------------------- |
| 430 | |
| 431 | A page that is being migrated has been isolated from the LRU lists and is held |
| 432 | locked across unmapping of the page, updating the page's address space entry |
| 433 | and copying the contents and state, until the page table entry has been |
| 434 | replaced with an entry that refers to the new page. Linux supports migration |
| 435 | of mlocked pages and other unevictable pages. This involves simply moving the |
| 436 | PG_mlocked and PG_unevictable states from the old page to the new page. |
| 437 | |
| 438 | Note that page migration can race with mlocking or munlocking of the same page. |
| 439 | This has been discussed from the mlock/munlock perspective in the respective |
| 440 | sections above. Both processes (migration and m[un]locking) hold the page |
| 441 | locked. This provides the first level of synchronization. Page migration |
| 442 | zeros out the page_mapping of the old page before unlocking it, so m[un]lock |
| 443 | can skip these pages by testing the page mapping under page lock. |
| 444 | |
| 445 | To complete page migration, we place the new and old pages back onto the LRU |
| 446 | after dropping the page lock. The "unneeded" page - old page on success, new |
| 447 | page on failure - will be freed when the reference count held by the migration |
| 448 | process is released. To ensure that we don't strand pages on the unevictable |
| 449 | list because of a race between munlock and migration, page migration uses the |
| 450 | putback_lru_page() function to add migrated pages back to the LRU. |
| 451 | |
| 452 | |
| 453 | mmap(MAP_LOCKED) SYSTEM CALL HANDLING |
| 454 | ------------------------------------- |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 455 | |
| 456 | In addition the the mlock()/mlockall() system calls, an application can request |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 457 | that a region of memory be mlocked supplying the MAP_LOCKED flag to the mmap() |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 458 | call. Furthermore, any mmap() call or brk() call that expands the heap by a |
| 459 | task that has previously called mlockall() with the MCL_FUTURE flag will result |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 460 | in the newly mapped memory being mlocked. Before the unevictable/mlock |
| 461 | changes, the kernel simply called make_pages_present() to allocate pages and |
| 462 | populate the page table. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 463 | |
| 464 | To mlock a range of memory under the unevictable/mlock infrastructure, the |
| 465 | mmap() handler and task address space expansion functions call |
| 466 | mlock_vma_pages_range() specifying the vma and the address range to mlock. |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 467 | mlock_vma_pages_range() filters VMAs like mlock_fixup(), as described above in |
| 468 | "Filtering Special VMAs". It will clear the VM_LOCKED flag, which will have |
| 469 | already been set by the caller, in filtered VMAs. Thus these VMA's need not be |
| 470 | visited for munlock when the region is unmapped. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 471 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 472 | For "normal" VMAs, mlock_vma_pages_range() calls __mlock_vma_pages_range() to |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 473 | fault/allocate the pages and mlock them. Again, like mlock_fixup(), |
| 474 | mlock_vma_pages_range() downgrades the mmap semaphore to read mode before |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 475 | attempting to fault/allocate and mlock the pages and "upgrades" the semaphore |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 476 | back to write mode before returning. |
| 477 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 478 | The callers of mlock_vma_pages_range() will have already added the memory range |
| 479 | to be mlocked to the task's "locked_vm". To account for filtered VMAs, |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 480 | mlock_vma_pages_range() returns the number of pages NOT mlocked. All of the |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 481 | callers then subtract a non-negative return value from the task's locked_vm. A |
| 482 | negative return value represent an error - for example, from get_user_pages() |
| 483 | attempting to fault in a VMA with PROT_NONE access. In this case, we leave the |
| 484 | memory range accounted as locked_vm, as the protections could be changed later |
| 485 | and pages allocated into that region. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 486 | |
| 487 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 488 | munmap()/exit()/exec() SYSTEM CALL HANDLING |
| 489 | ------------------------------------------- |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 490 | |
| 491 | When unmapping an mlocked region of memory, whether by an explicit call to |
| 492 | munmap() or via an internal unmap from exit() or exec() processing, we must |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 493 | munlock the pages if we're removing the last VM_LOCKED VMA that maps the pages. |
Hugh Dickins | 63d6c5a | 2009-01-06 14:39:38 -0800 | [diff] [blame] | 494 | Before the unevictable/mlock changes, mlocking did not mark the pages in any |
| 495 | way, so unmapping them required no processing. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 496 | |
| 497 | To munlock a range of memory under the unevictable/mlock infrastructure, the |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 498 | munmap() handler and task address space call tear down function |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 499 | munlock_vma_pages_all(). The name reflects the observation that one always |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 500 | specifies the entire VMA range when munlock()ing during unmap of a region. |
| 501 | Because of the VMA filtering when mlocking() regions, only "normal" VMAs that |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 502 | actually contain mlocked pages will be passed to munlock_vma_pages_all(). |
| 503 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 504 | munlock_vma_pages_all() clears the VM_LOCKED VMA flag and, like mlock_fixup() |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 505 | for the munlock case, calls __munlock_vma_pages_range() to walk the page table |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 506 | for the VMA's memory range and munlock_vma_page() each resident page mapped by |
| 507 | the VMA. This effectively munlocks the page, only if this is the last |
| 508 | VM_LOCKED VMA that maps the page. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 509 | |
| 510 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 511 | try_to_unmap() |
| 512 | -------------- |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 513 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 514 | Pages can, of course, be mapped into multiple VMAs. Some of these VMAs may |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 515 | have VM_LOCKED flag set. It is possible for a page mapped into one or more |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 516 | VM_LOCKED VMAs not to have the PG_mlocked flag set and therefore reside on one |
| 517 | of the active or inactive LRU lists. This could happen if, for example, a task |
| 518 | in the process of munlocking the page could not isolate the page from the LRU. |
| 519 | As a result, vmscan/shrink_page_list() might encounter such a page as described |
| 520 | in section "vmscan's handling of unevictable pages". To handle this situation, |
| 521 | try_to_unmap() checks for VM_LOCKED VMAs while it is walking a page's reverse |
| 522 | map. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 523 | |
| 524 | try_to_unmap() is always called, by either vmscan for reclaim or for page |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 525 | migration, with the argument page locked and isolated from the LRU. Separate |
| 526 | functions handle anonymous and mapped file pages, as these types of pages have |
| 527 | different reverse map mechanisms. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 528 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 529 | (*) try_to_unmap_anon() |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 530 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 531 | To unmap anonymous pages, each VMA in the list anchored in the anon_vma |
| 532 | must be visited - at least until a VM_LOCKED VMA is encountered. If the |
| 533 | page is being unmapped for migration, VM_LOCKED VMAs do not stop the |
| 534 | process because mlocked pages are migratable. However, for reclaim, if |
| 535 | the page is mapped into a VM_LOCKED VMA, the scan stops. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 536 | |
Masanari Iida | 3cd0b62 | 2012-02-08 23:22:21 +0900 | [diff] [blame] | 537 | try_to_unmap_anon() attempts to acquire in read mode the mmap semaphore of |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 538 | the mm_struct to which the VMA belongs. If this is successful, it will |
| 539 | mlock the page via mlock_vma_page() - we wouldn't have gotten to |
| 540 | try_to_unmap_anon() if the page were already mlocked - and will return |
| 541 | SWAP_MLOCK, indicating that the page is unevictable. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 542 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 543 | If the mmap semaphore cannot be acquired, we are not sure whether the page |
| 544 | is really unevictable or not. In this case, try_to_unmap_anon() will |
| 545 | return SWAP_AGAIN. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 546 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 547 | (*) try_to_unmap_file() - linear mappings |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 548 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 549 | Unmapping of a mapped file page works the same as for anonymous mappings, |
| 550 | except that the scan visits all VMAs that map the page's index/page offset |
| 551 | in the page's mapping's reverse map priority search tree. It also visits |
| 552 | each VMA in the page's mapping's non-linear list, if the list is |
| 553 | non-empty. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 554 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 555 | As for anonymous pages, on encountering a VM_LOCKED VMA for a mapped file |
| 556 | page, try_to_unmap_file() will attempt to acquire the associated |
| 557 | mm_struct's mmap semaphore to mlock the page, returning SWAP_MLOCK if this |
| 558 | is successful, and SWAP_AGAIN, if not. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 559 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 560 | (*) try_to_unmap_file() - non-linear mappings |
| 561 | |
| 562 | If a page's mapping contains a non-empty non-linear mapping VMA list, then |
| 563 | try_to_un{map|lock}() must also visit each VMA in that list to determine |
| 564 | whether the page is mapped in a VM_LOCKED VMA. Again, the scan must visit |
| 565 | all VMAs in the non-linear list to ensure that the pages is not/should not |
| 566 | be mlocked. |
| 567 | |
| 568 | If a VM_LOCKED VMA is found in the list, the scan could terminate. |
| 569 | However, there is no easy way to determine whether the page is actually |
| 570 | mapped in a given VMA - either for unmapping or testing whether the |
| 571 | VM_LOCKED VMA actually pins the page. |
| 572 | |
| 573 | try_to_unmap_file() handles non-linear mappings by scanning a certain |
| 574 | number of pages - a "cluster" - in each non-linear VMA associated with the |
| 575 | page's mapping, for each file mapped page that vmscan tries to unmap. If |
| 576 | this happens to unmap the page we're trying to unmap, try_to_unmap() will |
| 577 | notice this on return (page_mapcount(page) will be 0) and return |
| 578 | SWAP_SUCCESS. Otherwise, it will return SWAP_AGAIN, causing vmscan to |
| 579 | recirculate this page. We take advantage of the cluster scan in |
| 580 | try_to_unmap_cluster() as follows: |
| 581 | |
| 582 | For each non-linear VMA, try_to_unmap_cluster() attempts to acquire the |
| 583 | mmap semaphore of the associated mm_struct for read without blocking. |
| 584 | |
| 585 | If this attempt is successful and the VMA is VM_LOCKED, |
| 586 | try_to_unmap_cluster() will retain the mmap semaphore for the scan; |
| 587 | otherwise it drops it here. |
| 588 | |
| 589 | Then, for each page in the cluster, if we're holding the mmap semaphore |
| 590 | for a locked VMA, try_to_unmap_cluster() calls mlock_vma_page() to |
| 591 | mlock the page. This call is a no-op if the page is already locked, |
| 592 | but will mlock any pages in the non-linear mapping that happen to be |
| 593 | unlocked. |
| 594 | |
| 595 | If one of the pages so mlocked is the page passed in to try_to_unmap(), |
| 596 | try_to_unmap_cluster() will return SWAP_MLOCK, rather than the default |
| 597 | SWAP_AGAIN. This will allow vmscan to cull the page, rather than |
| 598 | recirculating it on the inactive list. |
| 599 | |
| 600 | Again, if try_to_unmap_cluster() cannot acquire the VMA's mmap sem, it |
| 601 | returns SWAP_AGAIN, indicating that the page is mapped by a VM_LOCKED |
| 602 | VMA, but couldn't be mlocked. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 603 | |
| 604 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 605 | try_to_munlock() REVERSE MAP SCAN |
| 606 | --------------------------------- |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 607 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 608 | [!] TODO/FIXME: a better name might be page_mlocked() - analogous to the |
| 609 | page_referenced() reverse map walker. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 610 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 611 | When munlock_vma_page() [see section "munlock()/munlockall() System Call |
| 612 | Handling" above] tries to munlock a page, it needs to determine whether or not |
| 613 | the page is mapped by any VM_LOCKED VMA without actually attempting to unmap |
| 614 | all PTEs from the page. For this purpose, the unevictable/mlock infrastructure |
| 615 | introduced a variant of try_to_unmap() called try_to_munlock(). |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 616 | |
| 617 | try_to_munlock() calls the same functions as try_to_unmap() for anonymous and |
Masanari Iida | 40e4712 | 2012-03-04 23:16:11 +0900 | [diff] [blame] | 618 | mapped file pages with an additional argument specifying unlock versus unmap |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 619 | processing. Again, these functions walk the respective reverse maps looking |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 620 | for VM_LOCKED VMAs. When such a VMA is found for anonymous pages and file |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 621 | pages mapped in linear VMAs, as in the try_to_unmap() case, the functions |
Masanari Iida | 3cd0b62 | 2012-02-08 23:22:21 +0900 | [diff] [blame] | 622 | attempt to acquire the associated mmap semaphore, mlock the page via |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 623 | mlock_vma_page() and return SWAP_MLOCK. This effectively undoes the |
Hugh Dickins | 63d6c5a | 2009-01-06 14:39:38 -0800 | [diff] [blame] | 624 | pre-clearing of the page's PG_mlocked done by munlock_vma_page. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 625 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 626 | If try_to_unmap() is unable to acquire a VM_LOCKED VMA's associated mmap |
| 627 | semaphore, it will return SWAP_AGAIN. This will allow shrink_page_list() to |
| 628 | recycle the page on the inactive list and hope that it has better luck with the |
| 629 | page next time. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 630 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 631 | For file pages mapped into non-linear VMAs, the try_to_munlock() logic works |
| 632 | slightly differently. On encountering a VM_LOCKED non-linear VMA that might |
| 633 | map the page, try_to_munlock() returns SWAP_AGAIN without actually mlocking the |
| 634 | page. munlock_vma_page() will just leave the page unlocked and let vmscan deal |
| 635 | with it - the usual fallback position. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 636 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 637 | Note that try_to_munlock()'s reverse map walk must visit every VMA in a page's |
| 638 | reverse map to determine that a page is NOT mapped into any VM_LOCKED VMA. |
| 639 | However, the scan can terminate when it encounters a VM_LOCKED VMA and can |
Masanari Iida | 3cd0b62 | 2012-02-08 23:22:21 +0900 | [diff] [blame] | 640 | successfully acquire the VMA's mmap semaphore for read and mlock the page. |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 641 | Although try_to_munlock() might be called a great many times when munlocking a |
| 642 | large region or tearing down a large address space that has been mlocked via |
| 643 | mlockall(), overall this is a fairly rare event. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 644 | |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 645 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 646 | PAGE RECLAIM IN shrink_*_list() |
| 647 | ------------------------------- |
| 648 | |
| 649 | shrink_active_list() culls any obviously unevictable pages - i.e. |
Hugh Dickins | 39b5f29 | 2012-10-08 16:33:18 -0700 | [diff] [blame] | 650 | !page_evictable(page) - diverting these to the unevictable list. |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 651 | However, shrink_active_list() only sees unevictable pages that made it onto the |
| 652 | active/inactive lru lists. Note that these pages do not have PageUnevictable |
| 653 | set - otherwise they would be on the unevictable list and shrink_active_list |
| 654 | would never see them. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 655 | |
| 656 | Some examples of these unevictable pages on the LRU lists are: |
| 657 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 658 | (1) ramfs pages that have been placed on the LRU lists when first allocated. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 659 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 660 | (2) SHM_LOCK'd shared memory pages. shmctl(SHM_LOCK) does not attempt to |
| 661 | allocate or fault in the pages in the shared memory region. This happens |
| 662 | when an application accesses the page the first time after SHM_LOCK'ing |
| 663 | the segment. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 664 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 665 | (3) mlocked pages that could not be isolated from the LRU and moved to the |
| 666 | unevictable list in mlock_vma_page(). |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 667 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 668 | (4) Pages mapped into multiple VM_LOCKED VMAs, but try_to_munlock() couldn't |
| 669 | acquire the VMA's mmap semaphore to test the flags and set PageMlocked. |
| 670 | munlock_vma_page() was forced to let the page back on to the normal LRU |
| 671 | list for vmscan to handle. |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 672 | |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 673 | shrink_inactive_list() also diverts any unevictable pages that it finds on the |
| 674 | inactive lists to the appropriate zone's unevictable list. |
| 675 | |
| 676 | shrink_inactive_list() should only see SHM_LOCK'd pages that became SHM_LOCK'd |
| 677 | after shrink_active_list() had moved them to the inactive list, or pages mapped |
| 678 | into VM_LOCKED VMAs that munlock_vma_page() couldn't isolate from the LRU to |
| 679 | recheck via try_to_munlock(). shrink_inactive_list() won't notice the latter, |
| 680 | but will pass on to shrink_page_list(). |
Lee Schermerhorn | fa07e78 | 2008-10-18 20:26:47 -0700 | [diff] [blame] | 681 | |
| 682 | shrink_page_list() again culls obviously unevictable pages that it could |
Hugh Dickins | 63d6c5a | 2009-01-06 14:39:38 -0800 | [diff] [blame] | 683 | encounter for similar reason to shrink_inactive_list(). Pages mapped into |
David Howells | c24b720 | 2009-04-13 14:40:01 -0700 | [diff] [blame] | 684 | VM_LOCKED VMAs but without PG_mlocked set will make it all the way to |
Hugh Dickins | 63d6c5a | 2009-01-06 14:39:38 -0800 | [diff] [blame] | 685 | try_to_unmap(). shrink_page_list() will divert them to the unevictable list |
| 686 | when try_to_unmap() returns SWAP_MLOCK, as discussed above. |