Rafael J. Wysocki | 8314418 | 2007-07-17 04:03:35 -0700 | [diff] [blame] | 1 | Freezing of tasks |
| 2 | (C) 2007 Rafael J. Wysocki <rjw@sisk.pl>, GPL |
| 3 | |
| 4 | I. What is the freezing of tasks? |
| 5 | |
| 6 | The freezing of tasks is a mechanism by which user space processes and some |
| 7 | kernel threads are controlled during hibernation or system-wide suspend (on some |
| 8 | architectures). |
| 9 | |
| 10 | II. How does it work? |
| 11 | |
Marcos Paulo de Souza | 26e0f90 | 2012-04-29 22:29:30 +0200 | [diff] [blame] | 12 | There are three per-task flags used for that, PF_NOFREEZE, PF_FROZEN |
Rafael J. Wysocki | 8314418 | 2007-07-17 04:03:35 -0700 | [diff] [blame] | 13 | and PF_FREEZER_SKIP (the last one is auxiliary). The tasks that have |
| 14 | PF_NOFREEZE unset (all user space processes and some kernel threads) are |
| 15 | regarded as 'freezable' and treated in a special way before the system enters a |
| 16 | suspend state as well as before a hibernation image is created (in what follows |
| 17 | we only consider hibernation, but the description also applies to suspend). |
| 18 | |
| 19 | Namely, as the first step of the hibernation procedure the function |
Marcos Paulo de Souza | 26e0f90 | 2012-04-29 22:29:30 +0200 | [diff] [blame] | 20 | freeze_processes() (defined in kernel/power/process.c) is called. A system-wide |
| 21 | variable system_freezing_cnt (as opposed to a per-task flag) is used to indicate |
| 22 | whether the system is to undergo a freezing operation. And freeze_processes() |
| 23 | sets this variable. After this, it executes try_to_freeze_tasks() that sends a |
| 24 | fake signal to all user space processes, and wakes up all the kernel threads. |
| 25 | All freezable tasks must react to that by calling try_to_freeze(), which |
| 26 | results in a call to __refrigerator() (defined in kernel/freezer.c), which sets |
| 27 | the task's PF_FROZEN flag, changes its state to TASK_UNINTERRUPTIBLE and makes |
| 28 | it loop until PF_FROZEN is cleared for it. Then, we say that the task is |
| 29 | 'frozen' and therefore the set of functions handling this mechanism is referred |
| 30 | to as 'the freezer' (these functions are defined in kernel/power/process.c, |
| 31 | kernel/freezer.c & include/linux/freezer.h). User space processes are generally |
| 32 | frozen before kernel threads. |
Rafael J. Wysocki | 8314418 | 2007-07-17 04:03:35 -0700 | [diff] [blame] | 33 | |
Tejun Heo | a0acae0 | 2011-11-21 12:32:22 -0800 | [diff] [blame] | 34 | __refrigerator() must not be called directly. Instead, use the |
| 35 | try_to_freeze() function (defined in include/linux/freezer.h), that checks |
Marcos Paulo de Souza | 26e0f90 | 2012-04-29 22:29:30 +0200 | [diff] [blame] | 36 | if the task is to be frozen and makes the task enter __refrigerator(). |
Rafael J. Wysocki | 8314418 | 2007-07-17 04:03:35 -0700 | [diff] [blame] | 37 | |
| 38 | For user space processes try_to_freeze() is called automatically from the |
| 39 | signal-handling code, but the freezable kernel threads need to call it |
Rafael J. Wysocki | d5d8c59 | 2007-10-18 03:04:46 -0700 | [diff] [blame] | 40 | explicitly in suitable places or use the wait_event_freezable() or |
| 41 | wait_event_freezable_timeout() macros (defined in include/linux/freezer.h) |
Marcos Paulo de Souza | 26e0f90 | 2012-04-29 22:29:30 +0200 | [diff] [blame] | 42 | that combine interruptible sleep with checking if the task is to be frozen and |
| 43 | calling try_to_freeze(). The main loop of a freezable kernel thread may look |
| 44 | like the following one: |
Rafael J. Wysocki | 8314418 | 2007-07-17 04:03:35 -0700 | [diff] [blame] | 45 | |
Rafael J. Wysocki | d5d8c59 | 2007-10-18 03:04:46 -0700 | [diff] [blame] | 46 | set_freezable(); |
Rafael J. Wysocki | 8314418 | 2007-07-17 04:03:35 -0700 | [diff] [blame] | 47 | do { |
| 48 | hub_events(); |
Rafael J. Wysocki | d5d8c59 | 2007-10-18 03:04:46 -0700 | [diff] [blame] | 49 | wait_event_freezable(khubd_wait, |
| 50 | !list_empty(&hub_event_list) || |
| 51 | kthread_should_stop()); |
| 52 | } while (!kthread_should_stop() || !list_empty(&hub_event_list)); |
Rafael J. Wysocki | 8314418 | 2007-07-17 04:03:35 -0700 | [diff] [blame] | 53 | |
| 54 | (from drivers/usb/core/hub.c::hub_thread()). |
| 55 | |
| 56 | If a freezable kernel thread fails to call try_to_freeze() after the freezer has |
Marcos Paulo de Souza | 26e0f90 | 2012-04-29 22:29:30 +0200 | [diff] [blame] | 57 | initiated a freezing operation, the freezing of tasks will fail and the entire |
Rafael J. Wysocki | 8314418 | 2007-07-17 04:03:35 -0700 | [diff] [blame] | 58 | hibernation operation will be cancelled. For this reason, freezable kernel |
Rafael J. Wysocki | d5d8c59 | 2007-10-18 03:04:46 -0700 | [diff] [blame] | 59 | threads must call try_to_freeze() somewhere or use one of the |
| 60 | wait_event_freezable() and wait_event_freezable_timeout() macros. |
Rafael J. Wysocki | 8314418 | 2007-07-17 04:03:35 -0700 | [diff] [blame] | 61 | |
| 62 | After the system memory state has been restored from a hibernation image and |
| 63 | devices have been reinitialized, the function thaw_processes() is called in |
| 64 | order to clear the PF_FROZEN flag for each frozen task. Then, the tasks that |
Tejun Heo | a0acae0 | 2011-11-21 12:32:22 -0800 | [diff] [blame] | 65 | have been frozen leave __refrigerator() and continue running. |
Rafael J. Wysocki | 8314418 | 2007-07-17 04:03:35 -0700 | [diff] [blame] | 66 | |
Srivatsa S. Bhat | 9045a05 | 2012-02-04 22:26:26 +0100 | [diff] [blame] | 67 | |
| 68 | Rationale behind the functions dealing with freezing and thawing of tasks: |
| 69 | ------------------------------------------------------------------------- |
| 70 | |
| 71 | freeze_processes(): |
| 72 | - freezes only userspace tasks |
| 73 | |
| 74 | freeze_kernel_threads(): |
| 75 | - freezes all tasks (including kernel threads) because we can't freeze |
| 76 | kernel threads without freezing userspace tasks |
| 77 | |
| 78 | thaw_kernel_threads(): |
| 79 | - thaws only kernel threads; this is particularly useful if we need to do |
| 80 | anything special in between thawing of kernel threads and thawing of |
| 81 | userspace tasks, or if we want to postpone the thawing of userspace tasks |
| 82 | |
| 83 | thaw_processes(): |
| 84 | - thaws all tasks (including kernel threads) because we can't thaw userspace |
| 85 | tasks without thawing kernel threads |
| 86 | |
| 87 | |
Rafael J. Wysocki | 8314418 | 2007-07-17 04:03:35 -0700 | [diff] [blame] | 88 | III. Which kernel threads are freezable? |
| 89 | |
| 90 | Kernel threads are not freezable by default. However, a kernel thread may clear |
| 91 | PF_NOFREEZE for itself by calling set_freezable() (the resetting of PF_NOFREEZE |
Tejun Heo | 3a7cbd5 | 2011-11-21 12:32:22 -0800 | [diff] [blame] | 92 | directly is not allowed). From this point it is regarded as freezable |
Rafael J. Wysocki | 8314418 | 2007-07-17 04:03:35 -0700 | [diff] [blame] | 93 | and must call try_to_freeze() in a suitable place. |
| 94 | |
| 95 | IV. Why do we do that? |
| 96 | |
| 97 | Generally speaking, there is a couple of reasons to use the freezing of tasks: |
| 98 | |
| 99 | 1. The principal reason is to prevent filesystems from being damaged after |
| 100 | hibernation. At the moment we have no simple means of checkpointing |
| 101 | filesystems, so if there are any modifications made to filesystem data and/or |
| 102 | metadata on disks, we cannot bring them back to the state from before the |
| 103 | modifications. At the same time each hibernation image contains some |
| 104 | filesystem-related information that must be consistent with the state of the |
| 105 | on-disk data and metadata after the system memory state has been restored from |
| 106 | the image (otherwise the filesystems will be damaged in a nasty way, usually |
| 107 | making them almost impossible to repair). We therefore freeze tasks that might |
| 108 | cause the on-disk filesystems' data and metadata to be modified after the |
| 109 | hibernation image has been created and before the system is finally powered off. |
| 110 | The majority of these are user space processes, but if any of the kernel threads |
| 111 | may cause something like this to happen, they have to be freezable. |
| 112 | |
Rafael J. Wysocki | 2776365 | 2007-10-18 03:04:43 -0700 | [diff] [blame] | 113 | 2. Next, to create the hibernation image we need to free a sufficient amount of |
| 114 | memory (approximately 50% of available RAM) and we need to do that before |
| 115 | devices are deactivated, because we generally need them for swapping out. Then, |
| 116 | after the memory for the image has been freed, we don't want tasks to allocate |
| 117 | additional memory and we prevent them from doing that by freezing them earlier. |
| 118 | [Of course, this also means that device drivers should not allocate substantial |
| 119 | amounts of memory from their .suspend() callbacks before hibernation, but this |
Srivatsa S. Bhat | e9db50b | 2011-11-03 00:59:52 +0100 | [diff] [blame] | 120 | is a separate issue.] |
Rafael J. Wysocki | 2776365 | 2007-10-18 03:04:43 -0700 | [diff] [blame] | 121 | |
| 122 | 3. The third reason is to prevent user space processes and some kernel threads |
Rafael J. Wysocki | 8314418 | 2007-07-17 04:03:35 -0700 | [diff] [blame] | 123 | from interfering with the suspending and resuming of devices. A user space |
| 124 | process running on a second CPU while we are suspending devices may, for |
| 125 | example, be troublesome and without the freezing of tasks we would need some |
| 126 | safeguards against race conditions that might occur in such a case. |
| 127 | |
| 128 | Although Linus Torvalds doesn't like the freezing of tasks, he said this in one |
| 129 | of the discussions on LKML (http://lkml.org/lkml/2007/4/27/608): |
| 130 | |
| 131 | "RJW:> Why we freeze tasks at all or why we freeze kernel threads? |
| 132 | |
| 133 | Linus: In many ways, 'at all'. |
| 134 | |
| 135 | I _do_ realize the IO request queue issues, and that we cannot actually do |
| 136 | s2ram with some devices in the middle of a DMA. So we want to be able to |
| 137 | avoid *that*, there's no question about that. And I suspect that stopping |
| 138 | user threads and then waiting for a sync is practically one of the easier |
| 139 | ways to do so. |
| 140 | |
| 141 | So in practice, the 'at all' may become a 'why freeze kernel threads?' and |
| 142 | freezing user threads I don't find really objectionable." |
| 143 | |
| 144 | Still, there are kernel threads that may want to be freezable. For example, if |
Viresh Kumar | 5eb6f9a | 2012-01-19 23:22:49 +0100 | [diff] [blame] | 145 | a kernel thread that belongs to a device driver accesses the device directly, it |
| 146 | in principle needs to know when the device is suspended, so that it doesn't try |
| 147 | to access it at that time. However, if the kernel thread is freezable, it will |
| 148 | be frozen before the driver's .suspend() callback is executed and it will be |
Rafael J. Wysocki | 8314418 | 2007-07-17 04:03:35 -0700 | [diff] [blame] | 149 | thawed after the driver's .resume() callback has run, so it won't be accessing |
| 150 | the device while it's suspended. |
| 151 | |
Rafael J. Wysocki | 2776365 | 2007-10-18 03:04:43 -0700 | [diff] [blame] | 152 | 4. Another reason for freezing tasks is to prevent user space processes from |
Rafael J. Wysocki | 8314418 | 2007-07-17 04:03:35 -0700 | [diff] [blame] | 153 | realizing that hibernation (or suspend) operation takes place. Ideally, user |
| 154 | space processes should not notice that such a system-wide operation has occurred |
| 155 | and should continue running without any problems after the restore (or resume |
| 156 | from suspend). Unfortunately, in the most general case this is quite difficult |
| 157 | to achieve without the freezing of tasks. Consider, for example, a process |
| 158 | that depends on all CPUs being online while it's running. Since we need to |
| 159 | disable nonboot CPUs during the hibernation, if this process is not frozen, it |
| 160 | may notice that the number of CPUs has changed and may start to work incorrectly |
| 161 | because of that. |
| 162 | |
| 163 | V. Are there any problems related to the freezing of tasks? |
| 164 | |
| 165 | Yes, there are. |
| 166 | |
| 167 | First of all, the freezing of kernel threads may be tricky if they depend one |
| 168 | on another. For example, if kernel thread A waits for a completion (in the |
| 169 | TASK_UNINTERRUPTIBLE state) that needs to be done by freezable kernel thread B |
| 170 | and B is frozen in the meantime, then A will be blocked until B is thawed, which |
| 171 | may be undesirable. That's why kernel threads are not freezable by default. |
| 172 | |
| 173 | Second, there are the following two problems related to the freezing of user |
| 174 | space processes: |
| 175 | 1. Putting processes into an uninterruptible sleep distorts the load average. |
| 176 | 2. Now that we have FUSE, plus the framework for doing device drivers in |
| 177 | userspace, it gets even more complicated because some userspace processes are |
| 178 | now doing the sorts of things that kernel threads do |
| 179 | (https://lists.linux-foundation.org/pipermail/linux-pm/2007-May/012309.html). |
| 180 | |
| 181 | The problem 1. seems to be fixable, although it hasn't been fixed so far. The |
| 182 | other one is more serious, but it seems that we can work around it by using |
| 183 | hibernation (and suspend) notifiers (in that case, though, we won't be able to |
| 184 | avoid the realization by the user space processes that the hibernation is taking |
| 185 | place). |
| 186 | |
| 187 | There are also problems that the freezing of tasks tends to expose, although |
| 188 | they are not directly related to it. For example, if request_firmware() is |
| 189 | called from a device driver's .resume() routine, it will timeout and eventually |
| 190 | fail, because the user land process that should respond to the request is frozen |
| 191 | at this point. So, seemingly, the failure is due to the freezing of tasks. |
| 192 | Suppose, however, that the firmware file is located on a filesystem accessible |
| 193 | only through another device that hasn't been resumed yet. In that case, |
| 194 | request_firmware() will fail regardless of whether or not the freezing of tasks |
| 195 | is used. Consequently, the problem is not really related to the freezing of |
Oliver Neukum | fccdb5a | 2007-07-21 04:37:43 -0700 | [diff] [blame] | 196 | tasks, since it generally exists anyway. |
| 197 | |
| 198 | A driver must have all firmwares it may need in RAM before suspend() is called. |
| 199 | If keeping them is not practical, for example due to their size, they must be |
Rafael J. Wysocki | 730c4c0 | 2017-02-02 01:38:54 +0100 | [diff] [blame] | 200 | requested early enough using the suspend notifier API described in |
| 201 | Documentation/driver-api/pm/notifiers.rst. |
Srivatsa S. Bhat | cba3176 | 2011-12-07 22:30:09 +0100 | [diff] [blame] | 202 | |
| 203 | VI. Are there any precautions to be taken to prevent freezing failures? |
| 204 | |
| 205 | Yes, there are. |
| 206 | |
| 207 | First of all, grabbing the 'pm_mutex' lock to mutually exclude a piece of code |
| 208 | from system-wide sleep such as suspend/hibernation is not encouraged. |
| 209 | If possible, that piece of code must instead hook onto the suspend/hibernation |
| 210 | notifiers to achieve mutual exclusion. Look at the CPU-Hotplug code |
| 211 | (kernel/cpu.c) for an example. |
| 212 | |
| 213 | However, if that is not feasible, and grabbing 'pm_mutex' is deemed necessary, |
| 214 | it is strongly discouraged to directly call mutex_[un]lock(&pm_mutex) since |
| 215 | that could lead to freezing failures, because if the suspend/hibernate code |
| 216 | successfully acquired the 'pm_mutex' lock, and hence that other entity failed |
| 217 | to acquire the lock, then that task would get blocked in TASK_UNINTERRUPTIBLE |
| 218 | state. As a consequence, the freezer would not be able to freeze that task, |
| 219 | leading to freezing failure. |
| 220 | |
| 221 | However, the [un]lock_system_sleep() APIs are safe to use in this scenario, |
| 222 | since they ask the freezer to skip freezing this task, since it is anyway |
| 223 | "frozen enough" as it is blocked on 'pm_mutex', which will be released |
| 224 | only after the entire suspend/hibernation sequence is complete. |
| 225 | So, to summarize, use [un]lock_system_sleep() instead of directly using |
| 226 | mutex_[un]lock(&pm_mutex). That would prevent freezing failures. |
Li Fei | 957d128 | 2013-02-01 08:56:03 +0000 | [diff] [blame] | 227 | |
| 228 | V. Miscellaneous |
| 229 | /sys/power/pm_freeze_timeout controls how long it will cost at most to freeze |
| 230 | all user space processes or all freezable kernel threads, in unit of millisecond. |
| 231 | The default value is 20000, with range of unsigned integer. |