blob: ec715cd78fbb7028bde53c0664a513701b162858 [file] [log] [blame]
Rafael J. Wysocki83144182007-07-17 04:03:35 -07001Freezing of tasks
2 (C) 2007 Rafael J. Wysocki <rjw@sisk.pl>, GPL
3
4I. What is the freezing of tasks?
5
6The freezing of tasks is a mechanism by which user space processes and some
7kernel threads are controlled during hibernation or system-wide suspend (on some
8architectures).
9
10II. How does it work?
11
12There are four per-task flags used for that, PF_NOFREEZE, PF_FROZEN, TIF_FREEZE
13and PF_FREEZER_SKIP (the last one is auxiliary). The tasks that have
14PF_NOFREEZE unset (all user space processes and some kernel threads) are
15regarded as 'freezable' and treated in a special way before the system enters a
16suspend state as well as before a hibernation image is created (in what follows
17we only consider hibernation, but the description also applies to suspend).
18
19Namely, as the first step of the hibernation procedure the function
20freeze_processes() (defined in kernel/power/process.c) is called. It executes
21try_to_freeze_tasks() that sets TIF_FREEZE for all of the freezable tasks and
Rafael J. Wysockid5d8c592007-10-18 03:04:46 -070022either wakes them up, if they are kernel threads, or sends fake signals to them,
23if they are user space processes. A task that has TIF_FREEZE set, should react
Tejun Heoa0acae02011-11-21 12:32:22 -080024to it by calling the function called __refrigerator() (defined in
Srivatsa S. Bhate9db50b2011-11-03 00:59:52 +010025kernel/freezer.c), which sets the task's PF_FROZEN flag, changes its state
Rafael J. Wysockid5d8c592007-10-18 03:04:46 -070026to TASK_UNINTERRUPTIBLE and makes it loop until PF_FROZEN is cleared for it.
27Then, we say that the task is 'frozen' and therefore the set of functions
28handling this mechanism is referred to as 'the freezer' (these functions are
Srivatsa S. Bhate9db50b2011-11-03 00:59:52 +010029defined in kernel/power/process.c, kernel/freezer.c & include/linux/freezer.h).
30User space processes are generally frozen before kernel threads.
Rafael J. Wysocki83144182007-07-17 04:03:35 -070031
Tejun Heoa0acae02011-11-21 12:32:22 -080032__refrigerator() must not be called directly. Instead, use the
33try_to_freeze() function (defined in include/linux/freezer.h), that checks
34the task's TIF_FREEZE flag and makes the task enter __refrigerator() if the
35flag is set.
Rafael J. Wysocki83144182007-07-17 04:03:35 -070036
37For user space processes try_to_freeze() is called automatically from the
38signal-handling code, but the freezable kernel threads need to call it
Rafael J. Wysockid5d8c592007-10-18 03:04:46 -070039explicitly in suitable places or use the wait_event_freezable() or
40wait_event_freezable_timeout() macros (defined in include/linux/freezer.h)
41that combine interruptible sleep with checking if TIF_FREEZE is set and calling
42try_to_freeze(). The main loop of a freezable kernel thread may look like the
43following one:
Rafael J. Wysocki83144182007-07-17 04:03:35 -070044
Rafael J. Wysockid5d8c592007-10-18 03:04:46 -070045 set_freezable();
Rafael J. Wysocki83144182007-07-17 04:03:35 -070046 do {
47 hub_events();
Rafael J. Wysockid5d8c592007-10-18 03:04:46 -070048 wait_event_freezable(khubd_wait,
49 !list_empty(&hub_event_list) ||
50 kthread_should_stop());
51 } while (!kthread_should_stop() || !list_empty(&hub_event_list));
Rafael J. Wysocki83144182007-07-17 04:03:35 -070052
53(from drivers/usb/core/hub.c::hub_thread()).
54
55If a freezable kernel thread fails to call try_to_freeze() after the freezer has
56set TIF_FREEZE for it, the freezing of tasks will fail and the entire
57hibernation operation will be cancelled. For this reason, freezable kernel
Rafael J. Wysockid5d8c592007-10-18 03:04:46 -070058threads must call try_to_freeze() somewhere or use one of the
59wait_event_freezable() and wait_event_freezable_timeout() macros.
Rafael J. Wysocki83144182007-07-17 04:03:35 -070060
61After the system memory state has been restored from a hibernation image and
62devices have been reinitialized, the function thaw_processes() is called in
63order to clear the PF_FROZEN flag for each frozen task. Then, the tasks that
Tejun Heoa0acae02011-11-21 12:32:22 -080064have been frozen leave __refrigerator() and continue running.
Rafael J. Wysocki83144182007-07-17 04:03:35 -070065
Srivatsa S. Bhat9045a052012-02-04 22:26:26 +010066
67Rationale behind the functions dealing with freezing and thawing of tasks:
68-------------------------------------------------------------------------
69
70freeze_processes():
71 - freezes only userspace tasks
72
73freeze_kernel_threads():
74 - freezes all tasks (including kernel threads) because we can't freeze
75 kernel threads without freezing userspace tasks
76
77thaw_kernel_threads():
78 - thaws only kernel threads; this is particularly useful if we need to do
79 anything special in between thawing of kernel threads and thawing of
80 userspace tasks, or if we want to postpone the thawing of userspace tasks
81
82thaw_processes():
83 - thaws all tasks (including kernel threads) because we can't thaw userspace
84 tasks without thawing kernel threads
85
86
Rafael J. Wysocki83144182007-07-17 04:03:35 -070087III. Which kernel threads are freezable?
88
89Kernel threads are not freezable by default. However, a kernel thread may clear
90PF_NOFREEZE for itself by calling set_freezable() (the resetting of PF_NOFREEZE
Tejun Heo3a7cbd52011-11-21 12:32:22 -080091directly is not allowed). From this point it is regarded as freezable
Rafael J. Wysocki83144182007-07-17 04:03:35 -070092and must call try_to_freeze() in a suitable place.
93
94IV. Why do we do that?
95
96Generally speaking, there is a couple of reasons to use the freezing of tasks:
97
981. The principal reason is to prevent filesystems from being damaged after
99hibernation. At the moment we have no simple means of checkpointing
100filesystems, so if there are any modifications made to filesystem data and/or
101metadata on disks, we cannot bring them back to the state from before the
102modifications. At the same time each hibernation image contains some
103filesystem-related information that must be consistent with the state of the
104on-disk data and metadata after the system memory state has been restored from
105the image (otherwise the filesystems will be damaged in a nasty way, usually
106making them almost impossible to repair). We therefore freeze tasks that might
107cause the on-disk filesystems' data and metadata to be modified after the
108hibernation image has been created and before the system is finally powered off.
109The majority of these are user space processes, but if any of the kernel threads
110may cause something like this to happen, they have to be freezable.
111
Rafael J. Wysocki27763652007-10-18 03:04:43 -07001122. Next, to create the hibernation image we need to free a sufficient amount of
113memory (approximately 50% of available RAM) and we need to do that before
114devices are deactivated, because we generally need them for swapping out. Then,
115after the memory for the image has been freed, we don't want tasks to allocate
116additional memory and we prevent them from doing that by freezing them earlier.
117[Of course, this also means that device drivers should not allocate substantial
118amounts of memory from their .suspend() callbacks before hibernation, but this
Srivatsa S. Bhate9db50b2011-11-03 00:59:52 +0100119is a separate issue.]
Rafael J. Wysocki27763652007-10-18 03:04:43 -0700120
1213. The third reason is to prevent user space processes and some kernel threads
Rafael J. Wysocki83144182007-07-17 04:03:35 -0700122from interfering with the suspending and resuming of devices. A user space
123process running on a second CPU while we are suspending devices may, for
124example, be troublesome and without the freezing of tasks we would need some
125safeguards against race conditions that might occur in such a case.
126
127Although Linus Torvalds doesn't like the freezing of tasks, he said this in one
128of the discussions on LKML (http://lkml.org/lkml/2007/4/27/608):
129
130"RJW:> Why we freeze tasks at all or why we freeze kernel threads?
131
132Linus: In many ways, 'at all'.
133
134I _do_ realize the IO request queue issues, and that we cannot actually do
135s2ram with some devices in the middle of a DMA. So we want to be able to
136avoid *that*, there's no question about that. And I suspect that stopping
137user threads and then waiting for a sync is practically one of the easier
138ways to do so.
139
140So in practice, the 'at all' may become a 'why freeze kernel threads?' and
141freezing user threads I don't find really objectionable."
142
143Still, there are kernel threads that may want to be freezable. For example, if
Viresh Kumar5eb6f9a2012-01-19 23:22:49 +0100144a kernel thread that belongs to a device driver accesses the device directly, it
145in principle needs to know when the device is suspended, so that it doesn't try
146to access it at that time. However, if the kernel thread is freezable, it will
147be frozen before the driver's .suspend() callback is executed and it will be
Rafael J. Wysocki83144182007-07-17 04:03:35 -0700148thawed after the driver's .resume() callback has run, so it won't be accessing
149the device while it's suspended.
150
Rafael J. Wysocki27763652007-10-18 03:04:43 -07001514. Another reason for freezing tasks is to prevent user space processes from
Rafael J. Wysocki83144182007-07-17 04:03:35 -0700152realizing that hibernation (or suspend) operation takes place. Ideally, user
153space processes should not notice that such a system-wide operation has occurred
154and should continue running without any problems after the restore (or resume
155from suspend). Unfortunately, in the most general case this is quite difficult
156to achieve without the freezing of tasks. Consider, for example, a process
157that depends on all CPUs being online while it's running. Since we need to
158disable nonboot CPUs during the hibernation, if this process is not frozen, it
159may notice that the number of CPUs has changed and may start to work incorrectly
160because of that.
161
162V. Are there any problems related to the freezing of tasks?
163
164Yes, there are.
165
166First of all, the freezing of kernel threads may be tricky if they depend one
167on another. For example, if kernel thread A waits for a completion (in the
168TASK_UNINTERRUPTIBLE state) that needs to be done by freezable kernel thread B
169and B is frozen in the meantime, then A will be blocked until B is thawed, which
170may be undesirable. That's why kernel threads are not freezable by default.
171
172Second, there are the following two problems related to the freezing of user
173space processes:
1741. Putting processes into an uninterruptible sleep distorts the load average.
1752. Now that we have FUSE, plus the framework for doing device drivers in
176userspace, it gets even more complicated because some userspace processes are
177now doing the sorts of things that kernel threads do
178(https://lists.linux-foundation.org/pipermail/linux-pm/2007-May/012309.html).
179
180The problem 1. seems to be fixable, although it hasn't been fixed so far. The
181other one is more serious, but it seems that we can work around it by using
182hibernation (and suspend) notifiers (in that case, though, we won't be able to
183avoid the realization by the user space processes that the hibernation is taking
184place).
185
186There are also problems that the freezing of tasks tends to expose, although
187they are not directly related to it. For example, if request_firmware() is
188called from a device driver's .resume() routine, it will timeout and eventually
189fail, because the user land process that should respond to the request is frozen
190at this point. So, seemingly, the failure is due to the freezing of tasks.
191Suppose, however, that the firmware file is located on a filesystem accessible
192only through another device that hasn't been resumed yet. In that case,
193request_firmware() will fail regardless of whether or not the freezing of tasks
194is used. Consequently, the problem is not really related to the freezing of
Oliver Neukumfccdb5a2007-07-21 04:37:43 -0700195tasks, since it generally exists anyway.
196
197A driver must have all firmwares it may need in RAM before suspend() is called.
198If keeping them is not practical, for example due to their size, they must be
199requested early enough using the suspend notifier API described in notifiers.txt.
Srivatsa S. Bhatcba31762011-12-07 22:30:09 +0100200
201VI. Are there any precautions to be taken to prevent freezing failures?
202
203Yes, there are.
204
205First of all, grabbing the 'pm_mutex' lock to mutually exclude a piece of code
206from system-wide sleep such as suspend/hibernation is not encouraged.
207If possible, that piece of code must instead hook onto the suspend/hibernation
208notifiers to achieve mutual exclusion. Look at the CPU-Hotplug code
209(kernel/cpu.c) for an example.
210
211However, if that is not feasible, and grabbing 'pm_mutex' is deemed necessary,
212it is strongly discouraged to directly call mutex_[un]lock(&pm_mutex) since
213that could lead to freezing failures, because if the suspend/hibernate code
214successfully acquired the 'pm_mutex' lock, and hence that other entity failed
215to acquire the lock, then that task would get blocked in TASK_UNINTERRUPTIBLE
216state. As a consequence, the freezer would not be able to freeze that task,
217leading to freezing failure.
218
219However, the [un]lock_system_sleep() APIs are safe to use in this scenario,
220since they ask the freezer to skip freezing this task, since it is anyway
221"frozen enough" as it is blocked on 'pm_mutex', which will be released
222only after the entire suspend/hibernation sequence is complete.
223So, to summarize, use [un]lock_system_sleep() instead of directly using
224mutex_[un]lock(&pm_mutex). That would prevent freezing failures.