blob: 6ef692667e2f67aac0dbf87b232c16153180739e [file] [log] [blame]
Paul E. McKenney32300752008-05-12 21:21:05 +02001Please note that the "What is RCU?" LWN series is an excellent place
2to start learning about RCU:
3
41. What is RCU, Fundamentally? http://lwn.net/Articles/262464/
52. What is RCU? Part 2: Usage http://lwn.net/Articles/263130/
63. RCU part 3: the RCU API http://lwn.net/Articles/264090/
7
8
Paul E. McKenneydd81eca2005-09-10 00:26:24 -07009What is RCU?
10
11RCU is a synchronization mechanism that was added to the Linux kernel
12during the 2.5 development effort that is optimized for read-mostly
13situations. Although RCU is actually quite simple once you understand it,
14getting there can sometimes be a challenge. Part of the problem is that
15most of the past descriptions of RCU have been written with the mistaken
16assumption that there is "one true way" to describe RCU. Instead,
17the experience has been that different people must take different paths
18to arrive at an understanding of RCU. This document provides several
19different paths, as follows:
20
211. RCU OVERVIEW
222. WHAT IS RCU'S CORE API?
233. WHAT ARE SOME EXAMPLE USES OF CORE RCU API?
244. WHAT IF MY UPDATING THREAD CANNOT BLOCK?
255. WHAT ARE SOME SIMPLE IMPLEMENTATIONS OF RCU?
266. ANALOGY WITH READER-WRITER LOCKING
277. FULL LIST OF RCU APIs
288. ANSWERS TO QUICK QUIZZES
29
30People who prefer starting with a conceptual overview should focus on
31Section 1, though most readers will profit by reading this section at
32some point. People who prefer to start with an API that they can then
33experiment with should focus on Section 2. People who prefer to start
34with example uses should focus on Sections 3 and 4. People who need to
35understand the RCU implementation should focus on Section 5, then dive
36into the kernel source code. People who reason best by analogy should
37focus on Section 6. Section 7 serves as an index to the docbook API
38documentation, and Section 8 is the traditional answer key.
39
40So, start with the section that makes the most sense to you and your
41preferred method of learning. If you need to know everything about
42everything, feel free to read the whole thing -- but if you are really
43that type of person, you have perused the source code and will therefore
44never need this document anyway. ;-)
45
46
471. RCU OVERVIEW
48
49The basic idea behind RCU is to split updates into "removal" and
50"reclamation" phases. The removal phase removes references to data items
51within a data structure (possibly by replacing them with references to
52new versions of these data items), and can run concurrently with readers.
53The reason that it is safe to run the removal phase concurrently with
54readers is the semantics of modern CPUs guarantee that readers will see
55either the old or the new version of the data structure rather than a
56partially updated reference. The reclamation phase does the work of reclaiming
57(e.g., freeing) the data items removed from the data structure during the
58removal phase. Because reclaiming data items can disrupt any readers
59concurrently referencing those data items, the reclamation phase must
60not start until readers no longer hold references to those data items.
61
62Splitting the update into removal and reclamation phases permits the
63updater to perform the removal phase immediately, and to defer the
64reclamation phase until all readers active during the removal phase have
65completed, either by blocking until they finish or by registering a
66callback that is invoked after they finish. Only readers that are active
67during the removal phase need be considered, because any reader starting
68after the removal phase will be unable to gain a reference to the removed
69data items, and therefore cannot be disrupted by the reclamation phase.
70
71So the typical RCU update sequence goes something like the following:
72
73a. Remove pointers to a data structure, so that subsequent
74 readers cannot gain a reference to it.
75
76b. Wait for all previous readers to complete their RCU read-side
77 critical sections.
78
79c. At this point, there cannot be any readers who hold references
80 to the data structure, so it now may safely be reclaimed
81 (e.g., kfree()d).
82
83Step (b) above is the key idea underlying RCU's deferred destruction.
84The ability to wait until all readers are done allows RCU readers to
85use much lighter-weight synchronization, in some cases, absolutely no
86synchronization at all. In contrast, in more conventional lock-based
87schemes, readers must use heavy-weight synchronization in order to
88prevent an updater from deleting the data structure out from under them.
89This is because lock-based updaters typically update data items in place,
90and must therefore exclude readers. In contrast, RCU-based updaters
91typically take advantage of the fact that writes to single aligned
92pointers are atomic on modern CPUs, allowing atomic insertion, removal,
93and replacement of data items in a linked structure without disrupting
94readers. Concurrent RCU readers can then continue accessing the old
95versions, and can dispense with the atomic operations, memory barriers,
96and communications cache misses that are so expensive on present-day
97SMP computer systems, even in absence of lock contention.
98
99In the three-step procedure shown above, the updater is performing both
100the removal and the reclamation step, but it is often helpful for an
101entirely different thread to do the reclamation, as is in fact the case
102in the Linux kernel's directory-entry cache (dcache). Even if the same
103thread performs both the update step (step (a) above) and the reclamation
104step (step (c) above), it is often helpful to think of them separately.
105For example, RCU readers and updaters need not communicate at all,
106but RCU provides implicit low-overhead communication between readers
107and reclaimers, namely, in step (b) above.
108
109So how the heck can a reclaimer tell when a reader is done, given
110that readers are not doing any sort of synchronization operations???
111Read on to learn about how RCU's API makes this easy.
112
113
1142. WHAT IS RCU'S CORE API?
115
116The core RCU API is quite small:
117
118a. rcu_read_lock()
119b. rcu_read_unlock()
120c. synchronize_rcu() / call_rcu()
121d. rcu_assign_pointer()
122e. rcu_dereference()
123
124There are many other members of the RCU API, but the rest can be
125expressed in terms of these five, though most implementations instead
126express synchronize_rcu() in terms of the call_rcu() callback API.
127
128The five core RCU APIs are described below, the other 18 will be enumerated
129later. See the kernel docbook documentation for more info, or look directly
130at the function header comments.
131
132rcu_read_lock()
133
134 void rcu_read_lock(void);
135
136 Used by a reader to inform the reclaimer that the reader is
137 entering an RCU read-side critical section. It is illegal
138 to block while in an RCU read-side critical section, though
Paul E. McKenney6b3ef482009-08-22 13:56:53 -0700139 kernels built with CONFIG_TREE_PREEMPT_RCU can preempt RCU
140 read-side critical sections. Any RCU-protected data structure
141 accessed during an RCU read-side critical section is guaranteed to
142 remain unreclaimed for the full duration of that critical section.
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700143 Reference counts may be used in conjunction with RCU to maintain
144 longer-term references to data structures.
145
146rcu_read_unlock()
147
148 void rcu_read_unlock(void);
149
150 Used by a reader to inform the reclaimer that the reader is
151 exiting an RCU read-side critical section. Note that RCU
152 read-side critical sections may be nested and/or overlapping.
153
154synchronize_rcu()
155
156 void synchronize_rcu(void);
157
158 Marks the end of updater code and the beginning of reclaimer
159 code. It does this by blocking until all pre-existing RCU
160 read-side critical sections on all CPUs have completed.
161 Note that synchronize_rcu() will -not- necessarily wait for
162 any subsequent RCU read-side critical sections to complete.
163 For example, consider the following sequence of events:
164
165 CPU 0 CPU 1 CPU 2
166 ----------------- ------------------------- ---------------
167 1. rcu_read_lock()
168 2. enters synchronize_rcu()
169 3. rcu_read_lock()
170 4. rcu_read_unlock()
171 5. exits synchronize_rcu()
172 6. rcu_read_unlock()
173
174 To reiterate, synchronize_rcu() waits only for ongoing RCU
175 read-side critical sections to complete, not necessarily for
176 any that begin after synchronize_rcu() is invoked.
177
178 Of course, synchronize_rcu() does not necessarily return
179 -immediately- after the last pre-existing RCU read-side critical
180 section completes. For one thing, there might well be scheduling
181 delays. For another thing, many RCU implementations process
182 requests in batches in order to improve efficiencies, which can
183 further delay synchronize_rcu().
184
185 Since synchronize_rcu() is the API that must figure out when
186 readers are done, its implementation is key to RCU. For RCU
187 to be useful in all but the most read-intensive situations,
188 synchronize_rcu()'s overhead must also be quite small.
189
190 The call_rcu() API is a callback form of synchronize_rcu(),
191 and is described in more detail in a later section. Instead of
192 blocking, it registers a function and argument which are invoked
193 after all ongoing RCU read-side critical sections have completed.
194 This callback variant is particularly useful in situations where
Paul E. McKenney165d6c72006-06-25 05:48:44 -0700195 it is illegal to block or where update-side performance is
196 critically important.
197
198 However, the call_rcu() API should not be used lightly, as use
199 of the synchronize_rcu() API generally results in simpler code.
200 In addition, the synchronize_rcu() API has the nice property
201 of automatically limiting update rate should grace periods
202 be delayed. This property results in system resilience in face
203 of denial-of-service attacks. Code using call_rcu() should limit
204 update rate in order to gain this same sort of resilience. See
205 checklist.txt for some approaches to limiting the update rate.
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700206
207rcu_assign_pointer()
208
209 typeof(p) rcu_assign_pointer(p, typeof(p) v);
210
211 Yes, rcu_assign_pointer() -is- implemented as a macro, though it
212 would be cool to be able to declare a function in this manner.
213 (Compiler experts will no doubt disagree.)
214
215 The updater uses this function to assign a new value to an
216 RCU-protected pointer, in order to safely communicate the change
217 in value from the updater to the reader. This function returns
218 the new value, and also executes any memory-barrier instructions
219 required for a given CPU architecture.
220
Paul E. McKenneyd19720a2006-02-01 03:06:42 -0800221 Perhaps just as important, it serves to document (1) which
222 pointers are protected by RCU and (2) the point at which a
223 given structure becomes accessible to other CPUs. That said,
224 rcu_assign_pointer() is most frequently used indirectly, via
225 the _rcu list-manipulation primitives such as list_add_rcu().
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700226
227rcu_dereference()
228
229 typeof(p) rcu_dereference(p);
230
231 Like rcu_assign_pointer(), rcu_dereference() must be implemented
232 as a macro.
233
234 The reader uses rcu_dereference() to fetch an RCU-protected
235 pointer, which returns a value that may then be safely
236 dereferenced. Note that rcu_deference() does not actually
237 dereference the pointer, instead, it protects the pointer for
238 later dereferencing. It also executes any needed memory-barrier
239 instructions for a given CPU architecture. Currently, only Alpha
240 needs memory barriers within rcu_dereference() -- on other CPUs,
241 it compiles to nothing, not even a compiler directive.
242
243 Common coding practice uses rcu_dereference() to copy an
244 RCU-protected pointer to a local variable, then dereferences
245 this local variable, for example as follows:
246
247 p = rcu_dereference(head.next);
248 return p->data;
249
250 However, in this case, one could just as easily combine these
251 into one statement:
252
253 return rcu_dereference(head.next)->data;
254
255 If you are going to be fetching multiple fields from the
256 RCU-protected structure, using the local variable is of
257 course preferred. Repeated rcu_dereference() calls look
258 ugly and incur unnecessary overhead on Alpha CPUs.
259
260 Note that the value returned by rcu_dereference() is valid
261 only within the enclosing RCU read-side critical section.
262 For example, the following is -not- legal:
263
264 rcu_read_lock();
265 p = rcu_dereference(head.next);
266 rcu_read_unlock();
267 x = p->address;
268 rcu_read_lock();
269 y = p->data;
270 rcu_read_unlock();
271
272 Holding a reference from one RCU read-side critical section
273 to another is just as illegal as holding a reference from
274 one lock-based critical section to another! Similarly,
275 using a reference outside of the critical section in which
276 it was acquired is just as illegal as doing so with normal
277 locking.
278
279 As with rcu_assign_pointer(), an important function of
Paul E. McKenneyd19720a2006-02-01 03:06:42 -0800280 rcu_dereference() is to document which pointers are protected by
281 RCU, in particular, flagging a pointer that is subject to changing
282 at any time, including immediately after the rcu_dereference().
283 And, again like rcu_assign_pointer(), rcu_dereference() is
284 typically used indirectly, via the _rcu list-manipulation
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700285 primitives, such as list_for_each_entry_rcu().
286
287The following diagram shows how each API communicates among the
288reader, updater, and reclaimer.
289
290
291 rcu_assign_pointer()
292 +--------+
293 +---------------------->| reader |---------+
294 | +--------+ |
295 | | |
296 | | | Protect:
297 | | | rcu_read_lock()
298 | | | rcu_read_unlock()
299 | rcu_dereference() | |
300 +---------+ | |
301 | updater |<---------------------+ |
302 +---------+ V
303 | +-----------+
304 +----------------------------------->| reclaimer |
305 +-----------+
306 Defer:
307 synchronize_rcu() & call_rcu()
308
309
310The RCU infrastructure observes the time sequence of rcu_read_lock(),
311rcu_read_unlock(), synchronize_rcu(), and call_rcu() invocations in
312order to determine when (1) synchronize_rcu() invocations may return
313to their callers and (2) call_rcu() callbacks may be invoked. Efficient
314implementations of the RCU infrastructure make heavy use of batching in
315order to amortize their overhead over many uses of the corresponding APIs.
316
317There are no fewer than three RCU mechanisms in the Linux kernel; the
318diagram above shows the first one, which is by far the most commonly used.
319The rcu_dereference() and rcu_assign_pointer() primitives are used for
320all three mechanisms, but different defer and protect primitives are
321used as follows:
322
323 Defer Protect
324
325a. synchronize_rcu() rcu_read_lock() / rcu_read_unlock()
Paul E. McKenneyc598a072010-02-22 17:04:57 -0800326 call_rcu() rcu_dereference()
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700327
328b. call_rcu_bh() rcu_read_lock_bh() / rcu_read_unlock_bh()
Paul E. McKenneyc598a072010-02-22 17:04:57 -0800329 rcu_dereference_bh()
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700330
Paul E. McKenney4c540052010-01-14 16:10:57 -0800331c. synchronize_sched() rcu_read_lock_sched() / rcu_read_unlock_sched()
332 preempt_disable() / preempt_enable()
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700333 local_irq_save() / local_irq_restore()
334 hardirq enter / hardirq exit
335 NMI enter / NMI exit
Paul E. McKenneyc598a072010-02-22 17:04:57 -0800336 rcu_dereference_sched()
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700337
338These three mechanisms are used as follows:
339
340a. RCU applied to normal data structures.
341
342b. RCU applied to networking data structures that may be subjected
343 to remote denial-of-service attacks.
344
345c. RCU applied to scheduler and interrupt/NMI-handler tasks.
346
347Again, most uses will be of (a). The (b) and (c) cases are important
348for specialized uses, but are relatively uncommon.
349
350
3513. WHAT ARE SOME EXAMPLE USES OF CORE RCU API?
352
353This section shows a simple use of the core RCU API to protect a
Paul E. McKenneyd19720a2006-02-01 03:06:42 -0800354global pointer to a dynamically allocated structure. More-typical
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700355uses of RCU may be found in listRCU.txt, arrayRCU.txt, and NMI-RCU.txt.
356
357 struct foo {
358 int a;
359 char b;
360 long c;
361 };
362 DEFINE_SPINLOCK(foo_mutex);
363
364 struct foo *gbl_foo;
365
366 /*
367 * Create a new struct foo that is the same as the one currently
368 * pointed to by gbl_foo, except that field "a" is replaced
369 * with "new_a". Points gbl_foo to the new structure, and
370 * frees up the old structure after a grace period.
371 *
372 * Uses rcu_assign_pointer() to ensure that concurrent readers
373 * see the initialized version of the new structure.
374 *
375 * Uses synchronize_rcu() to ensure that any readers that might
376 * have references to the old structure complete before freeing
377 * the old structure.
378 */
379 void foo_update_a(int new_a)
380 {
381 struct foo *new_fp;
382 struct foo *old_fp;
383
Baruch Evende0dfcd2006-03-24 18:25:25 +0100384 new_fp = kmalloc(sizeof(*new_fp), GFP_KERNEL);
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700385 spin_lock(&foo_mutex);
386 old_fp = gbl_foo;
387 *new_fp = *old_fp;
388 new_fp->a = new_a;
389 rcu_assign_pointer(gbl_foo, new_fp);
390 spin_unlock(&foo_mutex);
391 synchronize_rcu();
392 kfree(old_fp);
393 }
394
395 /*
396 * Return the value of field "a" of the current gbl_foo
397 * structure. Use rcu_read_lock() and rcu_read_unlock()
398 * to ensure that the structure does not get deleted out
399 * from under us, and use rcu_dereference() to ensure that
400 * we see the initialized version of the structure (important
401 * for DEC Alpha and for people reading the code).
402 */
403 int foo_get_a(void)
404 {
405 int retval;
406
407 rcu_read_lock();
408 retval = rcu_dereference(gbl_foo)->a;
409 rcu_read_unlock();
410 return retval;
411 }
412
413So, to sum up:
414
415o Use rcu_read_lock() and rcu_read_unlock() to guard RCU
416 read-side critical sections.
417
418o Within an RCU read-side critical section, use rcu_dereference()
419 to dereference RCU-protected pointers.
420
421o Use some solid scheme (such as locks or semaphores) to
422 keep concurrent updates from interfering with each other.
423
424o Use rcu_assign_pointer() to update an RCU-protected pointer.
425 This primitive protects concurrent readers from the updater,
426 -not- concurrent updates from each other! You therefore still
427 need to use locking (or something similar) to keep concurrent
428 rcu_assign_pointer() primitives from interfering with each other.
429
430o Use synchronize_rcu() -after- removing a data element from an
431 RCU-protected data structure, but -before- reclaiming/freeing
432 the data element, in order to wait for the completion of all
433 RCU read-side critical sections that might be referencing that
434 data item.
435
436See checklist.txt for additional rules to follow when using RCU.
Paul E. McKenneyd19720a2006-02-01 03:06:42 -0800437And again, more-typical uses of RCU may be found in listRCU.txt,
438arrayRCU.txt, and NMI-RCU.txt.
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700439
440
4414. WHAT IF MY UPDATING THREAD CANNOT BLOCK?
442
443In the example above, foo_update_a() blocks until a grace period elapses.
444This is quite simple, but in some cases one cannot afford to wait so
445long -- there might be other high-priority work to be done.
446
447In such cases, one uses call_rcu() rather than synchronize_rcu().
448The call_rcu() API is as follows:
449
450 void call_rcu(struct rcu_head * head,
451 void (*func)(struct rcu_head *head));
452
453This function invokes func(head) after a grace period has elapsed.
454This invocation might happen from either softirq or process context,
455so the function is not permitted to block. The foo struct needs to
456have an rcu_head structure added, perhaps as follows:
457
458 struct foo {
459 int a;
460 char b;
461 long c;
462 struct rcu_head rcu;
463 };
464
465The foo_update_a() function might then be written as follows:
466
467 /*
468 * Create a new struct foo that is the same as the one currently
469 * pointed to by gbl_foo, except that field "a" is replaced
470 * with "new_a". Points gbl_foo to the new structure, and
471 * frees up the old structure after a grace period.
472 *
473 * Uses rcu_assign_pointer() to ensure that concurrent readers
474 * see the initialized version of the new structure.
475 *
476 * Uses call_rcu() to ensure that any readers that might have
477 * references to the old structure complete before freeing the
478 * old structure.
479 */
480 void foo_update_a(int new_a)
481 {
482 struct foo *new_fp;
483 struct foo *old_fp;
484
Baruch Evende0dfcd2006-03-24 18:25:25 +0100485 new_fp = kmalloc(sizeof(*new_fp), GFP_KERNEL);
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700486 spin_lock(&foo_mutex);
487 old_fp = gbl_foo;
488 *new_fp = *old_fp;
489 new_fp->a = new_a;
490 rcu_assign_pointer(gbl_foo, new_fp);
491 spin_unlock(&foo_mutex);
492 call_rcu(&old_fp->rcu, foo_reclaim);
493 }
494
495The foo_reclaim() function might appear as follows:
496
497 void foo_reclaim(struct rcu_head *rp)
498 {
499 struct foo *fp = container_of(rp, struct foo, rcu);
500
501 kfree(fp);
502 }
503
504The container_of() primitive is a macro that, given a pointer into a
505struct, the type of the struct, and the pointed-to field within the
506struct, returns a pointer to the beginning of the struct.
507
508The use of call_rcu() permits the caller of foo_update_a() to
509immediately regain control, without needing to worry further about the
510old version of the newly updated element. It also clearly shows the
511RCU distinction between updater, namely foo_update_a(), and reclaimer,
512namely foo_reclaim().
513
514The summary of advice is the same as for the previous section, except
515that we are now using call_rcu() rather than synchronize_rcu():
516
517o Use call_rcu() -after- removing a data element from an
518 RCU-protected data structure in order to register a callback
519 function that will be invoked after the completion of all RCU
520 read-side critical sections that might be referencing that
521 data item.
522
523Again, see checklist.txt for additional rules governing the use of RCU.
524
525
5265. WHAT ARE SOME SIMPLE IMPLEMENTATIONS OF RCU?
527
528One of the nice things about RCU is that it has extremely simple "toy"
529implementations that are a good first step towards understanding the
530production-quality implementations in the Linux kernel. This section
531presents two such "toy" implementations of RCU, one that is implemented
532in terms of familiar locking primitives, and another that more closely
533resembles "classic" RCU. Both are way too simple for real-world use,
534lacking both functionality and performance. However, they are useful
535in getting a feel for how RCU works. See kernel/rcupdate.c for a
536production-quality implementation, and see:
537
538 http://www.rdrop.com/users/paulmck/RCU
539
540for papers describing the Linux kernel RCU implementation. The OLS'01
541and OLS'02 papers are a good introduction, and the dissertation provides
Paul E. McKenneyd19720a2006-02-01 03:06:42 -0800542more details on the current implementation as of early 2004.
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700543
544
5455A. "TOY" IMPLEMENTATION #1: LOCKING
546
547This section presents a "toy" RCU implementation that is based on
548familiar locking primitives. Its overhead makes it a non-starter for
549real-life use, as does its lack of scalability. It is also unsuitable
550for realtime use, since it allows scheduling latency to "bleed" from
551one read-side critical section to another.
552
553However, it is probably the easiest implementation to relate to, so is
554a good starting point.
555
556It is extremely simple:
557
558 static DEFINE_RWLOCK(rcu_gp_mutex);
559
560 void rcu_read_lock(void)
561 {
562 read_lock(&rcu_gp_mutex);
563 }
564
565 void rcu_read_unlock(void)
566 {
567 read_unlock(&rcu_gp_mutex);
568 }
569
570 void synchronize_rcu(void)
571 {
572 write_lock(&rcu_gp_mutex);
573 write_unlock(&rcu_gp_mutex);
574 }
575
576[You can ignore rcu_assign_pointer() and rcu_dereference() without
577missing much. But here they are anyway. And whatever you do, don't
578forget about them when submitting patches making use of RCU!]
579
580 #define rcu_assign_pointer(p, v) ({ \
581 smp_wmb(); \
582 (p) = (v); \
583 })
584
585 #define rcu_dereference(p) ({ \
586 typeof(p) _________p1 = p; \
587 smp_read_barrier_depends(); \
588 (_________p1); \
589 })
590
591
592The rcu_read_lock() and rcu_read_unlock() primitive read-acquire
593and release a global reader-writer lock. The synchronize_rcu()
594primitive write-acquires this same lock, then immediately releases
595it. This means that once synchronize_rcu() exits, all RCU read-side
Matt LaPlante53cb4722006-10-03 22:55:17 +0200596critical sections that were in progress before synchronize_rcu() was
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700597called are guaranteed to have completed -- there is no way that
598synchronize_rcu() would have been able to write-acquire the lock
599otherwise.
600
601It is possible to nest rcu_read_lock(), since reader-writer locks may
602be recursively acquired. Note also that rcu_read_lock() is immune
603from deadlock (an important property of RCU). The reason for this is
604that the only thing that can block rcu_read_lock() is a synchronize_rcu().
605But synchronize_rcu() does not acquire any locks while holding rcu_gp_mutex,
606so there can be no deadlock cycle.
607
608Quick Quiz #1: Why is this argument naive? How could a deadlock
609 occur when using this algorithm in a real-world Linux
610 kernel? How could this deadlock be avoided?
611
612
6135B. "TOY" EXAMPLE #2: CLASSIC RCU
614
615This section presents a "toy" RCU implementation that is based on
616"classic RCU". It is also short on performance (but only for updates) and
617on features such as hotplug CPU and the ability to run in CONFIG_PREEMPT
618kernels. The definitions of rcu_dereference() and rcu_assign_pointer()
619are the same as those shown in the preceding section, so they are omitted.
620
621 void rcu_read_lock(void) { }
622
623 void rcu_read_unlock(void) { }
624
625 void synchronize_rcu(void)
626 {
627 int cpu;
628
KAMEZAWA Hiroyuki3c30a752006-03-28 01:56:39 -0800629 for_each_possible_cpu(cpu)
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700630 run_on(cpu);
631 }
632
633Note that rcu_read_lock() and rcu_read_unlock() do absolutely nothing.
634This is the great strength of classic RCU in a non-preemptive kernel:
635read-side overhead is precisely zero, at least on non-Alpha CPUs.
636And there is absolutely no way that rcu_read_lock() can possibly
637participate in a deadlock cycle!
638
639The implementation of synchronize_rcu() simply schedules itself on each
640CPU in turn. The run_on() primitive can be implemented straightforwardly
641in terms of the sched_setaffinity() primitive. Of course, a somewhat less
642"toy" implementation would restore the affinity upon completion rather
643than just leaving all tasks running on the last CPU, but when I said
644"toy", I meant -toy-!
645
646So how the heck is this supposed to work???
647
648Remember that it is illegal to block while in an RCU read-side critical
649section. Therefore, if a given CPU executes a context switch, we know
650that it must have completed all preceding RCU read-side critical sections.
651Once -all- CPUs have executed a context switch, then -all- preceding
652RCU read-side critical sections will have completed.
653
654So, suppose that we remove a data item from its structure and then invoke
655synchronize_rcu(). Once synchronize_rcu() returns, we are guaranteed
656that there are no RCU read-side critical sections holding a reference
657to that data item, so we can safely reclaim it.
658
659Quick Quiz #2: Give an example where Classic RCU's read-side
660 overhead is -negative-.
661
662Quick Quiz #3: If it is illegal to block in an RCU read-side
663 critical section, what the heck do you do in
664 PREEMPT_RT, where normal spinlocks can block???
665
666
6676. ANALOGY WITH READER-WRITER LOCKING
668
669Although RCU can be used in many different ways, a very common use of
670RCU is analogous to reader-writer locking. The following unified
671diff shows how closely related RCU and reader-writer locking can be.
672
673 @@ -13,15 +14,15 @@
674 struct list_head *lp;
675 struct el *p;
676
677 - read_lock();
678 - list_for_each_entry(p, head, lp) {
679 + rcu_read_lock();
680 + list_for_each_entry_rcu(p, head, lp) {
681 if (p->key == key) {
682 *result = p->data;
683 - read_unlock();
684 + rcu_read_unlock();
685 return 1;
686 }
687 }
688 - read_unlock();
689 + rcu_read_unlock();
690 return 0;
691 }
692
693 @@ -29,15 +30,16 @@
694 {
695 struct el *p;
696
697 - write_lock(&listmutex);
698 + spin_lock(&listmutex);
699 list_for_each_entry(p, head, lp) {
700 if (p->key == key) {
Urs Thuermann82a854e2006-07-10 04:44:06 -0700701 - list_del(&p->list);
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700702 - write_unlock(&listmutex);
Urs Thuermann82a854e2006-07-10 04:44:06 -0700703 + list_del_rcu(&p->list);
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700704 + spin_unlock(&listmutex);
705 + synchronize_rcu();
706 kfree(p);
707 return 1;
708 }
709 }
710 - write_unlock(&listmutex);
711 + spin_unlock(&listmutex);
712 return 0;
713 }
714
715Or, for those who prefer a side-by-side listing:
716
717 1 struct el { 1 struct el {
718 2 struct list_head list; 2 struct list_head list;
719 3 long key; 3 long key;
720 4 spinlock_t mutex; 4 spinlock_t mutex;
721 5 int data; 5 int data;
722 6 /* Other data fields */ 6 /* Other data fields */
723 7 }; 7 };
724 8 spinlock_t listmutex; 8 spinlock_t listmutex;
725 9 struct el head; 9 struct el head;
726
727 1 int search(long key, int *result) 1 int search(long key, int *result)
728 2 { 2 {
729 3 struct list_head *lp; 3 struct list_head *lp;
730 4 struct el *p; 4 struct el *p;
731 5 5
732 6 read_lock(); 6 rcu_read_lock();
733 7 list_for_each_entry(p, head, lp) { 7 list_for_each_entry_rcu(p, head, lp) {
734 8 if (p->key == key) { 8 if (p->key == key) {
735 9 *result = p->data; 9 *result = p->data;
73610 read_unlock(); 10 rcu_read_unlock();
73711 return 1; 11 return 1;
73812 } 12 }
73913 } 13 }
74014 read_unlock(); 14 rcu_read_unlock();
74115 return 0; 15 return 0;
74216 } 16 }
743
744 1 int delete(long key) 1 int delete(long key)
745 2 { 2 {
746 3 struct el *p; 3 struct el *p;
747 4 4
748 5 write_lock(&listmutex); 5 spin_lock(&listmutex);
749 6 list_for_each_entry(p, head, lp) { 6 list_for_each_entry(p, head, lp) {
750 7 if (p->key == key) { 7 if (p->key == key) {
Urs Thuermann82a854e2006-07-10 04:44:06 -0700751 8 list_del(&p->list); 8 list_del_rcu(&p->list);
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700752 9 write_unlock(&listmutex); 9 spin_unlock(&listmutex);
753 10 synchronize_rcu();
75410 kfree(p); 11 kfree(p);
75511 return 1; 12 return 1;
75612 } 13 }
75713 } 14 }
75814 write_unlock(&listmutex); 15 spin_unlock(&listmutex);
75915 return 0; 16 return 0;
76016 } 17 }
761
762Either way, the differences are quite small. Read-side locking moves
763to rcu_read_lock() and rcu_read_unlock, update-side locking moves from
Paolo Ornati670e9f32006-10-03 22:57:56 +0200764a reader-writer lock to a simple spinlock, and a synchronize_rcu()
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700765precedes the kfree().
766
767However, there is one potential catch: the read-side and update-side
768critical sections can now run concurrently. In many cases, this will
769not be a problem, but it is necessary to check carefully regardless.
770For example, if multiple independent list updates must be seen as
771a single atomic update, converting to RCU will require special care.
772
773Also, the presence of synchronize_rcu() means that the RCU version of
774delete() can now block. If this is a problem, there is a callback-based
775mechanism that never blocks, namely call_rcu(), that can be used in
776place of synchronize_rcu().
777
778
7797. FULL LIST OF RCU APIs
780
781The RCU APIs are documented in docbook-format header comments in the
782Linux-kernel source code, but it helps to have a full list of the
783APIs, since there does not appear to be a way to categorize them
784in docbook. Here is the list, by category.
785
Paul E. McKenneyc598a072010-02-22 17:04:57 -0800786RCU list traversal:
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700787
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700788 list_for_each_entry_rcu
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700789 hlist_for_each_entry_rcu
Paul E. McKenney240ebbf2009-06-25 09:08:18 -0700790 hlist_nulls_for_each_entry_rcu
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700791
Paul E. McKenney32300752008-05-12 21:21:05 +0200792 list_for_each_continue_rcu (to be deprecated in favor of new
793 list_for_each_entry_continue_rcu)
794
795RCU pointer/list update:
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700796
797 rcu_assign_pointer
798 list_add_rcu
799 list_add_tail_rcu
800 list_del_rcu
801 list_replace_rcu
802 hlist_del_rcu
Paul E. McKenney32300752008-05-12 21:21:05 +0200803 hlist_add_after_rcu
804 hlist_add_before_rcu
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700805 hlist_add_head_rcu
Paul E. McKenney32300752008-05-12 21:21:05 +0200806 hlist_replace_rcu
807 list_splice_init_rcu()
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700808
Paul E. McKenney32300752008-05-12 21:21:05 +0200809RCU: Critical sections Grace period Barrier
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700810
Paul E. McKenney32300752008-05-12 21:21:05 +0200811 rcu_read_lock synchronize_net rcu_barrier
812 rcu_read_unlock synchronize_rcu
Paul E. McKenneyc598a072010-02-22 17:04:57 -0800813 rcu_dereference synchronize_rcu_expedited
Paul E. McKenney32300752008-05-12 21:21:05 +0200814 call_rcu
815
816
817bh: Critical sections Grace period Barrier
818
819 rcu_read_lock_bh call_rcu_bh rcu_barrier_bh
Paul E. McKenney240ebbf2009-06-25 09:08:18 -0700820 rcu_read_unlock_bh synchronize_rcu_bh
Paul E. McKenneyc598a072010-02-22 17:04:57 -0800821 rcu_dereference_bh synchronize_rcu_bh_expedited
Paul E. McKenney32300752008-05-12 21:21:05 +0200822
823
824sched: Critical sections Grace period Barrier
825
Paul E. McKenney240ebbf2009-06-25 09:08:18 -0700826 rcu_read_lock_sched synchronize_sched rcu_barrier_sched
827 rcu_read_unlock_sched call_rcu_sched
828 [preempt_disable] synchronize_sched_expedited
829 [and friends]
Paul E. McKenneyc598a072010-02-22 17:04:57 -0800830 rcu_dereference_sched
Paul E. McKenney32300752008-05-12 21:21:05 +0200831
832
833SRCU: Critical sections Grace period Barrier
834
835 srcu_read_lock synchronize_srcu N/A
Paul E. McKenney64179862009-10-25 19:03:53 -0700836 srcu_read_unlock synchronize_srcu_expedited
Paul E. McKenneyc598a072010-02-22 17:04:57 -0800837 srcu_dereference
Paul E. McKenney32300752008-05-12 21:21:05 +0200838
Paul E. McKenney240ebbf2009-06-25 09:08:18 -0700839SRCU: Initialization/cleanup
840 init_srcu_struct
841 cleanup_srcu_struct
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700842
Paul E. McKenney50aec002010-04-09 15:39:12 -0700843All: lockdep-checked RCU-protected pointer access
844
845 rcu_dereference_check
846 rcu_dereference_protected
847 rcu_access_pointer
848
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700849See the comment headers in the source code (or the docbook generated
850from them) for more information.
851
Paul E. McKenneyfea65122011-01-23 22:35:45 -0800852However, given that there are no fewer than four families of RCU APIs
853in the Linux kernel, how do you choose which one to use? The following
854list can be helpful:
855
856a. Will readers need to block? If so, you need SRCU.
857
858b. What about the -rt patchset? If readers would need to block
859 in an non-rt kernel, you need SRCU. If readers would block
860 in a -rt kernel, but not in a non-rt kernel, SRCU is not
861 necessary.
862
863c. Do you need to treat NMI handlers, hardirq handlers,
864 and code segments with preemption disabled (whether
865 via preempt_disable(), local_irq_save(), local_bh_disable(),
866 or some other mechanism) as if they were explicit RCU readers?
867 If so, you need RCU-sched.
868
869d. Do you need RCU grace periods to complete even in the face
870 of softirq monopolization of one or more of the CPUs? For
871 example, is your code subject to network-based denial-of-service
872 attacks? If so, you need RCU-bh.
873
874e. Is your workload too update-intensive for normal use of
875 RCU, but inappropriate for other synchronization mechanisms?
876 If so, consider SLAB_DESTROY_BY_RCU. But please be careful!
877
878f. Otherwise, use RCU.
879
880Of course, this all assumes that you have determined that RCU is in fact
881the right tool for your job.
882
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700883
8848. ANSWERS TO QUICK QUIZZES
885
886Quick Quiz #1: Why is this argument naive? How could a deadlock
887 occur when using this algorithm in a real-world Linux
888 kernel? [Referring to the lock-based "toy" RCU
889 algorithm.]
890
891Answer: Consider the following sequence of events:
892
893 1. CPU 0 acquires some unrelated lock, call it
Paul E. McKenneyd19720a2006-02-01 03:06:42 -0800894 "problematic_lock", disabling irq via
895 spin_lock_irqsave().
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700896
897 2. CPU 1 enters synchronize_rcu(), write-acquiring
898 rcu_gp_mutex.
899
900 3. CPU 0 enters rcu_read_lock(), but must wait
901 because CPU 1 holds rcu_gp_mutex.
902
903 4. CPU 1 is interrupted, and the irq handler
904 attempts to acquire problematic_lock.
905
906 The system is now deadlocked.
907
908 One way to avoid this deadlock is to use an approach like
909 that of CONFIG_PREEMPT_RT, where all normal spinlocks
910 become blocking locks, and all irq handlers execute in
911 the context of special tasks. In this case, in step 4
912 above, the irq handler would block, allowing CPU 1 to
913 release rcu_gp_mutex, avoiding the deadlock.
914
915 Even in the absence of deadlock, this RCU implementation
916 allows latency to "bleed" from readers to other
917 readers through synchronize_rcu(). To see this,
918 consider task A in an RCU read-side critical section
919 (thus read-holding rcu_gp_mutex), task B blocked
920 attempting to write-acquire rcu_gp_mutex, and
921 task C blocked in rcu_read_lock() attempting to
922 read_acquire rcu_gp_mutex. Task A's RCU read-side
923 latency is holding up task C, albeit indirectly via
924 task B.
925
926 Realtime RCU implementations therefore use a counter-based
927 approach where tasks in RCU read-side critical sections
928 cannot be blocked by tasks executing synchronize_rcu().
929
930Quick Quiz #2: Give an example where Classic RCU's read-side
931 overhead is -negative-.
932
933Answer: Imagine a single-CPU system with a non-CONFIG_PREEMPT
934 kernel where a routing table is used by process-context
935 code, but can be updated by irq-context code (for example,
936 by an "ICMP REDIRECT" packet). The usual way of handling
937 this would be to have the process-context code disable
938 interrupts while searching the routing table. Use of
939 RCU allows such interrupt-disabling to be dispensed with.
940 Thus, without RCU, you pay the cost of disabling interrupts,
941 and with RCU you don't.
942
943 One can argue that the overhead of RCU in this
944 case is negative with respect to the single-CPU
945 interrupt-disabling approach. Others might argue that
946 the overhead of RCU is merely zero, and that replacing
947 the positive overhead of the interrupt-disabling scheme
948 with the zero-overhead RCU scheme does not constitute
949 negative overhead.
950
951 In real life, of course, things are more complex. But
952 even the theoretical possibility of negative overhead for
953 a synchronization primitive is a bit unexpected. ;-)
954
955Quick Quiz #3: If it is illegal to block in an RCU read-side
956 critical section, what the heck do you do in
957 PREEMPT_RT, where normal spinlocks can block???
958
959Answer: Just as PREEMPT_RT permits preemption of spinlock
960 critical sections, it permits preemption of RCU
961 read-side critical sections. It also permits
962 spinlocks blocking while in RCU read-side critical
963 sections.
964
965 Why the apparent inconsistency? Because it is it
966 possible to use priority boosting to keep the RCU
967 grace periods short if need be (for example, if running
968 short of memory). In contrast, if blocking waiting
969 for (say) network reception, there is no way to know
970 what should be boosted. Especially given that the
971 process we need to boost might well be a human being
972 who just went out for a pizza or something. And although
973 a computer-operated cattle prod might arouse serious
974 interest, it might also provoke serious objections.
975 Besides, how does the computer know what pizza parlor
976 the human being went to???
977
978
979ACKNOWLEDGEMENTS
980
981My thanks to the people who helped make this human-readable, including
Paul E. McKenneyd19720a2006-02-01 03:06:42 -0800982Jon Walpole, Josh Triplett, Serge Hallyn, Suzanne Wood, and Alan Stern.
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700983
984
985For more information, see http://www.rdrop.com/users/paulmck/RCU.