blob: e41a7fecf0d3c33354aef84985cb564619ab6c77 [file] [log] [blame]
Paul E. McKenney32300752008-05-12 21:21:05 +02001Please note that the "What is RCU?" LWN series is an excellent place
2to start learning about RCU:
3
41. What is RCU, Fundamentally? http://lwn.net/Articles/262464/
52. What is RCU? Part 2: Usage http://lwn.net/Articles/263130/
63. RCU part 3: the RCU API http://lwn.net/Articles/264090/
7
8
Paul E. McKenneydd81eca2005-09-10 00:26:24 -07009What is RCU?
10
11RCU is a synchronization mechanism that was added to the Linux kernel
12during the 2.5 development effort that is optimized for read-mostly
13situations. Although RCU is actually quite simple once you understand it,
14getting there can sometimes be a challenge. Part of the problem is that
15most of the past descriptions of RCU have been written with the mistaken
16assumption that there is "one true way" to describe RCU. Instead,
17the experience has been that different people must take different paths
18to arrive at an understanding of RCU. This document provides several
19different paths, as follows:
20
211. RCU OVERVIEW
222. WHAT IS RCU'S CORE API?
233. WHAT ARE SOME EXAMPLE USES OF CORE RCU API?
244. WHAT IF MY UPDATING THREAD CANNOT BLOCK?
255. WHAT ARE SOME SIMPLE IMPLEMENTATIONS OF RCU?
266. ANALOGY WITH READER-WRITER LOCKING
277. FULL LIST OF RCU APIs
288. ANSWERS TO QUICK QUIZZES
29
30People who prefer starting with a conceptual overview should focus on
31Section 1, though most readers will profit by reading this section at
32some point. People who prefer to start with an API that they can then
33experiment with should focus on Section 2. People who prefer to start
34with example uses should focus on Sections 3 and 4. People who need to
35understand the RCU implementation should focus on Section 5, then dive
36into the kernel source code. People who reason best by analogy should
37focus on Section 6. Section 7 serves as an index to the docbook API
38documentation, and Section 8 is the traditional answer key.
39
40So, start with the section that makes the most sense to you and your
41preferred method of learning. If you need to know everything about
42everything, feel free to read the whole thing -- but if you are really
43that type of person, you have perused the source code and will therefore
44never need this document anyway. ;-)
45
46
471. RCU OVERVIEW
48
49The basic idea behind RCU is to split updates into "removal" and
50"reclamation" phases. The removal phase removes references to data items
51within a data structure (possibly by replacing them with references to
52new versions of these data items), and can run concurrently with readers.
53The reason that it is safe to run the removal phase concurrently with
54readers is the semantics of modern CPUs guarantee that readers will see
55either the old or the new version of the data structure rather than a
56partially updated reference. The reclamation phase does the work of reclaiming
57(e.g., freeing) the data items removed from the data structure during the
58removal phase. Because reclaiming data items can disrupt any readers
59concurrently referencing those data items, the reclamation phase must
60not start until readers no longer hold references to those data items.
61
62Splitting the update into removal and reclamation phases permits the
63updater to perform the removal phase immediately, and to defer the
64reclamation phase until all readers active during the removal phase have
65completed, either by blocking until they finish or by registering a
66callback that is invoked after they finish. Only readers that are active
67during the removal phase need be considered, because any reader starting
68after the removal phase will be unable to gain a reference to the removed
69data items, and therefore cannot be disrupted by the reclamation phase.
70
71So the typical RCU update sequence goes something like the following:
72
73a. Remove pointers to a data structure, so that subsequent
74 readers cannot gain a reference to it.
75
76b. Wait for all previous readers to complete their RCU read-side
77 critical sections.
78
79c. At this point, there cannot be any readers who hold references
80 to the data structure, so it now may safely be reclaimed
81 (e.g., kfree()d).
82
83Step (b) above is the key idea underlying RCU's deferred destruction.
84The ability to wait until all readers are done allows RCU readers to
85use much lighter-weight synchronization, in some cases, absolutely no
86synchronization at all. In contrast, in more conventional lock-based
87schemes, readers must use heavy-weight synchronization in order to
88prevent an updater from deleting the data structure out from under them.
89This is because lock-based updaters typically update data items in place,
90and must therefore exclude readers. In contrast, RCU-based updaters
91typically take advantage of the fact that writes to single aligned
92pointers are atomic on modern CPUs, allowing atomic insertion, removal,
93and replacement of data items in a linked structure without disrupting
94readers. Concurrent RCU readers can then continue accessing the old
95versions, and can dispense with the atomic operations, memory barriers,
96and communications cache misses that are so expensive on present-day
97SMP computer systems, even in absence of lock contention.
98
99In the three-step procedure shown above, the updater is performing both
100the removal and the reclamation step, but it is often helpful for an
101entirely different thread to do the reclamation, as is in fact the case
102in the Linux kernel's directory-entry cache (dcache). Even if the same
103thread performs both the update step (step (a) above) and the reclamation
104step (step (c) above), it is often helpful to think of them separately.
105For example, RCU readers and updaters need not communicate at all,
106but RCU provides implicit low-overhead communication between readers
107and reclaimers, namely, in step (b) above.
108
109So how the heck can a reclaimer tell when a reader is done, given
110that readers are not doing any sort of synchronization operations???
111Read on to learn about how RCU's API makes this easy.
112
113
1142. WHAT IS RCU'S CORE API?
115
116The core RCU API is quite small:
117
118a. rcu_read_lock()
119b. rcu_read_unlock()
120c. synchronize_rcu() / call_rcu()
121d. rcu_assign_pointer()
122e. rcu_dereference()
123
124There are many other members of the RCU API, but the rest can be
125expressed in terms of these five, though most implementations instead
126express synchronize_rcu() in terms of the call_rcu() callback API.
127
128The five core RCU APIs are described below, the other 18 will be enumerated
129later. See the kernel docbook documentation for more info, or look directly
130at the function header comments.
131
132rcu_read_lock()
133
134 void rcu_read_lock(void);
135
136 Used by a reader to inform the reclaimer that the reader is
137 entering an RCU read-side critical section. It is illegal
138 to block while in an RCU read-side critical section, though
Paul E. McKenney6b3ef482009-08-22 13:56:53 -0700139 kernels built with CONFIG_TREE_PREEMPT_RCU can preempt RCU
140 read-side critical sections. Any RCU-protected data structure
141 accessed during an RCU read-side critical section is guaranteed to
142 remain unreclaimed for the full duration of that critical section.
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700143 Reference counts may be used in conjunction with RCU to maintain
144 longer-term references to data structures.
145
146rcu_read_unlock()
147
148 void rcu_read_unlock(void);
149
150 Used by a reader to inform the reclaimer that the reader is
151 exiting an RCU read-side critical section. Note that RCU
152 read-side critical sections may be nested and/or overlapping.
153
154synchronize_rcu()
155
156 void synchronize_rcu(void);
157
158 Marks the end of updater code and the beginning of reclaimer
159 code. It does this by blocking until all pre-existing RCU
160 read-side critical sections on all CPUs have completed.
161 Note that synchronize_rcu() will -not- necessarily wait for
162 any subsequent RCU read-side critical sections to complete.
163 For example, consider the following sequence of events:
164
165 CPU 0 CPU 1 CPU 2
166 ----------------- ------------------------- ---------------
167 1. rcu_read_lock()
168 2. enters synchronize_rcu()
169 3. rcu_read_lock()
170 4. rcu_read_unlock()
171 5. exits synchronize_rcu()
172 6. rcu_read_unlock()
173
174 To reiterate, synchronize_rcu() waits only for ongoing RCU
175 read-side critical sections to complete, not necessarily for
176 any that begin after synchronize_rcu() is invoked.
177
178 Of course, synchronize_rcu() does not necessarily return
179 -immediately- after the last pre-existing RCU read-side critical
180 section completes. For one thing, there might well be scheduling
181 delays. For another thing, many RCU implementations process
182 requests in batches in order to improve efficiencies, which can
183 further delay synchronize_rcu().
184
185 Since synchronize_rcu() is the API that must figure out when
186 readers are done, its implementation is key to RCU. For RCU
187 to be useful in all but the most read-intensive situations,
188 synchronize_rcu()'s overhead must also be quite small.
189
190 The call_rcu() API is a callback form of synchronize_rcu(),
191 and is described in more detail in a later section. Instead of
192 blocking, it registers a function and argument which are invoked
193 after all ongoing RCU read-side critical sections have completed.
194 This callback variant is particularly useful in situations where
Paul E. McKenney165d6c72006-06-25 05:48:44 -0700195 it is illegal to block or where update-side performance is
196 critically important.
197
198 However, the call_rcu() API should not be used lightly, as use
199 of the synchronize_rcu() API generally results in simpler code.
200 In addition, the synchronize_rcu() API has the nice property
201 of automatically limiting update rate should grace periods
202 be delayed. This property results in system resilience in face
203 of denial-of-service attacks. Code using call_rcu() should limit
204 update rate in order to gain this same sort of resilience. See
205 checklist.txt for some approaches to limiting the update rate.
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700206
207rcu_assign_pointer()
208
209 typeof(p) rcu_assign_pointer(p, typeof(p) v);
210
211 Yes, rcu_assign_pointer() -is- implemented as a macro, though it
212 would be cool to be able to declare a function in this manner.
213 (Compiler experts will no doubt disagree.)
214
215 The updater uses this function to assign a new value to an
216 RCU-protected pointer, in order to safely communicate the change
217 in value from the updater to the reader. This function returns
218 the new value, and also executes any memory-barrier instructions
219 required for a given CPU architecture.
220
Paul E. McKenneyd19720a2006-02-01 03:06:42 -0800221 Perhaps just as important, it serves to document (1) which
222 pointers are protected by RCU and (2) the point at which a
223 given structure becomes accessible to other CPUs. That said,
224 rcu_assign_pointer() is most frequently used indirectly, via
225 the _rcu list-manipulation primitives such as list_add_rcu().
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700226
227rcu_dereference()
228
229 typeof(p) rcu_dereference(p);
230
231 Like rcu_assign_pointer(), rcu_dereference() must be implemented
232 as a macro.
233
234 The reader uses rcu_dereference() to fetch an RCU-protected
235 pointer, which returns a value that may then be safely
236 dereferenced. Note that rcu_deference() does not actually
237 dereference the pointer, instead, it protects the pointer for
238 later dereferencing. It also executes any needed memory-barrier
239 instructions for a given CPU architecture. Currently, only Alpha
240 needs memory barriers within rcu_dereference() -- on other CPUs,
241 it compiles to nothing, not even a compiler directive.
242
243 Common coding practice uses rcu_dereference() to copy an
244 RCU-protected pointer to a local variable, then dereferences
245 this local variable, for example as follows:
246
247 p = rcu_dereference(head.next);
248 return p->data;
249
250 However, in this case, one could just as easily combine these
251 into one statement:
252
253 return rcu_dereference(head.next)->data;
254
255 If you are going to be fetching multiple fields from the
256 RCU-protected structure, using the local variable is of
257 course preferred. Repeated rcu_dereference() calls look
258 ugly and incur unnecessary overhead on Alpha CPUs.
259
260 Note that the value returned by rcu_dereference() is valid
261 only within the enclosing RCU read-side critical section.
262 For example, the following is -not- legal:
263
264 rcu_read_lock();
265 p = rcu_dereference(head.next);
266 rcu_read_unlock();
267 x = p->address;
268 rcu_read_lock();
269 y = p->data;
270 rcu_read_unlock();
271
272 Holding a reference from one RCU read-side critical section
273 to another is just as illegal as holding a reference from
274 one lock-based critical section to another! Similarly,
275 using a reference outside of the critical section in which
276 it was acquired is just as illegal as doing so with normal
277 locking.
278
279 As with rcu_assign_pointer(), an important function of
Paul E. McKenneyd19720a2006-02-01 03:06:42 -0800280 rcu_dereference() is to document which pointers are protected by
281 RCU, in particular, flagging a pointer that is subject to changing
282 at any time, including immediately after the rcu_dereference().
283 And, again like rcu_assign_pointer(), rcu_dereference() is
284 typically used indirectly, via the _rcu list-manipulation
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700285 primitives, such as list_for_each_entry_rcu().
286
287The following diagram shows how each API communicates among the
288reader, updater, and reclaimer.
289
290
291 rcu_assign_pointer()
292 +--------+
293 +---------------------->| reader |---------+
294 | +--------+ |
295 | | |
296 | | | Protect:
297 | | | rcu_read_lock()
298 | | | rcu_read_unlock()
299 | rcu_dereference() | |
300 +---------+ | |
301 | updater |<---------------------+ |
302 +---------+ V
303 | +-----------+
304 +----------------------------------->| reclaimer |
305 +-----------+
306 Defer:
307 synchronize_rcu() & call_rcu()
308
309
310The RCU infrastructure observes the time sequence of rcu_read_lock(),
311rcu_read_unlock(), synchronize_rcu(), and call_rcu() invocations in
312order to determine when (1) synchronize_rcu() invocations may return
313to their callers and (2) call_rcu() callbacks may be invoked. Efficient
314implementations of the RCU infrastructure make heavy use of batching in
315order to amortize their overhead over many uses of the corresponding APIs.
316
317There are no fewer than three RCU mechanisms in the Linux kernel; the
318diagram above shows the first one, which is by far the most commonly used.
319The rcu_dereference() and rcu_assign_pointer() primitives are used for
320all three mechanisms, but different defer and protect primitives are
321used as follows:
322
323 Defer Protect
324
325a. synchronize_rcu() rcu_read_lock() / rcu_read_unlock()
326 call_rcu()
327
328b. call_rcu_bh() rcu_read_lock_bh() / rcu_read_unlock_bh()
329
330c. synchronize_sched() preempt_disable() / preempt_enable()
331 local_irq_save() / local_irq_restore()
332 hardirq enter / hardirq exit
333 NMI enter / NMI exit
334
335These three mechanisms are used as follows:
336
337a. RCU applied to normal data structures.
338
339b. RCU applied to networking data structures that may be subjected
340 to remote denial-of-service attacks.
341
342c. RCU applied to scheduler and interrupt/NMI-handler tasks.
343
344Again, most uses will be of (a). The (b) and (c) cases are important
345for specialized uses, but are relatively uncommon.
346
347
3483. WHAT ARE SOME EXAMPLE USES OF CORE RCU API?
349
350This section shows a simple use of the core RCU API to protect a
Paul E. McKenneyd19720a2006-02-01 03:06:42 -0800351global pointer to a dynamically allocated structure. More-typical
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700352uses of RCU may be found in listRCU.txt, arrayRCU.txt, and NMI-RCU.txt.
353
354 struct foo {
355 int a;
356 char b;
357 long c;
358 };
359 DEFINE_SPINLOCK(foo_mutex);
360
361 struct foo *gbl_foo;
362
363 /*
364 * Create a new struct foo that is the same as the one currently
365 * pointed to by gbl_foo, except that field "a" is replaced
366 * with "new_a". Points gbl_foo to the new structure, and
367 * frees up the old structure after a grace period.
368 *
369 * Uses rcu_assign_pointer() to ensure that concurrent readers
370 * see the initialized version of the new structure.
371 *
372 * Uses synchronize_rcu() to ensure that any readers that might
373 * have references to the old structure complete before freeing
374 * the old structure.
375 */
376 void foo_update_a(int new_a)
377 {
378 struct foo *new_fp;
379 struct foo *old_fp;
380
Baruch Evende0dfcd2006-03-24 18:25:25 +0100381 new_fp = kmalloc(sizeof(*new_fp), GFP_KERNEL);
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700382 spin_lock(&foo_mutex);
383 old_fp = gbl_foo;
384 *new_fp = *old_fp;
385 new_fp->a = new_a;
386 rcu_assign_pointer(gbl_foo, new_fp);
387 spin_unlock(&foo_mutex);
388 synchronize_rcu();
389 kfree(old_fp);
390 }
391
392 /*
393 * Return the value of field "a" of the current gbl_foo
394 * structure. Use rcu_read_lock() and rcu_read_unlock()
395 * to ensure that the structure does not get deleted out
396 * from under us, and use rcu_dereference() to ensure that
397 * we see the initialized version of the structure (important
398 * for DEC Alpha and for people reading the code).
399 */
400 int foo_get_a(void)
401 {
402 int retval;
403
404 rcu_read_lock();
405 retval = rcu_dereference(gbl_foo)->a;
406 rcu_read_unlock();
407 return retval;
408 }
409
410So, to sum up:
411
412o Use rcu_read_lock() and rcu_read_unlock() to guard RCU
413 read-side critical sections.
414
415o Within an RCU read-side critical section, use rcu_dereference()
416 to dereference RCU-protected pointers.
417
418o Use some solid scheme (such as locks or semaphores) to
419 keep concurrent updates from interfering with each other.
420
421o Use rcu_assign_pointer() to update an RCU-protected pointer.
422 This primitive protects concurrent readers from the updater,
423 -not- concurrent updates from each other! You therefore still
424 need to use locking (or something similar) to keep concurrent
425 rcu_assign_pointer() primitives from interfering with each other.
426
427o Use synchronize_rcu() -after- removing a data element from an
428 RCU-protected data structure, but -before- reclaiming/freeing
429 the data element, in order to wait for the completion of all
430 RCU read-side critical sections that might be referencing that
431 data item.
432
433See checklist.txt for additional rules to follow when using RCU.
Paul E. McKenneyd19720a2006-02-01 03:06:42 -0800434And again, more-typical uses of RCU may be found in listRCU.txt,
435arrayRCU.txt, and NMI-RCU.txt.
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700436
437
4384. WHAT IF MY UPDATING THREAD CANNOT BLOCK?
439
440In the example above, foo_update_a() blocks until a grace period elapses.
441This is quite simple, but in some cases one cannot afford to wait so
442long -- there might be other high-priority work to be done.
443
444In such cases, one uses call_rcu() rather than synchronize_rcu().
445The call_rcu() API is as follows:
446
447 void call_rcu(struct rcu_head * head,
448 void (*func)(struct rcu_head *head));
449
450This function invokes func(head) after a grace period has elapsed.
451This invocation might happen from either softirq or process context,
452so the function is not permitted to block. The foo struct needs to
453have an rcu_head structure added, perhaps as follows:
454
455 struct foo {
456 int a;
457 char b;
458 long c;
459 struct rcu_head rcu;
460 };
461
462The foo_update_a() function might then be written as follows:
463
464 /*
465 * Create a new struct foo that is the same as the one currently
466 * pointed to by gbl_foo, except that field "a" is replaced
467 * with "new_a". Points gbl_foo to the new structure, and
468 * frees up the old structure after a grace period.
469 *
470 * Uses rcu_assign_pointer() to ensure that concurrent readers
471 * see the initialized version of the new structure.
472 *
473 * Uses call_rcu() to ensure that any readers that might have
474 * references to the old structure complete before freeing the
475 * old structure.
476 */
477 void foo_update_a(int new_a)
478 {
479 struct foo *new_fp;
480 struct foo *old_fp;
481
Baruch Evende0dfcd2006-03-24 18:25:25 +0100482 new_fp = kmalloc(sizeof(*new_fp), GFP_KERNEL);
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700483 spin_lock(&foo_mutex);
484 old_fp = gbl_foo;
485 *new_fp = *old_fp;
486 new_fp->a = new_a;
487 rcu_assign_pointer(gbl_foo, new_fp);
488 spin_unlock(&foo_mutex);
489 call_rcu(&old_fp->rcu, foo_reclaim);
490 }
491
492The foo_reclaim() function might appear as follows:
493
494 void foo_reclaim(struct rcu_head *rp)
495 {
496 struct foo *fp = container_of(rp, struct foo, rcu);
497
498 kfree(fp);
499 }
500
501The container_of() primitive is a macro that, given a pointer into a
502struct, the type of the struct, and the pointed-to field within the
503struct, returns a pointer to the beginning of the struct.
504
505The use of call_rcu() permits the caller of foo_update_a() to
506immediately regain control, without needing to worry further about the
507old version of the newly updated element. It also clearly shows the
508RCU distinction between updater, namely foo_update_a(), and reclaimer,
509namely foo_reclaim().
510
511The summary of advice is the same as for the previous section, except
512that we are now using call_rcu() rather than synchronize_rcu():
513
514o Use call_rcu() -after- removing a data element from an
515 RCU-protected data structure in order to register a callback
516 function that will be invoked after the completion of all RCU
517 read-side critical sections that might be referencing that
518 data item.
519
520Again, see checklist.txt for additional rules governing the use of RCU.
521
522
5235. WHAT ARE SOME SIMPLE IMPLEMENTATIONS OF RCU?
524
525One of the nice things about RCU is that it has extremely simple "toy"
526implementations that are a good first step towards understanding the
527production-quality implementations in the Linux kernel. This section
528presents two such "toy" implementations of RCU, one that is implemented
529in terms of familiar locking primitives, and another that more closely
530resembles "classic" RCU. Both are way too simple for real-world use,
531lacking both functionality and performance. However, they are useful
532in getting a feel for how RCU works. See kernel/rcupdate.c for a
533production-quality implementation, and see:
534
535 http://www.rdrop.com/users/paulmck/RCU
536
537for papers describing the Linux kernel RCU implementation. The OLS'01
538and OLS'02 papers are a good introduction, and the dissertation provides
Paul E. McKenneyd19720a2006-02-01 03:06:42 -0800539more details on the current implementation as of early 2004.
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700540
541
5425A. "TOY" IMPLEMENTATION #1: LOCKING
543
544This section presents a "toy" RCU implementation that is based on
545familiar locking primitives. Its overhead makes it a non-starter for
546real-life use, as does its lack of scalability. It is also unsuitable
547for realtime use, since it allows scheduling latency to "bleed" from
548one read-side critical section to another.
549
550However, it is probably the easiest implementation to relate to, so is
551a good starting point.
552
553It is extremely simple:
554
555 static DEFINE_RWLOCK(rcu_gp_mutex);
556
557 void rcu_read_lock(void)
558 {
559 read_lock(&rcu_gp_mutex);
560 }
561
562 void rcu_read_unlock(void)
563 {
564 read_unlock(&rcu_gp_mutex);
565 }
566
567 void synchronize_rcu(void)
568 {
569 write_lock(&rcu_gp_mutex);
570 write_unlock(&rcu_gp_mutex);
571 }
572
573[You can ignore rcu_assign_pointer() and rcu_dereference() without
574missing much. But here they are anyway. And whatever you do, don't
575forget about them when submitting patches making use of RCU!]
576
577 #define rcu_assign_pointer(p, v) ({ \
578 smp_wmb(); \
579 (p) = (v); \
580 })
581
582 #define rcu_dereference(p) ({ \
583 typeof(p) _________p1 = p; \
584 smp_read_barrier_depends(); \
585 (_________p1); \
586 })
587
588
589The rcu_read_lock() and rcu_read_unlock() primitive read-acquire
590and release a global reader-writer lock. The synchronize_rcu()
591primitive write-acquires this same lock, then immediately releases
592it. This means that once synchronize_rcu() exits, all RCU read-side
Matt LaPlante53cb4722006-10-03 22:55:17 +0200593critical sections that were in progress before synchronize_rcu() was
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700594called are guaranteed to have completed -- there is no way that
595synchronize_rcu() would have been able to write-acquire the lock
596otherwise.
597
598It is possible to nest rcu_read_lock(), since reader-writer locks may
599be recursively acquired. Note also that rcu_read_lock() is immune
600from deadlock (an important property of RCU). The reason for this is
601that the only thing that can block rcu_read_lock() is a synchronize_rcu().
602But synchronize_rcu() does not acquire any locks while holding rcu_gp_mutex,
603so there can be no deadlock cycle.
604
605Quick Quiz #1: Why is this argument naive? How could a deadlock
606 occur when using this algorithm in a real-world Linux
607 kernel? How could this deadlock be avoided?
608
609
6105B. "TOY" EXAMPLE #2: CLASSIC RCU
611
612This section presents a "toy" RCU implementation that is based on
613"classic RCU". It is also short on performance (but only for updates) and
614on features such as hotplug CPU and the ability to run in CONFIG_PREEMPT
615kernels. The definitions of rcu_dereference() and rcu_assign_pointer()
616are the same as those shown in the preceding section, so they are omitted.
617
618 void rcu_read_lock(void) { }
619
620 void rcu_read_unlock(void) { }
621
622 void synchronize_rcu(void)
623 {
624 int cpu;
625
KAMEZAWA Hiroyuki3c30a752006-03-28 01:56:39 -0800626 for_each_possible_cpu(cpu)
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700627 run_on(cpu);
628 }
629
630Note that rcu_read_lock() and rcu_read_unlock() do absolutely nothing.
631This is the great strength of classic RCU in a non-preemptive kernel:
632read-side overhead is precisely zero, at least on non-Alpha CPUs.
633And there is absolutely no way that rcu_read_lock() can possibly
634participate in a deadlock cycle!
635
636The implementation of synchronize_rcu() simply schedules itself on each
637CPU in turn. The run_on() primitive can be implemented straightforwardly
638in terms of the sched_setaffinity() primitive. Of course, a somewhat less
639"toy" implementation would restore the affinity upon completion rather
640than just leaving all tasks running on the last CPU, but when I said
641"toy", I meant -toy-!
642
643So how the heck is this supposed to work???
644
645Remember that it is illegal to block while in an RCU read-side critical
646section. Therefore, if a given CPU executes a context switch, we know
647that it must have completed all preceding RCU read-side critical sections.
648Once -all- CPUs have executed a context switch, then -all- preceding
649RCU read-side critical sections will have completed.
650
651So, suppose that we remove a data item from its structure and then invoke
652synchronize_rcu(). Once synchronize_rcu() returns, we are guaranteed
653that there are no RCU read-side critical sections holding a reference
654to that data item, so we can safely reclaim it.
655
656Quick Quiz #2: Give an example where Classic RCU's read-side
657 overhead is -negative-.
658
659Quick Quiz #3: If it is illegal to block in an RCU read-side
660 critical section, what the heck do you do in
661 PREEMPT_RT, where normal spinlocks can block???
662
663
6646. ANALOGY WITH READER-WRITER LOCKING
665
666Although RCU can be used in many different ways, a very common use of
667RCU is analogous to reader-writer locking. The following unified
668diff shows how closely related RCU and reader-writer locking can be.
669
670 @@ -13,15 +14,15 @@
671 struct list_head *lp;
672 struct el *p;
673
674 - read_lock();
675 - list_for_each_entry(p, head, lp) {
676 + rcu_read_lock();
677 + list_for_each_entry_rcu(p, head, lp) {
678 if (p->key == key) {
679 *result = p->data;
680 - read_unlock();
681 + rcu_read_unlock();
682 return 1;
683 }
684 }
685 - read_unlock();
686 + rcu_read_unlock();
687 return 0;
688 }
689
690 @@ -29,15 +30,16 @@
691 {
692 struct el *p;
693
694 - write_lock(&listmutex);
695 + spin_lock(&listmutex);
696 list_for_each_entry(p, head, lp) {
697 if (p->key == key) {
Urs Thuermann82a854e2006-07-10 04:44:06 -0700698 - list_del(&p->list);
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700699 - write_unlock(&listmutex);
Urs Thuermann82a854e2006-07-10 04:44:06 -0700700 + list_del_rcu(&p->list);
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700701 + spin_unlock(&listmutex);
702 + synchronize_rcu();
703 kfree(p);
704 return 1;
705 }
706 }
707 - write_unlock(&listmutex);
708 + spin_unlock(&listmutex);
709 return 0;
710 }
711
712Or, for those who prefer a side-by-side listing:
713
714 1 struct el { 1 struct el {
715 2 struct list_head list; 2 struct list_head list;
716 3 long key; 3 long key;
717 4 spinlock_t mutex; 4 spinlock_t mutex;
718 5 int data; 5 int data;
719 6 /* Other data fields */ 6 /* Other data fields */
720 7 }; 7 };
721 8 spinlock_t listmutex; 8 spinlock_t listmutex;
722 9 struct el head; 9 struct el head;
723
724 1 int search(long key, int *result) 1 int search(long key, int *result)
725 2 { 2 {
726 3 struct list_head *lp; 3 struct list_head *lp;
727 4 struct el *p; 4 struct el *p;
728 5 5
729 6 read_lock(); 6 rcu_read_lock();
730 7 list_for_each_entry(p, head, lp) { 7 list_for_each_entry_rcu(p, head, lp) {
731 8 if (p->key == key) { 8 if (p->key == key) {
732 9 *result = p->data; 9 *result = p->data;
73310 read_unlock(); 10 rcu_read_unlock();
73411 return 1; 11 return 1;
73512 } 12 }
73613 } 13 }
73714 read_unlock(); 14 rcu_read_unlock();
73815 return 0; 15 return 0;
73916 } 16 }
740
741 1 int delete(long key) 1 int delete(long key)
742 2 { 2 {
743 3 struct el *p; 3 struct el *p;
744 4 4
745 5 write_lock(&listmutex); 5 spin_lock(&listmutex);
746 6 list_for_each_entry(p, head, lp) { 6 list_for_each_entry(p, head, lp) {
747 7 if (p->key == key) { 7 if (p->key == key) {
Urs Thuermann82a854e2006-07-10 04:44:06 -0700748 8 list_del(&p->list); 8 list_del_rcu(&p->list);
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700749 9 write_unlock(&listmutex); 9 spin_unlock(&listmutex);
750 10 synchronize_rcu();
75110 kfree(p); 11 kfree(p);
75211 return 1; 12 return 1;
75312 } 13 }
75413 } 14 }
75514 write_unlock(&listmutex); 15 spin_unlock(&listmutex);
75615 return 0; 16 return 0;
75716 } 17 }
758
759Either way, the differences are quite small. Read-side locking moves
760to rcu_read_lock() and rcu_read_unlock, update-side locking moves from
Paolo Ornati670e9f32006-10-03 22:57:56 +0200761a reader-writer lock to a simple spinlock, and a synchronize_rcu()
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700762precedes the kfree().
763
764However, there is one potential catch: the read-side and update-side
765critical sections can now run concurrently. In many cases, this will
766not be a problem, but it is necessary to check carefully regardless.
767For example, if multiple independent list updates must be seen as
768a single atomic update, converting to RCU will require special care.
769
770Also, the presence of synchronize_rcu() means that the RCU version of
771delete() can now block. If this is a problem, there is a callback-based
772mechanism that never blocks, namely call_rcu(), that can be used in
773place of synchronize_rcu().
774
775
7767. FULL LIST OF RCU APIs
777
778The RCU APIs are documented in docbook-format header comments in the
779Linux-kernel source code, but it helps to have a full list of the
780APIs, since there does not appear to be a way to categorize them
781in docbook. Here is the list, by category.
782
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700783RCU pointer/list traversal:
784
785 rcu_dereference
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700786 list_for_each_entry_rcu
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700787 hlist_for_each_entry_rcu
Paul E. McKenney240ebbf2009-06-25 09:08:18 -0700788 hlist_nulls_for_each_entry_rcu
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700789
Paul E. McKenney32300752008-05-12 21:21:05 +0200790 list_for_each_continue_rcu (to be deprecated in favor of new
791 list_for_each_entry_continue_rcu)
792
793RCU pointer/list update:
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700794
795 rcu_assign_pointer
796 list_add_rcu
797 list_add_tail_rcu
798 list_del_rcu
799 list_replace_rcu
800 hlist_del_rcu
Paul E. McKenney32300752008-05-12 21:21:05 +0200801 hlist_add_after_rcu
802 hlist_add_before_rcu
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700803 hlist_add_head_rcu
Paul E. McKenney32300752008-05-12 21:21:05 +0200804 hlist_replace_rcu
805 list_splice_init_rcu()
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700806
Paul E. McKenney32300752008-05-12 21:21:05 +0200807RCU: Critical sections Grace period Barrier
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700808
Paul E. McKenney32300752008-05-12 21:21:05 +0200809 rcu_read_lock synchronize_net rcu_barrier
810 rcu_read_unlock synchronize_rcu
Paul E. McKenney240ebbf2009-06-25 09:08:18 -0700811 synchronize_rcu_expedited
Paul E. McKenney32300752008-05-12 21:21:05 +0200812 call_rcu
813
814
815bh: Critical sections Grace period Barrier
816
817 rcu_read_lock_bh call_rcu_bh rcu_barrier_bh
Paul E. McKenney240ebbf2009-06-25 09:08:18 -0700818 rcu_read_unlock_bh synchronize_rcu_bh
819 synchronize_rcu_bh_expedited
Paul E. McKenney32300752008-05-12 21:21:05 +0200820
821
822sched: Critical sections Grace period Barrier
823
Paul E. McKenney240ebbf2009-06-25 09:08:18 -0700824 rcu_read_lock_sched synchronize_sched rcu_barrier_sched
825 rcu_read_unlock_sched call_rcu_sched
826 [preempt_disable] synchronize_sched_expedited
827 [and friends]
Paul E. McKenney32300752008-05-12 21:21:05 +0200828
829
830SRCU: Critical sections Grace period Barrier
831
832 srcu_read_lock synchronize_srcu N/A
833 srcu_read_unlock
834
Paul E. McKenney240ebbf2009-06-25 09:08:18 -0700835SRCU: Initialization/cleanup
836 init_srcu_struct
837 cleanup_srcu_struct
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700838
839See the comment headers in the source code (or the docbook generated
840from them) for more information.
841
842
8438. ANSWERS TO QUICK QUIZZES
844
845Quick Quiz #1: Why is this argument naive? How could a deadlock
846 occur when using this algorithm in a real-world Linux
847 kernel? [Referring to the lock-based "toy" RCU
848 algorithm.]
849
850Answer: Consider the following sequence of events:
851
852 1. CPU 0 acquires some unrelated lock, call it
Paul E. McKenneyd19720a2006-02-01 03:06:42 -0800853 "problematic_lock", disabling irq via
854 spin_lock_irqsave().
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700855
856 2. CPU 1 enters synchronize_rcu(), write-acquiring
857 rcu_gp_mutex.
858
859 3. CPU 0 enters rcu_read_lock(), but must wait
860 because CPU 1 holds rcu_gp_mutex.
861
862 4. CPU 1 is interrupted, and the irq handler
863 attempts to acquire problematic_lock.
864
865 The system is now deadlocked.
866
867 One way to avoid this deadlock is to use an approach like
868 that of CONFIG_PREEMPT_RT, where all normal spinlocks
869 become blocking locks, and all irq handlers execute in
870 the context of special tasks. In this case, in step 4
871 above, the irq handler would block, allowing CPU 1 to
872 release rcu_gp_mutex, avoiding the deadlock.
873
874 Even in the absence of deadlock, this RCU implementation
875 allows latency to "bleed" from readers to other
876 readers through synchronize_rcu(). To see this,
877 consider task A in an RCU read-side critical section
878 (thus read-holding rcu_gp_mutex), task B blocked
879 attempting to write-acquire rcu_gp_mutex, and
880 task C blocked in rcu_read_lock() attempting to
881 read_acquire rcu_gp_mutex. Task A's RCU read-side
882 latency is holding up task C, albeit indirectly via
883 task B.
884
885 Realtime RCU implementations therefore use a counter-based
886 approach where tasks in RCU read-side critical sections
887 cannot be blocked by tasks executing synchronize_rcu().
888
889Quick Quiz #2: Give an example where Classic RCU's read-side
890 overhead is -negative-.
891
892Answer: Imagine a single-CPU system with a non-CONFIG_PREEMPT
893 kernel where a routing table is used by process-context
894 code, but can be updated by irq-context code (for example,
895 by an "ICMP REDIRECT" packet). The usual way of handling
896 this would be to have the process-context code disable
897 interrupts while searching the routing table. Use of
898 RCU allows such interrupt-disabling to be dispensed with.
899 Thus, without RCU, you pay the cost of disabling interrupts,
900 and with RCU you don't.
901
902 One can argue that the overhead of RCU in this
903 case is negative with respect to the single-CPU
904 interrupt-disabling approach. Others might argue that
905 the overhead of RCU is merely zero, and that replacing
906 the positive overhead of the interrupt-disabling scheme
907 with the zero-overhead RCU scheme does not constitute
908 negative overhead.
909
910 In real life, of course, things are more complex. But
911 even the theoretical possibility of negative overhead for
912 a synchronization primitive is a bit unexpected. ;-)
913
914Quick Quiz #3: If it is illegal to block in an RCU read-side
915 critical section, what the heck do you do in
916 PREEMPT_RT, where normal spinlocks can block???
917
918Answer: Just as PREEMPT_RT permits preemption of spinlock
919 critical sections, it permits preemption of RCU
920 read-side critical sections. It also permits
921 spinlocks blocking while in RCU read-side critical
922 sections.
923
924 Why the apparent inconsistency? Because it is it
925 possible to use priority boosting to keep the RCU
926 grace periods short if need be (for example, if running
927 short of memory). In contrast, if blocking waiting
928 for (say) network reception, there is no way to know
929 what should be boosted. Especially given that the
930 process we need to boost might well be a human being
931 who just went out for a pizza or something. And although
932 a computer-operated cattle prod might arouse serious
933 interest, it might also provoke serious objections.
934 Besides, how does the computer know what pizza parlor
935 the human being went to???
936
937
938ACKNOWLEDGEMENTS
939
940My thanks to the people who helped make this human-readable, including
Paul E. McKenneyd19720a2006-02-01 03:06:42 -0800941Jon Walpole, Josh Triplett, Serge Hallyn, Suzanne Wood, and Alan Stern.
Paul E. McKenneydd81eca2005-09-10 00:26:24 -0700942
943
944For more information, see http://www.rdrop.com/users/paulmck/RCU.