blob: 300da4bdfdbd703c4809bc4b3ec5433dfe634d1e [file] [log] [blame]
Mathieu Desnoyersf1f88102007-02-10 01:46:01 -08001 Semantics and Behavior of Local Atomic Operations
2
3 Mathieu Desnoyers
4
5
6 This document explains the purpose of the local atomic operations, how
7to implement them for any given architecture and shows how they can be used
8properly. It also stresses on the precautions that must be taken when reading
9those local variables across CPUs when the order of memory writes matters.
10
11
12
13* Purpose of local atomic operations
14
15Local atomic operations are meant to provide fast and highly reentrant per CPU
16counters. They minimize the performance cost of standard atomic operations by
17removing the LOCK prefix and memory barriers normally required to synchronize
18across CPUs.
19
20Having fast per CPU atomic counters is interesting in many cases : it does not
21require disabling interrupts to protect from interrupt handlers and it permits
22coherent counters in NMI handlers. It is especially useful for tracing purposes
23and for various performance monitoring counters.
24
25Local atomic operations only guarantee variable modification atomicity wrt the
26CPU which owns the data. Therefore, care must taken to make sure that only one
27CPU writes to the local_t data. This is done by using per cpu data and making
28sure that we modify it from within a preemption safe context. It is however
29permitted to read local_t data from any CPU : it will then appear to be written
Mathieu Desnoyers0e1ccb92007-10-16 23:29:29 -070030out of order wrt other memory writes by the owner CPU.
Mathieu Desnoyersf1f88102007-02-10 01:46:01 -080031
32
33* Implementation for a given architecture
34
35It can be done by slightly modifying the standard atomic operations : only
36their UP variant must be kept. It typically means removing LOCK prefix (on
Matt LaPlante19f59462009-04-27 15:06:31 +020037i386 and x86_64) and any SMP synchronization barrier. If the architecture does
Mathieu Desnoyersf1f88102007-02-10 01:46:01 -080038not have a different behavior between SMP and UP, including asm-generic/local.h
Matt LaPlanted9195882008-07-25 19:45:33 -070039in your architecture's local.h is sufficient.
Mathieu Desnoyersf1f88102007-02-10 01:46:01 -080040
41The local_t type is defined as an opaque signed long by embedding an
42atomic_long_t inside a structure. This is made so a cast from this type to a
43long fails. The definition looks like :
44
45typedef struct { atomic_long_t a; } local_t;
46
47
Mathieu Desnoyers74beb9d2007-10-16 23:29:28 -070048* Rules to follow when using local atomic operations
49
50- Variables touched by local ops must be per cpu variables.
51- _Only_ the CPU owner of these variables must write to them.
52- This CPU can use local ops from any context (process, irq, softirq, nmi, ...)
53 to update its local_t variables.
54- Preemption (or interrupts) must be disabled when using local ops in
55 process context to make sure the process won't be migrated to a
56 different CPU between getting the per-cpu variable and doing the
57 actual local op.
58- When using local ops in interrupt context, no special care must be
59 taken on a mainline kernel, since they will run on the local CPU with
60 preemption already disabled. I suggest, however, to explicitly
61 disable preemption anyway to make sure it will still work correctly on
62 -rt kernels.
63- Reading the local cpu variable will provide the current copy of the
64 variable.
65- Reads of these variables can be done from any CPU, because updates to
66 "long", aligned, variables are always atomic. Since no memory
67 synchronization is done by the writer CPU, an outdated copy of the
68 variable can be read when reading some _other_ cpu's variables.
69
70
Mathieu Desnoyersf1f88102007-02-10 01:46:01 -080071* How to use local atomic operations
72
73#include <linux/percpu.h>
74#include <asm/local.h>
75
76static DEFINE_PER_CPU(local_t, counters) = LOCAL_INIT(0);
77
78
79* Counting
80
81Counting is done on all the bits of a signed long.
82
83In preemptible context, use get_cpu_var() and put_cpu_var() around local atomic
84operations : it makes sure that preemption is disabled around write access to
85the per cpu variable. For instance :
86
87 local_inc(&get_cpu_var(counters));
88 put_cpu_var(counters);
89
90If you are already in a preemption-safe context, you can directly use
91__get_cpu_var() instead.
92
93 local_inc(&__get_cpu_var(counters));
94
95
96
97* Reading the counters
98
99Those local counters can be read from foreign CPUs to sum the count. Note that
100the data seen by local_read across CPUs must be considered to be out of order
101relatively to other memory writes happening on the CPU that owns the data.
102
103 long sum = 0;
104 for_each_online_cpu(cpu)
105 sum += local_read(&per_cpu(counters, cpu));
106
107If you want to use a remote local_read to synchronize access to a resource
108between CPUs, explicit smp_wmb() and smp_rmb() memory barriers must be used
109respectively on the writer and the reader CPUs. It would be the case if you use
110the local_t variable as a counter of bytes written in a buffer : there should
111be a smp_wmb() between the buffer write and the counter increment and also a
112smp_rmb() between the counter read and the buffer read.
113
114
115Here is a sample module which implements a basic per cpu counter using local.h.
116
117--- BEGIN ---
118/* test-local.c
119 *
120 * Sample module for local.h usage.
121 */
122
123
124#include <asm/local.h>
125#include <linux/module.h>
126#include <linux/timer.h>
127
128static DEFINE_PER_CPU(local_t, counters) = LOCAL_INIT(0);
129
130static struct timer_list test_timer;
131
132/* IPI called on each CPU. */
133static void test_each(void *info)
134{
135 /* Increment the counter from a non preemptible context */
136 printk("Increment on cpu %d\n", smp_processor_id());
137 local_inc(&__get_cpu_var(counters));
138
139 /* This is what incrementing the variable would look like within a
140 * preemptible context (it disables preemption) :
141 *
142 * local_inc(&get_cpu_var(counters));
143 * put_cpu_var(counters);
144 */
145}
146
147static void do_test_timer(unsigned long data)
148{
149 int cpu;
150
151 /* Increment the counters */
Mathieu Desnoyers02d43b12008-12-01 05:46:38 -0500152 on_each_cpu(test_each, NULL, 1);
Mathieu Desnoyersf1f88102007-02-10 01:46:01 -0800153 /* Read all the counters */
154 printk("Counters read from CPU %d\n", smp_processor_id());
155 for_each_online_cpu(cpu) {
156 printk("Read : CPU %d, count %ld\n", cpu,
157 local_read(&per_cpu(counters, cpu)));
158 }
159 del_timer(&test_timer);
160 test_timer.expires = jiffies + 1000;
161 add_timer(&test_timer);
162}
163
164static int __init test_init(void)
165{
166 /* initialize the timer that will increment the counter */
167 init_timer(&test_timer);
168 test_timer.function = do_test_timer;
169 test_timer.expires = jiffies + 1;
170 add_timer(&test_timer);
171
172 return 0;
173}
174
175static void __exit test_exit(void)
176{
177 del_timer_sync(&test_timer);
178}
179
180module_init(test_init);
181module_exit(test_exit);
182
183MODULE_LICENSE("GPL");
184MODULE_AUTHOR("Mathieu Desnoyers");
185MODULE_DESCRIPTION("Local Atomic Ops");
186--- END ---