Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 1 | ======================================================= |
| 2 | Semantics and Behavior of Atomic and Bitmask Operations |
| 3 | ======================================================= |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 4 | |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 5 | :Author: David S. Miller |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 6 | |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 7 | This document is intended to serve as a guide to Linux port |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 8 | maintainers on how to implement atomic counter, bitops, and spinlock |
| 9 | interfaces properly. |
| 10 | |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 11 | Atomic Type And Operations |
| 12 | ========================== |
| 13 | |
| 14 | The atomic_t type should be defined as a signed integer and |
Paul E. McKenney | 1f7870d | 2014-10-19 12:05:22 -0700 | [diff] [blame] | 15 | the atomic_long_t type as a signed long integer. Also, they should |
| 16 | be made opaque such that any kind of cast to a normal C integer type |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 17 | will fail. Something like the following should suffice:: |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 18 | |
Nikanth Karthikesan | 72eef0f | 2011-05-26 16:25:13 -0700 | [diff] [blame] | 19 | typedef struct { int counter; } atomic_t; |
Paul E. McKenney | 1f7870d | 2014-10-19 12:05:22 -0700 | [diff] [blame] | 20 | typedef struct { long counter; } atomic_long_t; |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 21 | |
Matti Linnanvuori | 8d7b52d | 2007-10-16 23:30:08 -0700 | [diff] [blame] | 22 | Historically, counter has been declared volatile. This is now discouraged. |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 23 | See :ref:`Documentation/process/volatile-considered-harmful.rst |
| 24 | <volatile_considered_harmful>` for the complete rationale. |
Matti Linnanvuori | 8d7b52d | 2007-10-16 23:30:08 -0700 | [diff] [blame] | 25 | |
Grant Grundler | 1a2142b | 2007-10-16 23:29:28 -0700 | [diff] [blame] | 26 | local_t is very similar to atomic_t. If the counter is per CPU and only |
| 27 | updated by one CPU, local_t is probably more appropriate. Please see |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 28 | :ref:`Documentation/core-api/local_ops.rst <local_ops>` for the semantics of |
| 29 | local_t. |
Grant Grundler | 1a2142b | 2007-10-16 23:29:28 -0700 | [diff] [blame] | 30 | |
Matti Linnanvuori | 8d7b52d | 2007-10-16 23:30:08 -0700 | [diff] [blame] | 31 | The first operations to implement for atomic_t's are the initializers and |
Jonathan Neuschäfer | c4081a4 | 2018-07-16 11:05:59 -0700 | [diff] [blame] | 32 | plain writes. :: |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 33 | |
| 34 | #define ATOMIC_INIT(i) { (i) } |
| 35 | #define atomic_set(v, i) ((v)->counter = (i)) |
| 36 | |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 37 | The first macro is used in definitions, such as:: |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 38 | |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 39 | static atomic_t my_counter = ATOMIC_INIT(1); |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 40 | |
Matti Linnanvuori | 8d7b52d | 2007-10-16 23:30:08 -0700 | [diff] [blame] | 41 | The initializer is atomic in that the return values of the atomic operations |
| 42 | are guaranteed to be correct reflecting the initialized value if the |
| 43 | initializer is used before runtime. If the initializer is used at runtime, a |
| 44 | proper implicit or explicit read memory barrier is needed before reading the |
| 45 | value with atomic_read from another thread. |
| 46 | |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 47 | As with all of the ``atomic_`` interfaces, replace the leading ``atomic_`` |
| 48 | with ``atomic_long_`` to operate on atomic_long_t. |
Paul E. McKenney | 1f7870d | 2014-10-19 12:05:22 -0700 | [diff] [blame] | 49 | |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 50 | The second interface can be used at runtime, as in:: |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 51 | |
| 52 | struct foo { atomic_t counter; }; |
| 53 | ... |
| 54 | |
| 55 | struct foo *k; |
| 56 | |
| 57 | k = kmalloc(sizeof(*k), GFP_KERNEL); |
| 58 | if (!k) |
| 59 | return -ENOMEM; |
| 60 | atomic_set(&k->counter, 0); |
| 61 | |
Matti Linnanvuori | 8d7b52d | 2007-10-16 23:30:08 -0700 | [diff] [blame] | 62 | The setting is atomic in that the return values of the atomic operations by |
| 63 | all threads are guaranteed to be correct reflecting either the value that has |
| 64 | been set with this operation or set with another operation. A proper implicit |
| 65 | or explicit memory barrier is needed before the value set with the operation |
| 66 | is guaranteed to be readable with atomic_read from another thread. |
| 67 | |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 68 | Next, we have:: |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 69 | |
| 70 | #define atomic_read(v) ((v)->counter) |
| 71 | |
Matti Linnanvuori | 8d7b52d | 2007-10-16 23:30:08 -0700 | [diff] [blame] | 72 | which simply reads the counter value currently visible to the calling thread. |
| 73 | The read is atomic in that the return value is guaranteed to be one of the |
| 74 | values initialized or modified with the interface operations if a proper |
| 75 | implicit or explicit memory barrier is used after possible runtime |
| 76 | initialization by any other thread and the value is modified only with the |
| 77 | interface operations. atomic_read does not guarantee that the runtime |
| 78 | initialization by any other thread is visible yet, so the user of the |
| 79 | interface must take care of that with a proper implicit or explicit memory |
| 80 | barrier. |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 81 | |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 82 | .. warning:: |
Matti Linnanvuori | 8d7b52d | 2007-10-16 23:30:08 -0700 | [diff] [blame] | 83 | |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 84 | ``atomic_read()`` and ``atomic_set()`` DO NOT IMPLY BARRIERS! |
Matti Linnanvuori | 8d7b52d | 2007-10-16 23:30:08 -0700 | [diff] [blame] | 85 | |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 86 | Some architectures may choose to use the volatile keyword, barriers, or |
| 87 | inline assembly to guarantee some degree of immediacy for atomic_read() |
| 88 | and atomic_set(). This is not uniformly guaranteed, and may change in |
| 89 | the future, so all users of atomic_t should treat atomic_read() and |
| 90 | atomic_set() as simple C statements that may be reordered or optimized |
| 91 | away entirely by the compiler or processor, and explicitly invoke the |
| 92 | appropriate compiler and/or memory barrier for each use case. Failure |
| 93 | to do so will result in code that may suddenly break when used with |
| 94 | different architectures or compiler optimizations, or even changes in |
| 95 | unrelated code which changes how the compiler optimizes the section |
| 96 | accessing atomic_t variables. |
Matti Linnanvuori | 8d7b52d | 2007-10-16 23:30:08 -0700 | [diff] [blame] | 97 | |
Paul E. McKenney | 182dd4b | 2011-11-22 10:55:12 -0800 | [diff] [blame] | 98 | Properly aligned pointers, longs, ints, and chars (and unsigned |
| 99 | equivalents) may be atomically loaded from and stored to in the same |
Mark Rutland | 47f4212 | 2016-11-16 11:13:59 +0000 | [diff] [blame] | 100 | sense as described for atomic_read() and atomic_set(). The READ_ONCE() |
| 101 | and WRITE_ONCE() macros should be used to prevent the compiler from using |
| 102 | optimizations that might otherwise optimize accesses out of existence on |
| 103 | the one hand, or that might create unsolicited accesses on the other. |
Paul E. McKenney | 182dd4b | 2011-11-22 10:55:12 -0800 | [diff] [blame] | 104 | |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 105 | For example consider the following code:: |
Paul E. McKenney | 182dd4b | 2011-11-22 10:55:12 -0800 | [diff] [blame] | 106 | |
| 107 | while (a > 0) |
| 108 | do_something(); |
| 109 | |
| 110 | If the compiler can prove that do_something() does not store to the |
| 111 | variable a, then the compiler is within its rights transforming this to |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 112 | the following:: |
Paul E. McKenney | 182dd4b | 2011-11-22 10:55:12 -0800 | [diff] [blame] | 113 | |
Paul E. McKenney | 182dd4b | 2011-11-22 10:55:12 -0800 | [diff] [blame] | 114 | if (a > 0) |
| 115 | for (;;) |
| 116 | do_something(); |
| 117 | |
| 118 | If you don't want the compiler to do this (and you probably don't), then |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 119 | you should use something like the following:: |
Paul E. McKenney | 182dd4b | 2011-11-22 10:55:12 -0800 | [diff] [blame] | 120 | |
SeongJae Park | fc7bdc9 | 2018-05-14 15:55:32 -0700 | [diff] [blame] | 121 | while (READ_ONCE(a) > 0) |
Paul E. McKenney | 182dd4b | 2011-11-22 10:55:12 -0800 | [diff] [blame] | 122 | do_something(); |
| 123 | |
| 124 | Alternatively, you could place a barrier() call in the loop. |
| 125 | |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 126 | For another example, consider the following code:: |
Paul E. McKenney | 182dd4b | 2011-11-22 10:55:12 -0800 | [diff] [blame] | 127 | |
| 128 | tmp_a = a; |
| 129 | do_something_with(tmp_a); |
| 130 | do_something_else_with(tmp_a); |
| 131 | |
| 132 | If the compiler can prove that do_something_with() does not store to the |
| 133 | variable a, then the compiler is within its rights to manufacture an |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 134 | additional load as follows:: |
Paul E. McKenney | 182dd4b | 2011-11-22 10:55:12 -0800 | [diff] [blame] | 135 | |
| 136 | tmp_a = a; |
| 137 | do_something_with(tmp_a); |
| 138 | tmp_a = a; |
| 139 | do_something_else_with(tmp_a); |
| 140 | |
| 141 | This could fatally confuse your code if it expected the same value |
| 142 | to be passed to do_something_with() and do_something_else_with(). |
| 143 | |
| 144 | The compiler would be likely to manufacture this additional load if |
| 145 | do_something_with() was an inline function that made very heavy use |
| 146 | of registers: reloading from variable a could save a flush to the |
| 147 | stack and later reload. To prevent the compiler from attacking your |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 148 | code in this manner, write the following:: |
Paul E. McKenney | 182dd4b | 2011-11-22 10:55:12 -0800 | [diff] [blame] | 149 | |
Mark Rutland | 47f4212 | 2016-11-16 11:13:59 +0000 | [diff] [blame] | 150 | tmp_a = READ_ONCE(a); |
Paul E. McKenney | 182dd4b | 2011-11-22 10:55:12 -0800 | [diff] [blame] | 151 | do_something_with(tmp_a); |
| 152 | do_something_else_with(tmp_a); |
| 153 | |
| 154 | For a final example, consider the following code, assuming that the |
| 155 | variable a is set at boot time before the second CPU is brought online |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 156 | and never changed later, so that memory barriers are not needed:: |
Paul E. McKenney | 182dd4b | 2011-11-22 10:55:12 -0800 | [diff] [blame] | 157 | |
| 158 | if (a) |
| 159 | b = 9; |
| 160 | else |
| 161 | b = 42; |
| 162 | |
| 163 | The compiler is within its rights to manufacture an additional store |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 164 | by transforming the above code into the following:: |
Paul E. McKenney | 182dd4b | 2011-11-22 10:55:12 -0800 | [diff] [blame] | 165 | |
| 166 | b = 42; |
| 167 | if (a) |
| 168 | b = 9; |
| 169 | |
| 170 | This could come as a fatal surprise to other code running concurrently |
| 171 | that expected b to never have the value 42 if a was zero. To prevent |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 172 | the compiler from doing this, write something like:: |
Paul E. McKenney | 182dd4b | 2011-11-22 10:55:12 -0800 | [diff] [blame] | 173 | |
| 174 | if (a) |
Mark Rutland | 47f4212 | 2016-11-16 11:13:59 +0000 | [diff] [blame] | 175 | WRITE_ONCE(b, 9); |
Paul E. McKenney | 182dd4b | 2011-11-22 10:55:12 -0800 | [diff] [blame] | 176 | else |
Mark Rutland | 47f4212 | 2016-11-16 11:13:59 +0000 | [diff] [blame] | 177 | WRITE_ONCE(b, 42); |
Paul E. McKenney | 182dd4b | 2011-11-22 10:55:12 -0800 | [diff] [blame] | 178 | |
| 179 | Don't even -think- about doing this without proper use of memory barriers, |
| 180 | locks, or atomic operations if variable a can change at runtime! |
| 181 | |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 182 | .. warning:: |
| 183 | |
| 184 | ``READ_ONCE()`` OR ``WRITE_ONCE()`` DO NOT IMPLY A BARRIER! |
Paul E. McKenney | 182dd4b | 2011-11-22 10:55:12 -0800 | [diff] [blame] | 185 | |
Matti Linnanvuori | 8d7b52d | 2007-10-16 23:30:08 -0700 | [diff] [blame] | 186 | Now, we move onto the atomic operation interfaces typically implemented with |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 187 | the help of assembly code. :: |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 188 | |
| 189 | void atomic_add(int i, atomic_t *v); |
| 190 | void atomic_sub(int i, atomic_t *v); |
| 191 | void atomic_inc(atomic_t *v); |
| 192 | void atomic_dec(atomic_t *v); |
| 193 | |
| 194 | These four routines add and subtract integral values to/from the given |
| 195 | atomic_t value. The first two routines pass explicit integers by |
| 196 | which to make the adjustment, whereas the latter two use an implicit |
| 197 | adjustment value of "1". |
| 198 | |
| 199 | One very important aspect of these two routines is that they DO NOT |
| 200 | require any explicit memory barriers. They need only perform the |
| 201 | atomic_t counter update in an SMP safe manner. |
| 202 | |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 203 | Next, we have:: |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 204 | |
| 205 | int atomic_inc_return(atomic_t *v); |
| 206 | int atomic_dec_return(atomic_t *v); |
| 207 | |
| 208 | These routines add 1 and subtract 1, respectively, from the given |
| 209 | atomic_t and return the new counter value after the operation is |
| 210 | performed. |
| 211 | |
Paul E. McKenney | daf1aab | 2015-02-02 08:08:25 -0800 | [diff] [blame] | 212 | Unlike the above routines, it is required that these primitives |
| 213 | include explicit memory barriers that are performed before and after |
| 214 | the operation. It must be done such that all memory operations before |
| 215 | and after the atomic operation calls are strongly ordered with respect |
| 216 | to the atomic operation itself. |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 217 | |
| 218 | For example, it should behave as if a smp_mb() call existed both |
| 219 | before and after the atomic operation. |
| 220 | |
| 221 | If the atomic instructions used in an implementation provide explicit |
| 222 | memory barrier semantics which satisfy the above requirements, that is |
| 223 | fine as well. |
| 224 | |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 225 | Let's move on:: |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 226 | |
| 227 | int atomic_add_return(int i, atomic_t *v); |
| 228 | int atomic_sub_return(int i, atomic_t *v); |
| 229 | |
| 230 | These behave just like atomic_{inc,dec}_return() except that an |
| 231 | explicit counter adjustment is given instead of the implicit "1". |
| 232 | This means that like atomic_{inc,dec}_return(), the memory barrier |
| 233 | semantics are required. |
| 234 | |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 235 | Next:: |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 236 | |
| 237 | int atomic_inc_and_test(atomic_t *v); |
| 238 | int atomic_dec_and_test(atomic_t *v); |
| 239 | |
| 240 | These two routines increment and decrement by 1, respectively, the |
| 241 | given atomic counter. They return a boolean indicating whether the |
| 242 | resulting counter value was zero or not. |
| 243 | |
Paul E. McKenney | daf1aab | 2015-02-02 08:08:25 -0800 | [diff] [blame] | 244 | Again, these primitives provide explicit memory barrier semantics around |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 245 | the atomic operation:: |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 246 | |
| 247 | int atomic_sub_and_test(int i, atomic_t *v); |
| 248 | |
| 249 | This is identical to atomic_dec_and_test() except that an explicit |
Paul E. McKenney | daf1aab | 2015-02-02 08:08:25 -0800 | [diff] [blame] | 250 | decrement is given instead of the implicit "1". This primitive must |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 251 | provide explicit memory barrier semantics around the operation:: |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 252 | |
| 253 | int atomic_add_negative(int i, atomic_t *v); |
| 254 | |
Paul E. McKenney | daf1aab | 2015-02-02 08:08:25 -0800 | [diff] [blame] | 255 | The given increment is added to the given atomic counter value. A boolean |
| 256 | is return which indicates whether the resulting counter value is negative. |
| 257 | This primitive must provide explicit memory barrier semantics around |
| 258 | the operation. |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 259 | |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 260 | Then:: |
Nick Piggin | 4a6dae6 | 2005-11-13 16:07:24 -0800 | [diff] [blame] | 261 | |
Matti Linnanvuori | 8d7b52d | 2007-10-16 23:30:08 -0700 | [diff] [blame] | 262 | int atomic_xchg(atomic_t *v, int new); |
| 263 | |
| 264 | This performs an atomic exchange operation on the atomic variable v, setting |
| 265 | the given new value. It returns the old value that the atomic variable v had |
| 266 | just before the operation. |
| 267 | |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 268 | atomic_xchg must provide explicit memory barriers around the operation. :: |
Richard Braun | 7e8b1e7 | 2012-12-13 11:07:32 +0100 | [diff] [blame] | 269 | |
Nick Piggin | 4a6dae6 | 2005-11-13 16:07:24 -0800 | [diff] [blame] | 270 | int atomic_cmpxchg(atomic_t *v, int old, int new); |
| 271 | |
| 272 | This performs an atomic compare exchange operation on the atomic value v, |
| 273 | with the given old and new values. Like all atomic_xxx operations, |
| 274 | atomic_cmpxchg will only satisfy its atomicity semantics as long as all |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 275 | other accesses of \*v are performed through atomic_xxx operations. |
Nick Piggin | 4a6dae6 | 2005-11-13 16:07:24 -0800 | [diff] [blame] | 276 | |
Will Deacon | ed2de9f | 2015-07-16 16:10:06 +0100 | [diff] [blame] | 277 | atomic_cmpxchg must provide explicit memory barriers around the operation, |
| 278 | although if the comparison fails then no memory ordering guarantees are |
| 279 | required. |
Nick Piggin | 4a6dae6 | 2005-11-13 16:07:24 -0800 | [diff] [blame] | 280 | |
| 281 | The semantics for atomic_cmpxchg are the same as those defined for 'cas' |
| 282 | below. |
| 283 | |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 284 | Finally:: |
Nick Piggin | 8426e1f | 2005-11-13 16:07:25 -0800 | [diff] [blame] | 285 | |
| 286 | int atomic_add_unless(atomic_t *v, int a, int u); |
| 287 | |
| 288 | If the atomic value v is not equal to u, this function adds a to v, and |
| 289 | returns non zero. If v is equal to u then it returns zero. This is done as |
| 290 | an atomic operation. |
| 291 | |
Paul E. McKenney | daf1aab | 2015-02-02 08:08:25 -0800 | [diff] [blame] | 292 | atomic_add_unless must provide explicit memory barriers around the |
| 293 | operation unless it fails (returns 0). |
Nick Piggin | 8426e1f | 2005-11-13 16:07:25 -0800 | [diff] [blame] | 294 | |
| 295 | atomic_inc_not_zero, equivalent to atomic_add_unless(v, 1, 0) |
| 296 | |
Nick Piggin | 4a6dae6 | 2005-11-13 16:07:24 -0800 | [diff] [blame] | 297 | |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 298 | If a caller requires memory barrier semantics around an atomic_t |
| 299 | operation which does not return a value, a set of interfaces are |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 300 | defined which accomplish this:: |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 301 | |
Peter Zijlstra | 1b15611 | 2014-03-13 19:00:35 +0100 | [diff] [blame] | 302 | void smp_mb__before_atomic(void); |
| 303 | void smp_mb__after_atomic(void); |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 304 | |
Paul E. McKenney | 79269ee | 2017-04-19 14:30:37 -0700 | [diff] [blame] | 305 | Preceding a non-value-returning read-modify-write atomic operation with |
| 306 | smp_mb__before_atomic() and following it with smp_mb__after_atomic() |
| 307 | provides the same full ordering that is provided by value-returning |
| 308 | read-modify-write atomic operations. |
| 309 | |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 310 | For example, smp_mb__before_atomic() can be used like so:: |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 311 | |
| 312 | obj->dead = 1; |
Peter Zijlstra | 1b15611 | 2014-03-13 19:00:35 +0100 | [diff] [blame] | 313 | smp_mb__before_atomic(); |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 314 | atomic_dec(&obj->ref_count); |
| 315 | |
Michael Hayes | a0ebb3f | 2006-06-26 18:27:35 +0200 | [diff] [blame] | 316 | It makes sure that all memory operations preceding the atomic_dec() |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 317 | call are strongly ordered with respect to the atomic counter |
Michael Hayes | a0ebb3f | 2006-06-26 18:27:35 +0200 | [diff] [blame] | 318 | operation. In the above example, it guarantees that the assignment of |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 319 | "1" to obj->dead will be globally visible to other cpus before the |
| 320 | atomic counter decrement. |
| 321 | |
Peter Zijlstra | 1b15611 | 2014-03-13 19:00:35 +0100 | [diff] [blame] | 322 | Without the explicit smp_mb__before_atomic() call, the |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 323 | implementation could legally allow the atomic counter update visible |
| 324 | to other cpus before the "obj->dead = 1;" assignment. |
| 325 | |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 326 | A missing memory barrier in the cases where they are required by the |
Michael Hayes | a0ebb3f | 2006-06-26 18:27:35 +0200 | [diff] [blame] | 327 | atomic_t implementation above can have disastrous results. Here is |
| 328 | an example, which follows a pattern occurring frequently in the Linux |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 329 | kernel. It is the use of atomic counters to implement reference |
| 330 | counting, and it works such that once the counter falls to zero it can |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 331 | be guaranteed that no other entity can be accessing the object:: |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 332 | |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 333 | static void obj_list_add(struct obj *obj, struct list_head *head) |
| 334 | { |
| 335 | obj->active = 1; |
| 336 | list_add(&obj->list, head); |
| 337 | } |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 338 | |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 339 | static void obj_list_del(struct obj *obj) |
| 340 | { |
| 341 | list_del(&obj->list); |
| 342 | obj->active = 0; |
| 343 | } |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 344 | |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 345 | static void obj_destroy(struct obj *obj) |
| 346 | { |
| 347 | BUG_ON(obj->active); |
| 348 | kfree(obj); |
| 349 | } |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 350 | |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 351 | struct obj *obj_list_peek(struct list_head *head) |
| 352 | { |
| 353 | if (!list_empty(head)) { |
| 354 | struct obj *obj; |
| 355 | |
| 356 | obj = list_entry(head->next, struct obj, list); |
| 357 | atomic_inc(&obj->refcnt); |
| 358 | return obj; |
| 359 | } |
| 360 | return NULL; |
| 361 | } |
| 362 | |
| 363 | void obj_poke(void) |
| 364 | { |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 365 | struct obj *obj; |
| 366 | |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 367 | spin_lock(&global_list_lock); |
| 368 | obj = obj_list_peek(&global_list); |
| 369 | spin_unlock(&global_list_lock); |
| 370 | |
| 371 | if (obj) { |
| 372 | obj->ops->poke(obj); |
| 373 | if (atomic_dec_and_test(&obj->refcnt)) |
| 374 | obj_destroy(obj); |
| 375 | } |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 376 | } |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 377 | |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 378 | void obj_timeout(struct obj *obj) |
| 379 | { |
| 380 | spin_lock(&global_list_lock); |
| 381 | obj_list_del(obj); |
| 382 | spin_unlock(&global_list_lock); |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 383 | |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 384 | if (atomic_dec_and_test(&obj->refcnt)) |
| 385 | obj_destroy(obj); |
| 386 | } |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 387 | |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 388 | .. note:: |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 389 | |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 390 | This is a simplification of the ARP queue management in the generic |
| 391 | neighbour discover code of the networking. Olaf Kirch found a bug wrt. |
| 392 | memory barriers in kfree_skb() that exposed the atomic_t memory barrier |
| 393 | requirements quite clearly. |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 394 | |
| 395 | Given the above scheme, it must be the case that the obj->active |
| 396 | update done by the obj list deletion be visible to other processors |
| 397 | before the atomic counter decrement is performed. |
| 398 | |
| 399 | Otherwise, the counter could fall to zero, yet obj->active would still |
| 400 | be set, thus triggering the assertion in obj_destroy(). The error |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 401 | sequence looks like this:: |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 402 | |
| 403 | cpu 0 cpu 1 |
| 404 | obj_poke() obj_timeout() |
| 405 | obj = obj_list_peek(); |
| 406 | ... gains ref to obj, refcnt=2 |
| 407 | obj_list_del(obj); |
| 408 | obj->active = 0 ... |
| 409 | ... visibility delayed ... |
| 410 | atomic_dec_and_test() |
| 411 | ... refcnt drops to 1 ... |
| 412 | atomic_dec_and_test() |
| 413 | ... refcount drops to 0 ... |
| 414 | obj_destroy() |
| 415 | BUG() triggers since obj->active |
| 416 | still seen as one |
| 417 | obj->active update visibility occurs |
| 418 | |
| 419 | With the memory barrier semantics required of the atomic_t operations |
| 420 | which return values, the above sequence of memory visibility can never |
| 421 | happen. Specifically, in the above case the atomic_dec_and_test() |
| 422 | counter decrement would not become globally visible until the |
| 423 | obj->active update does. |
| 424 | |
| 425 | As a historical note, 32-bit Sparc used to only allow usage of |
Francis Galiegue | a33f322 | 2010-04-23 00:08:02 +0200 | [diff] [blame] | 426 | 24-bits of its atomic_t type. This was because it used 8 bits |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 427 | as a spinlock for SMP safety. Sparc32 lacked a "compare and swap" |
| 428 | type instruction. However, 32-bit Sparc has since been moved over |
| 429 | to a "hash table of spinlocks" scheme, that allows the full 32-bit |
| 430 | counter to be realized. Essentially, an array of spinlocks are |
| 431 | indexed into based upon the address of the atomic_t being operated |
| 432 | on, and that lock protects the atomic operation. Parisc uses the |
| 433 | same scheme. |
| 434 | |
| 435 | Another note is that the atomic_t operations returning values are |
| 436 | extremely slow on an old 386. |
| 437 | |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 438 | |
| 439 | Atomic Bitmask |
| 440 | ============== |
| 441 | |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 442 | We will now cover the atomic bitmask operations. You will find that |
| 443 | their SMP and memory barrier semantics are similar in shape and scope |
| 444 | to the atomic_t ops above. |
| 445 | |
| 446 | Native atomic bit operations are defined to operate on objects aligned |
| 447 | to the size of an "unsigned long" C data type, and are least of that |
| 448 | size. The endianness of the bits within each "unsigned long" are the |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 449 | native endianness of the cpu. :: |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 450 | |
Michael Hayes | a0ebb3f | 2006-06-26 18:27:35 +0200 | [diff] [blame] | 451 | void set_bit(unsigned long nr, volatile unsigned long *addr); |
| 452 | void clear_bit(unsigned long nr, volatile unsigned long *addr); |
| 453 | void change_bit(unsigned long nr, volatile unsigned long *addr); |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 454 | |
| 455 | These routines set, clear, and change, respectively, the bit number |
| 456 | indicated by "nr" on the bit mask pointed to by "ADDR". |
| 457 | |
| 458 | They must execute atomically, yet there are no implicit memory barrier |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 459 | semantics required of these interfaces. :: |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 460 | |
Michael Hayes | a0ebb3f | 2006-06-26 18:27:35 +0200 | [diff] [blame] | 461 | int test_and_set_bit(unsigned long nr, volatile unsigned long *addr); |
| 462 | int test_and_clear_bit(unsigned long nr, volatile unsigned long *addr); |
| 463 | int test_and_change_bit(unsigned long nr, volatile unsigned long *addr); |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 464 | |
| 465 | Like the above, except that these routines return a boolean which |
| 466 | indicates whether the changed bit was set _BEFORE_ the atomic bit |
| 467 | operation. |
| 468 | |
SeongJae Park | 173af26 | 2018-05-14 15:55:33 -0700 | [diff] [blame] | 469 | |
| 470 | .. warning:: |
| 471 | It is incredibly important that the value be a boolean, ie. "0" or "1". |
| 472 | Do not try to be fancy and save a few instructions by declaring the |
| 473 | above to return "long" and just returning something like "old_val & |
| 474 | mask" because that will not work. |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 475 | |
| 476 | For one thing, this return value gets truncated to int in many code |
| 477 | paths using these interfaces, so on 64-bit if the bit is set in the |
| 478 | upper 32-bits then testers will never see that. |
| 479 | |
| 480 | One great example of where this problem crops up are the thread_info |
| 481 | flag operations. Routines such as test_and_set_ti_thread_flag() chop |
| 482 | the return value into an int. There are other places where things |
| 483 | like this occur as well. |
| 484 | |
| 485 | These routines, like the atomic_t counter operations returning values, |
Paul E. McKenney | daf1aab | 2015-02-02 08:08:25 -0800 | [diff] [blame] | 486 | must provide explicit memory barrier semantics around their execution. |
| 487 | All memory operations before the atomic bit operation call must be |
| 488 | made visible globally before the atomic bit operation is made visible. |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 489 | Likewise, the atomic bit operation must be visible globally before any |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 490 | subsequent memory operation is made visible. For example:: |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 491 | |
| 492 | obj->dead = 1; |
| 493 | if (test_and_set_bit(0, &obj->flags)) |
| 494 | /* ... */; |
| 495 | obj->killed = 1; |
| 496 | |
Michael Hayes | a0ebb3f | 2006-06-26 18:27:35 +0200 | [diff] [blame] | 497 | The implementation of test_and_set_bit() must guarantee that |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 498 | "obj->dead = 1;" is visible to cpus before the atomic memory operation |
| 499 | done by test_and_set_bit() becomes visible. Likewise, the atomic |
| 500 | memory operation done by test_and_set_bit() must become visible before |
| 501 | "obj->killed = 1;" is visible. |
| 502 | |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 503 | Finally there is the basic operation:: |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 504 | |
| 505 | int test_bit(unsigned long nr, __const__ volatile unsigned long *addr); |
| 506 | |
| 507 | Which returns a boolean indicating if bit "nr" is set in the bitmask |
| 508 | pointed to by "addr". |
| 509 | |
Peter Zijlstra | 1b15611 | 2014-03-13 19:00:35 +0100 | [diff] [blame] | 510 | If explicit memory barriers are required around {set,clear}_bit() (which do |
| 511 | not return a value, and thus does not need to provide memory barrier |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 512 | semantics), two interfaces are provided:: |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 513 | |
Peter Zijlstra | 1b15611 | 2014-03-13 19:00:35 +0100 | [diff] [blame] | 514 | void smp_mb__before_atomic(void); |
| 515 | void smp_mb__after_atomic(void); |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 516 | |
| 517 | They are used as follows, and are akin to their atomic_t operation |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 518 | brothers:: |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 519 | |
| 520 | /* All memory operations before this call will |
| 521 | * be globally visible before the clear_bit(). |
| 522 | */ |
Peter Zijlstra | 1b15611 | 2014-03-13 19:00:35 +0100 | [diff] [blame] | 523 | smp_mb__before_atomic(); |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 524 | clear_bit( ... ); |
| 525 | |
| 526 | /* The clear_bit() will be visible before all |
| 527 | * subsequent memory operations. |
| 528 | */ |
Peter Zijlstra | 1b15611 | 2014-03-13 19:00:35 +0100 | [diff] [blame] | 529 | smp_mb__after_atomic(); |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 530 | |
Nick Piggin | 2633357 | 2007-10-18 03:06:39 -0700 | [diff] [blame] | 531 | There are two special bitops with lock barrier semantics (acquire/release, |
| 532 | same as spinlocks). These operate in the same way as their non-_lock/unlock |
| 533 | postfixed variants, except that they are to provide acquire/release semantics, |
| 534 | respectively. This means they can be used for bit_spin_trylock and |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 535 | bit_spin_unlock type operations without specifying any more barriers. :: |
Nick Piggin | 2633357 | 2007-10-18 03:06:39 -0700 | [diff] [blame] | 536 | |
| 537 | int test_and_set_bit_lock(unsigned long nr, unsigned long *addr); |
| 538 | void clear_bit_unlock(unsigned long nr, unsigned long *addr); |
| 539 | void __clear_bit_unlock(unsigned long nr, unsigned long *addr); |
| 540 | |
| 541 | The __clear_bit_unlock version is non-atomic, however it still implements |
| 542 | unlock barrier semantics. This can be useful if the lock itself is protecting |
| 543 | the other bits in the word. |
| 544 | |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 545 | Finally, there are non-atomic versions of the bitmask operations |
| 546 | provided. They are used in contexts where some other higher-level SMP |
| 547 | locking scheme is being used to protect the bitmask, and thus less |
| 548 | expensive non-atomic operations may be used in the implementation. |
| 549 | They have names similar to the above bitmask operation interfaces, |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 550 | except that two underscores are prefixed to the interface name. :: |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 551 | |
| 552 | void __set_bit(unsigned long nr, volatile unsigned long *addr); |
| 553 | void __clear_bit(unsigned long nr, volatile unsigned long *addr); |
| 554 | void __change_bit(unsigned long nr, volatile unsigned long *addr); |
| 555 | int __test_and_set_bit(unsigned long nr, volatile unsigned long *addr); |
| 556 | int __test_and_clear_bit(unsigned long nr, volatile unsigned long *addr); |
| 557 | int __test_and_change_bit(unsigned long nr, volatile unsigned long *addr); |
| 558 | |
| 559 | These non-atomic variants also do not require any special memory |
| 560 | barrier semantics. |
| 561 | |
Paul E. McKenney | daf1aab | 2015-02-02 08:08:25 -0800 | [diff] [blame] | 562 | The routines xchg() and cmpxchg() must provide the same exact |
| 563 | memory-barrier semantics as the atomic and bit operations returning |
| 564 | values. |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 565 | |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 566 | .. note:: |
| 567 | |
| 568 | If someone wants to use xchg(), cmpxchg() and their variants, |
| 569 | linux/atomic.h should be included rather than asm/cmpxchg.h, unless the |
| 570 | code is in arch/* and can take care of itself. |
Boqun Feng | 8456799 | 2015-08-26 19:52:46 +0800 | [diff] [blame] | 571 | |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 572 | Spinlocks and rwlocks have memory barrier expectations as well. |
| 573 | The rule to follow is simple: |
| 574 | |
| 575 | 1) When acquiring a lock, the implementation must make it globally |
| 576 | visible before any subsequent memory operation. |
| 577 | |
| 578 | 2) When releasing a lock, the implementation must make it such that |
| 579 | all previous memory operations are globally visible before the |
| 580 | lock release. |
| 581 | |
| 582 | Which finally brings us to _atomic_dec_and_lock(). There is an |
| 583 | architecture-neutral version implemented in lib/dec_and_lock.c, |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 584 | but most platforms will wish to optimize this in assembler. :: |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 585 | |
| 586 | int _atomic_dec_and_lock(atomic_t *atomic, spinlock_t *lock); |
| 587 | |
| 588 | Atomically decrement the given counter, and if will drop to zero |
| 589 | atomically acquire the given spinlock and perform the decrement |
| 590 | of the counter to zero. If it does not drop to zero, do nothing |
| 591 | with the spinlock. |
| 592 | |
| 593 | It is actually pretty simple to get the memory barrier correct. |
| 594 | Simply satisfy the spinlock grab requirements, which is make |
| 595 | sure the spinlock operation is globally visible before any |
| 596 | subsequent memory operation. |
| 597 | |
| 598 | We can demonstrate this operation more clearly if we define |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 599 | an abstract atomic operation:: |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 600 | |
| 601 | long cas(long *mem, long old, long new); |
| 602 | |
| 603 | "cas" stands for "compare and swap". It atomically: |
| 604 | |
| 605 | 1) Compares "old" with the value currently at "mem". |
| 606 | 2) If they are equal, "new" is written to "mem". |
| 607 | 3) Regardless, the current value at "mem" is returned. |
| 608 | |
| 609 | As an example usage, here is what an atomic counter update |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 610 | might look like:: |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 611 | |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 612 | void example_atomic_inc(long *counter) |
| 613 | { |
| 614 | long old, new, ret; |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 615 | |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 616 | while (1) { |
| 617 | old = *counter; |
| 618 | new = old + 1; |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 619 | |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 620 | ret = cas(counter, old, new); |
| 621 | if (ret == old) |
| 622 | break; |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 623 | } |
| 624 | } |
| 625 | |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 626 | Let's use cas() in order to build a pseudo-C atomic_dec_and_lock():: |
| 627 | |
| 628 | int _atomic_dec_and_lock(atomic_t *atomic, spinlock_t *lock) |
| 629 | { |
| 630 | long old, new, ret; |
| 631 | int went_to_zero; |
| 632 | |
| 633 | went_to_zero = 0; |
| 634 | while (1) { |
| 635 | old = atomic_read(atomic); |
| 636 | new = old - 1; |
| 637 | if (new == 0) { |
| 638 | went_to_zero = 1; |
| 639 | spin_lock(lock); |
| 640 | } |
| 641 | ret = cas(atomic, old, new); |
| 642 | if (ret == old) |
| 643 | break; |
| 644 | if (went_to_zero) { |
| 645 | spin_unlock(lock); |
| 646 | went_to_zero = 0; |
| 647 | } |
| 648 | } |
| 649 | |
| 650 | return went_to_zero; |
| 651 | } |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 652 | |
| 653 | Now, as far as memory barriers go, as long as spin_lock() |
| 654 | strictly orders all subsequent memory operations (including |
| 655 | the cas()) with respect to itself, things will be fine. |
| 656 | |
Michael Hayes | a0ebb3f | 2006-06-26 18:27:35 +0200 | [diff] [blame] | 657 | Said another way, _atomic_dec_and_lock() must guarantee that |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 658 | a counter dropping to zero is never made visible before the |
| 659 | spinlock being acquired. |
| 660 | |
Silvio Fricke | 326bc87 | 2016-11-28 18:30:55 +0100 | [diff] [blame] | 661 | .. note:: |
| 662 | |
| 663 | Note that this also means that for the case where the counter is not |
| 664 | dropping to zero, there are no memory ordering requirements. |