| ============================ |
| LINUX KERNEL MEMORY BARRIERS |
| ============================ |
| |
| By: David Howells <dhowells@redhat.com> |
| |
| Contents: |
| |
| (*) Abstract memory access model. |
| |
| - Device operations. |
| - Guarantees. |
| |
| (*) What are memory barriers? |
| |
| - Varieties of memory barrier. |
| - What may not be assumed about memory barriers? |
| - Data dependency barriers. |
| - Control dependencies. |
| - SMP barrier pairing. |
| - Examples of memory barrier sequences. |
| |
| (*) Explicit kernel barriers. |
| |
| - Compiler barrier. |
| - The CPU memory barriers. |
| - MMIO write barrier. |
| |
| (*) Implicit kernel memory barriers. |
| |
| - Locking functions. |
| - Interrupt disabling functions. |
| - Miscellaneous functions. |
| |
| (*) Inter-CPU locking barrier effects. |
| |
| - Locks vs memory accesses. |
| - Locks vs I/O accesses. |
| |
| (*) Where are memory barriers needed? |
| |
| - Interprocessor interaction. |
| - Atomic operations. |
| - Accessing devices. |
| - Interrupts. |
| |
| (*) Kernel I/O barrier effects. |
| |
| (*) Assumed minimum execution ordering model. |
| |
| (*) The effects of the cpu cache. |
| |
| - Cache coherency. |
| - Cache coherency vs DMA. |
| - Cache coherency vs MMIO. |
| |
| (*) The things CPUs get up to. |
| |
| - And then there's the Alpha. |
| |
| (*) References. |
| |
| |
| ============================ |
| ABSTRACT MEMORY ACCESS MODEL |
| ============================ |
| |
| Consider the following abstract model of the system: |
| |
| : : |
| : : |
| : : |
| +-------+ : +--------+ : +-------+ |
| | | : | | : | | |
| | | : | | : | | |
| | CPU 1 |<----->| Memory |<----->| CPU 2 | |
| | | : | | : | | |
| | | : | | : | | |
| +-------+ : +--------+ : +-------+ |
| ^ : ^ : ^ |
| | : | : | |
| | : | : | |
| | : v : | |
| | : +--------+ : | |
| | : | | : | |
| | : | | : | |
| +---------->| Device |<----------+ |
| : | | : |
| : | | : |
| : +--------+ : |
| : : |
| |
| Each CPU executes a program that generates memory access operations. In the |
| abstract CPU, memory operation ordering is very relaxed, and a CPU may actually |
| perform the memory operations in any order it likes, provided program causality |
| appears to be maintained. Similarly, the compiler may also arrange the |
| instructions it emits in any order it likes, provided it doesn't affect the |
| apparent operation of the program. |
| |
| So in the above diagram, the effects of the memory operations performed by a |
| CPU are perceived by the rest of the system as the operations cross the |
| interface between the CPU and rest of the system (the dotted lines). |
| |
| |
| For example, consider the following sequence of events: |
| |
| CPU 1 CPU 2 |
| =============== =============== |
| { A == 1; B == 2 } |
| A = 3; x = A; |
| B = 4; y = B; |
| |
| The set of accesses as seen by the memory system in the middle can be arranged |
| in 24 different combinations: |
| |
| STORE A=3, STORE B=4, x=LOAD A->3, y=LOAD B->4 |
| STORE A=3, STORE B=4, y=LOAD B->4, x=LOAD A->3 |
| STORE A=3, x=LOAD A->3, STORE B=4, y=LOAD B->4 |
| STORE A=3, x=LOAD A->3, y=LOAD B->2, STORE B=4 |
| STORE A=3, y=LOAD B->2, STORE B=4, x=LOAD A->3 |
| STORE A=3, y=LOAD B->2, x=LOAD A->3, STORE B=4 |
| STORE B=4, STORE A=3, x=LOAD A->3, y=LOAD B->4 |
| STORE B=4, ... |
| ... |
| |
| and can thus result in four different combinations of values: |
| |
| x == 1, y == 2 |
| x == 1, y == 4 |
| x == 3, y == 2 |
| x == 3, y == 4 |
| |
| |
| Furthermore, the stores committed by a CPU to the memory system may not be |
| perceived by the loads made by another CPU in the same order as the stores were |
| committed. |
| |
| |
| As a further example, consider this sequence of events: |
| |
| CPU 1 CPU 2 |
| =============== =============== |
| { A == 1, B == 2, C = 3, P == &A, Q == &C } |
| B = 4; Q = P; |
| P = &B D = *Q; |
| |
| There is an obvious data dependency here, as the value loaded into D depends on |
| the address retrieved from P by CPU 2. At the end of the sequence, any of the |
| following results are possible: |
| |
| (Q == &A) and (D == 1) |
| (Q == &B) and (D == 2) |
| (Q == &B) and (D == 4) |
| |
| Note that CPU 2 will never try and load C into D because the CPU will load P |
| into Q before issuing the load of *Q. |
| |
| |
| DEVICE OPERATIONS |
| ----------------- |
| |
| Some devices present their control interfaces as collections of memory |
| locations, but the order in which the control registers are accessed is very |
| important. For instance, imagine an ethernet card with a set of internal |
| registers that are accessed through an address port register (A) and a data |
| port register (D). To read internal register 5, the following code might then |
| be used: |
| |
| *A = 5; |
| x = *D; |
| |
| but this might show up as either of the following two sequences: |
| |
| STORE *A = 5, x = LOAD *D |
| x = LOAD *D, STORE *A = 5 |
| |
| the second of which will almost certainly result in a malfunction, since it set |
| the address _after_ attempting to read the register. |
| |
| |
| GUARANTEES |
| ---------- |
| |
| There are some minimal guarantees that may be expected of a CPU: |
| |
| (*) On any given CPU, dependent memory accesses will be issued in order, with |
| respect to itself. This means that for: |
| |
| Q = P; D = *Q; |
| |
| the CPU will issue the following memory operations: |
| |
| Q = LOAD P, D = LOAD *Q |
| |
| and always in that order. |
| |
| (*) Overlapping loads and stores within a particular CPU will appear to be |
| ordered within that CPU. This means that for: |
| |
| a = *X; *X = b; |
| |
| the CPU will only issue the following sequence of memory operations: |
| |
| a = LOAD *X, STORE *X = b |
| |
| And for: |
| |
| *X = c; d = *X; |
| |
| the CPU will only issue: |
| |
| STORE *X = c, d = LOAD *X |
| |
| (Loads and stores overlap if they are targetted at overlapping pieces of |
| memory). |
| |
| And there are a number of things that _must_ or _must_not_ be assumed: |
| |
| (*) It _must_not_ be assumed that independent loads and stores will be issued |
| in the order given. This means that for: |
| |
| X = *A; Y = *B; *D = Z; |
| |
| we may get any of the following sequences: |
| |
| X = LOAD *A, Y = LOAD *B, STORE *D = Z |
| X = LOAD *A, STORE *D = Z, Y = LOAD *B |
| Y = LOAD *B, X = LOAD *A, STORE *D = Z |
| Y = LOAD *B, STORE *D = Z, X = LOAD *A |
| STORE *D = Z, X = LOAD *A, Y = LOAD *B |
| STORE *D = Z, Y = LOAD *B, X = LOAD *A |
| |
| (*) It _must_ be assumed that overlapping memory accesses may be merged or |
| discarded. This means that for: |
| |
| X = *A; Y = *(A + 4); |
| |
| we may get any one of the following sequences: |
| |
| X = LOAD *A; Y = LOAD *(A + 4); |
| Y = LOAD *(A + 4); X = LOAD *A; |
| {X, Y} = LOAD {*A, *(A + 4) }; |
| |
| And for: |
| |
| *A = X; Y = *A; |
| |
| we may get either of: |
| |
| STORE *A = X; Y = LOAD *A; |
| STORE *A = Y; |
| |
| |
| ========================= |
| WHAT ARE MEMORY BARRIERS? |
| ========================= |
| |
| As can be seen above, independent memory operations are effectively performed |
| in random order, but this can be a problem for CPU-CPU interaction and for I/O. |
| What is required is some way of intervening to instruct the compiler and the |
| CPU to restrict the order. |
| |
| Memory barriers are such interventions. They impose a perceived partial |
| ordering between the memory operations specified on either side of the barrier. |
| They request that the sequence of memory events generated appears to other |
| parts of the system as if the barrier is effective on that CPU. |
| |
| |
| VARIETIES OF MEMORY BARRIER |
| --------------------------- |
| |
| Memory barriers come in four basic varieties: |
| |
| (1) Write (or store) memory barriers. |
| |
| A write memory barrier gives a guarantee that all the STORE operations |
| specified before the barrier will appear to happen before all the STORE |
| operations specified after the barrier with respect to the other |
| components of the system. |
| |
| A write barrier is a partial ordering on stores only; it is not required |
| to have any effect on loads. |
| |
| A CPU can be viewed as as commiting a sequence of store operations to the |
| memory system as time progresses. All stores before a write barrier will |
| occur in the sequence _before_ all the stores after the write barrier. |
| |
| [!] Note that write barriers should normally be paired with read or data |
| dependency barriers; see the "SMP barrier pairing" subsection. |
| |
| |
| (2) Data dependency barriers. |
| |
| A data dependency barrier is a weaker form of read barrier. In the case |
| where two loads are performed such that the second depends on the result |
| of the first (eg: the first load retrieves the address to which the second |
| load will be directed), a data dependency barrier would be required to |
| make sure that the target of the second load is updated before the address |
| obtained by the first load is accessed. |
| |
| A data dependency barrier is a partial ordering on interdependent loads |
| only; it is not required to have any effect on stores, independent loads |
| or overlapping loads. |
| |
| As mentioned in (1), the other CPUs in the system can be viewed as |
| committing sequences of stores to the memory system that the CPU being |
| considered can then perceive. A data dependency barrier issued by the CPU |
| under consideration guarantees that for any load preceding it, if that |
| load touches one of a sequence of stores from another CPU, then by the |
| time the barrier completes, the effects of all the stores prior to that |
| touched by the load will be perceptible to any loads issued after the data |
| dependency barrier. |
| |
| See the "Examples of memory barrier sequences" subsection for diagrams |
| showing the ordering constraints. |
| |
| [!] Note that the first load really has to have a _data_ dependency and |
| not a control dependency. If the address for the second load is dependent |
| on the first load, but the dependency is through a conditional rather than |
| actually loading the address itself, then it's a _control_ dependency and |
| a full read barrier or better is required. See the "Control dependencies" |
| subsection for more information. |
| |
| [!] Note that data dependency barriers should normally be paired with |
| write barriers; see the "SMP barrier pairing" subsection. |
| |
| |
| (3) Read (or load) memory barriers. |
| |
| A read barrier is a data dependency barrier plus a guarantee that all the |
| LOAD operations specified before the barrier will appear to happen before |
| all the LOAD operations specified after the barrier with respect to the |
| other components of the system. |
| |
| A read barrier is a partial ordering on loads only; it is not required to |
| have any effect on stores. |
| |
| Read memory barriers imply data dependency barriers, and so can substitute |
| for them. |
| |
| [!] Note that read barriers should normally be paired with write barriers; |
| see the "SMP barrier pairing" subsection. |
| |
| |
| (4) General memory barriers. |
| |
| A general memory barrier is a combination of both a read memory barrier |
| and a write memory barrier. It is a partial ordering over both loads and |
| stores. |
| |
| General memory barriers imply both read and write memory barriers, and so |
| can substitute for either. |
| |
| |
| And a couple of implicit varieties: |
| |
| (5) LOCK operations. |
| |
| This acts as a one-way permeable barrier. It guarantees that all memory |
| operations after the LOCK operation will appear to happen after the LOCK |
| operation with respect to the other components of the system. |
| |
| Memory operations that occur before a LOCK operation may appear to happen |
| after it completes. |
| |
| A LOCK operation should almost always be paired with an UNLOCK operation. |
| |
| |
| (6) UNLOCK operations. |
| |
| This also acts as a one-way permeable barrier. It guarantees that all |
| memory operations before the UNLOCK operation will appear to happen before |
| the UNLOCK operation with respect to the other components of the system. |
| |
| Memory operations that occur after an UNLOCK operation may appear to |
| happen before it completes. |
| |
| LOCK and UNLOCK operations are guaranteed to appear with respect to each |
| other strictly in the order specified. |
| |
| The use of LOCK and UNLOCK operations generally precludes the need for |
| other sorts of memory barrier (but note the exceptions mentioned in the |
| subsection "MMIO write barrier"). |
| |
| |
| Memory barriers are only required where there's a possibility of interaction |
| between two CPUs or between a CPU and a device. If it can be guaranteed that |
| there won't be any such interaction in any particular piece of code, then |
| memory barriers are unnecessary in that piece of code. |
| |
| |
| Note that these are the _minimum_ guarantees. Different architectures may give |
| more substantial guarantees, but they may _not_ be relied upon outside of arch |
| specific code. |
| |
| |
| WHAT MAY NOT BE ASSUMED ABOUT MEMORY BARRIERS? |
| ---------------------------------------------- |
| |
| There are certain things that the Linux kernel memory barriers do not guarantee: |
| |
| (*) There is no guarantee that any of the memory accesses specified before a |
| memory barrier will be _complete_ by the completion of a memory barrier |
| instruction; the barrier can be considered to draw a line in that CPU's |
| access queue that accesses of the appropriate type may not cross. |
| |
| (*) There is no guarantee that issuing a memory barrier on one CPU will have |
| any direct effect on another CPU or any other hardware in the system. The |
| indirect effect will be the order in which the second CPU sees the effects |
| of the first CPU's accesses occur, but see the next point: |
| |
| (*) There is no guarantee that the a CPU will see the correct order of effects |
| from a second CPU's accesses, even _if_ the second CPU uses a memory |
| barrier, unless the first CPU _also_ uses a matching memory barrier (see |
| the subsection on "SMP Barrier Pairing"). |
| |
| (*) There is no guarantee that some intervening piece of off-the-CPU |
| hardware[*] will not reorder the memory accesses. CPU cache coherency |
| mechanisms should propagate the indirect effects of a memory barrier |
| between CPUs, but might not do so in order. |
| |
| [*] For information on bus mastering DMA and coherency please read: |
| |
| Documentation/pci.txt |
| Documentation/DMA-mapping.txt |
| Documentation/DMA-API.txt |
| |
| |
| DATA DEPENDENCY BARRIERS |
| ------------------------ |
| |
| The usage requirements of data dependency barriers are a little subtle, and |
| it's not always obvious that they're needed. To illustrate, consider the |
| following sequence of events: |
| |
| CPU 1 CPU 2 |
| =============== =============== |
| { A == 1, B == 2, C = 3, P == &A, Q == &C } |
| B = 4; |
| <write barrier> |
| P = &B |
| Q = P; |
| D = *Q; |
| |
| There's a clear data dependency here, and it would seem that by the end of the |
| sequence, Q must be either &A or &B, and that: |
| |
| (Q == &A) implies (D == 1) |
| (Q == &B) implies (D == 4) |
| |
| But! CPU 2's perception of P may be updated _before_ its perception of B, thus |
| leading to the following situation: |
| |
| (Q == &B) and (D == 2) ???? |
| |
| Whilst this may seem like a failure of coherency or causality maintenance, it |
| isn't, and this behaviour can be observed on certain real CPUs (such as the DEC |
| Alpha). |
| |
| To deal with this, a data dependency barrier must be inserted between the |
| address load and the data load: |
| |
| CPU 1 CPU 2 |
| =============== =============== |
| { A == 1, B == 2, C = 3, P == &A, Q == &C } |
| B = 4; |
| <write barrier> |
| P = &B |
| Q = P; |
| <data dependency barrier> |
| D = *Q; |
| |
| This enforces the occurrence of one of the two implications, and prevents the |
| third possibility from arising. |
| |
| [!] Note that this extremely counterintuitive situation arises most easily on |
| machines with split caches, so that, for example, one cache bank processes |
| even-numbered cache lines and the other bank processes odd-numbered cache |
| lines. The pointer P might be stored in an odd-numbered cache line, and the |
| variable B might be stored in an even-numbered cache line. Then, if the |
| even-numbered bank of the reading CPU's cache is extremely busy while the |
| odd-numbered bank is idle, one can see the new value of the pointer P (&B), |
| but the old value of the variable B (1). |
| |
| |
| Another example of where data dependency barriers might by required is where a |
| number is read from memory and then used to calculate the index for an array |
| access: |
| |
| CPU 1 CPU 2 |
| =============== =============== |
| { M[0] == 1, M[1] == 2, M[3] = 3, P == 0, Q == 3 } |
| M[1] = 4; |
| <write barrier> |
| P = 1 |
| Q = P; |
| <data dependency barrier> |
| D = M[Q]; |
| |
| |
| The data dependency barrier is very important to the RCU system, for example. |
| See rcu_dereference() in include/linux/rcupdate.h. This permits the current |
| target of an RCU'd pointer to be replaced with a new modified target, without |
| the replacement target appearing to be incompletely initialised. |
| |
| See also the subsection on "Cache Coherency" for a more thorough example. |
| |
| |
| CONTROL DEPENDENCIES |
| -------------------- |
| |
| A control dependency requires a full read memory barrier, not simply a data |
| dependency barrier to make it work correctly. Consider the following bit of |
| code: |
| |
| q = &a; |
| if (p) |
| q = &b; |
| <data dependency barrier> |
| x = *q; |
| |
| This will not have the desired effect because there is no actual data |
| dependency, but rather a control dependency that the CPU may short-circuit by |
| attempting to predict the outcome in advance. In such a case what's actually |
| required is: |
| |
| q = &a; |
| if (p) |
| q = &b; |
| <read barrier> |
| x = *q; |
| |
| |
| SMP BARRIER PAIRING |
| ------------------- |
| |
| When dealing with CPU-CPU interactions, certain types of memory barrier should |
| always be paired. A lack of appropriate pairing is almost certainly an error. |
| |
| A write barrier should always be paired with a data dependency barrier or read |
| barrier, though a general barrier would also be viable. Similarly a read |
| barrier or a data dependency barrier should always be paired with at least an |
| write barrier, though, again, a general barrier is viable: |
| |
| CPU 1 CPU 2 |
| =============== =============== |
| a = 1; |
| <write barrier> |
| b = 2; x = a; |
| <read barrier> |
| y = b; |
| |
| Or: |
| |
| CPU 1 CPU 2 |
| =============== =============================== |
| a = 1; |
| <write barrier> |
| b = &a; x = b; |
| <data dependency barrier> |
| y = *x; |
| |
| Basically, the read barrier always has to be there, even though it can be of |
| the "weaker" type. |
| |
| |
| EXAMPLES OF MEMORY BARRIER SEQUENCES |
| ------------------------------------ |
| |
| Firstly, write barriers act as a partial orderings on store operations. |
| Consider the following sequence of events: |
| |
| CPU 1 |
| ======================= |
| STORE A = 1 |
| STORE B = 2 |
| STORE C = 3 |
| <write barrier> |
| STORE D = 4 |
| STORE E = 5 |
| |
| This sequence of events is committed to the memory coherence system in an order |
| that the rest of the system might perceive as the unordered set of { STORE A, |
| STORE B, STORE C } all occuring before the unordered set of { STORE D, STORE E |
| }: |
| |
| +-------+ : : |
| | | +------+ |
| | |------>| C=3 | } /\ |
| | | : +------+ }----- \ -----> Events perceptible |
| | | : | A=1 | } \/ to rest of system |
| | | : +------+ } |
| | CPU 1 | : | B=2 | } |
| | | +------+ } |
| | | wwwwwwwwwwwwwwww } <--- At this point the write barrier |
| | | +------+ } requires all stores prior to the |
| | | : | E=5 | } barrier to be committed before |
| | | : +------+ } further stores may be take place. |
| | |------>| D=4 | } |
| | | +------+ |
| +-------+ : : |
| | |
| | Sequence in which stores committed to memory system |
| | by CPU 1 |
| V |
| |
| |
| Secondly, data dependency barriers act as a partial orderings on data-dependent |
| loads. Consider the following sequence of events: |
| |
| CPU 1 CPU 2 |
| ======================= ======================= |
| STORE A = 1 |
| STORE B = 2 |
| <write barrier> |
| STORE C = &B LOAD X |
| STORE D = 4 LOAD C (gets &B) |
| LOAD *C (reads B) |
| |
| Without intervention, CPU 2 may perceive the events on CPU 1 in some |
| effectively random order, despite the write barrier issued by CPU 1: |
| |
| +-------+ : : : : |
| | | +------+ +-------+ | Sequence of update |
| | |------>| B=2 |----- --->| Y->8 | | of perception on |
| | | : +------+ \ +-------+ | CPU 2 |
| | CPU 1 | : | A=1 | \ --->| C->&Y | V |
| | | +------+ | +-------+ |
| | | wwwwwwwwwwwwwwww | : : |
| | | +------+ | : : |
| | | : | C=&B |--- | : : +-------+ |
| | | : +------+ \ | +-------+ | | |
| | |------>| D=4 | ----------->| C->&B |------>| | |
| | | +------+ | +-------+ | | |
| +-------+ : : | : : | | |
| | : : | | |
| | : : | CPU 2 | |
| | +-------+ | | |
| Apparently incorrect ---> | | B->7 |------>| | |
| perception of B (!) | +-------+ | | |
| | : : | | |
| | +-------+ | | |
| The load of X holds ---> \ | X->9 |------>| | |
| up the maintenance \ +-------+ | | |
| of coherence of B ----->| B->2 | +-------+ |
| +-------+ |
| : : |
| |
| |
| In the above example, CPU 2 perceives that B is 7, despite the load of *C |
| (which would be B) coming after the the LOAD of C. |
| |
| If, however, a data dependency barrier were to be placed between the load of C |
| and the load of *C (ie: B) on CPU 2, then the following will occur: |
| |
| +-------+ : : : : |
| | | +------+ +-------+ |
| | |------>| B=2 |----- --->| Y->8 | |
| | | : +------+ \ +-------+ |
| | CPU 1 | : | A=1 | \ --->| C->&Y | |
| | | +------+ | +-------+ |
| | | wwwwwwwwwwwwwwww | : : |
| | | +------+ | : : |
| | | : | C=&B |--- | : : +-------+ |
| | | : +------+ \ | +-------+ | | |
| | |------>| D=4 | ----------->| C->&B |------>| | |
| | | +------+ | +-------+ | | |
| +-------+ : : | : : | | |
| | : : | | |
| | : : | CPU 2 | |
| | +-------+ | | |
| \ | X->9 |------>| | |
| \ +-------+ | | |
| ----->| B->2 | | | |
| +-------+ | | |
| Makes sure all effects ---> ddddddddddddddddd | | |
| prior to the store of C +-------+ | | |
| are perceptible to | B->2 |------>| | |
| successive loads +-------+ | | |
| : : +-------+ |
| |
| |
| And thirdly, a read barrier acts as a partial order on loads. Consider the |
| following sequence of events: |
| |
| CPU 1 CPU 2 |
| ======================= ======================= |
| STORE A=1 |
| STORE B=2 |
| STORE C=3 |
| <write barrier> |
| STORE D=4 |
| STORE E=5 |
| LOAD A |
| LOAD B |
| LOAD C |
| LOAD D |
| LOAD E |
| |
| Without intervention, CPU 2 may then choose to perceive the events on CPU 1 in |
| some effectively random order, despite the write barrier issued by CPU 1: |
| |
| +-------+ : : |
| | | +------+ |
| | |------>| C=3 | } |
| | | : +------+ } |
| | | : | A=1 | } |
| | | : +------+ } |
| | CPU 1 | : | B=2 | }--- |
| | | +------+ } \ |
| | | wwwwwwwwwwwww} \ |
| | | +------+ } \ : : +-------+ |
| | | : | E=5 | } \ +-------+ | | |
| | | : +------+ } \ { | C->3 |------>| | |
| | |------>| D=4 | } \ { +-------+ : | | |
| | | +------+ \ { | E->5 | : | | |
| +-------+ : : \ { +-------+ : | | |
| Transfer -->{ | A->1 | : | CPU 2 | |
| from CPU 1 { +-------+ : | | |
| to CPU 2 { | D->4 | : | | |
| { +-------+ : | | |
| { | B->2 |------>| | |
| +-------+ | | |
| : : +-------+ |
| |
| |
| If, however, a read barrier were to be placed between the load of C and the |
| load of D on CPU 2, then the partial ordering imposed by CPU 1 will be |
| perceived correctly by CPU 2. |
| |
| +-------+ : : |
| | | +------+ |
| | |------>| C=3 | } |
| | | : +------+ } |
| | | : | A=1 | }--- |
| | | : +------+ } \ |
| | CPU 1 | : | B=2 | } \ |
| | | +------+ \ |
| | | wwwwwwwwwwwwwwww \ |
| | | +------+ \ : : +-------+ |
| | | : | E=5 | } \ +-------+ | | |
| | | : +------+ }--- \ { | C->3 |------>| | |
| | |------>| D=4 | } \ \ { +-------+ : | | |
| | | +------+ \ -->{ | B->2 | : | | |
| +-------+ : : \ { +-------+ : | | |
| \ { | A->1 | : | CPU 2 | |
| \ +-------+ | | |
| At this point the read ----> \ rrrrrrrrrrrrrrrrr | | |
| barrier causes all effects \ +-------+ | | |
| prior to the storage of C \ { | E->5 | : | | |
| to be perceptible to CPU 2 -->{ +-------+ : | | |
| { | D->4 |------>| | |
| +-------+ | | |
| : : +-------+ |
| |
| |
| ======================== |
| EXPLICIT KERNEL BARRIERS |
| ======================== |
| |
| The Linux kernel has a variety of different barriers that act at different |
| levels: |
| |
| (*) Compiler barrier. |
| |
| (*) CPU memory barriers. |
| |
| (*) MMIO write barrier. |
| |
| |
| COMPILER BARRIER |
| ---------------- |
| |
| The Linux kernel has an explicit compiler barrier function that prevents the |
| compiler from moving the memory accesses either side of it to the other side: |
| |
| barrier(); |
| |
| This a general barrier - lesser varieties of compiler barrier do not exist. |
| |
| The compiler barrier has no direct effect on the CPU, which may then reorder |
| things however it wishes. |
| |
| |
| CPU MEMORY BARRIERS |
| ------------------- |
| |
| The Linux kernel has eight basic CPU memory barriers: |
| |
| TYPE MANDATORY SMP CONDITIONAL |
| =============== ======================= =========================== |
| GENERAL mb() smp_mb() |
| WRITE wmb() smp_wmb() |
| READ rmb() smp_rmb() |
| DATA DEPENDENCY read_barrier_depends() smp_read_barrier_depends() |
| |
| |
| All CPU memory barriers unconditionally imply compiler barriers. |
| |
| SMP memory barriers are reduced to compiler barriers on uniprocessor compiled |
| systems because it is assumed that a CPU will be appear to be self-consistent, |
| and will order overlapping accesses correctly with respect to itself. |
| |
| [!] Note that SMP memory barriers _must_ be used to control the ordering of |
| references to shared memory on SMP systems, though the use of locking instead |
| is sufficient. |
| |
| Mandatory barriers should not be used to control SMP effects, since mandatory |
| barriers unnecessarily impose overhead on UP systems. They may, however, be |
| used to control MMIO effects on accesses through relaxed memory I/O windows. |
| These are required even on non-SMP systems as they affect the order in which |
| memory operations appear to a device by prohibiting both the compiler and the |
| CPU from reordering them. |
| |
| |
| There are some more advanced barrier functions: |
| |
| (*) set_mb(var, value) |
| (*) set_wmb(var, value) |
| |
| These assign the value to the variable and then insert at least a write |
| barrier after it, depending on the function. They aren't guaranteed to |
| insert anything more than a compiler barrier in a UP compilation. |
| |
| |
| (*) smp_mb__before_atomic_dec(); |
| (*) smp_mb__after_atomic_dec(); |
| (*) smp_mb__before_atomic_inc(); |
| (*) smp_mb__after_atomic_inc(); |
| |
| These are for use with atomic add, subtract, increment and decrement |
| functions that don't return a value, especially when used for reference |
| counting. These functions do not imply memory barriers. |
| |
| As an example, consider a piece of code that marks an object as being dead |
| and then decrements the object's reference count: |
| |
| obj->dead = 1; |
| smp_mb__before_atomic_dec(); |
| atomic_dec(&obj->ref_count); |
| |
| This makes sure that the death mark on the object is perceived to be set |
| *before* the reference counter is decremented. |
| |
| See Documentation/atomic_ops.txt for more information. See the "Atomic |
| operations" subsection for information on where to use these. |
| |
| |
| (*) smp_mb__before_clear_bit(void); |
| (*) smp_mb__after_clear_bit(void); |
| |
| These are for use similar to the atomic inc/dec barriers. These are |
| typically used for bitwise unlocking operations, so care must be taken as |
| there are no implicit memory barriers here either. |
| |
| Consider implementing an unlock operation of some nature by clearing a |
| locking bit. The clear_bit() would then need to be barriered like this: |
| |
| smp_mb__before_clear_bit(); |
| clear_bit( ... ); |
| |
| This prevents memory operations before the clear leaking to after it. See |
| the subsection on "Locking Functions" with reference to UNLOCK operation |
| implications. |
| |
| See Documentation/atomic_ops.txt for more information. See the "Atomic |
| operations" subsection for information on where to use these. |
| |
| |
| MMIO WRITE BARRIER |
| ------------------ |
| |
| The Linux kernel also has a special barrier for use with memory-mapped I/O |
| writes: |
| |
| mmiowb(); |
| |
| This is a variation on the mandatory write barrier that causes writes to weakly |
| ordered I/O regions to be partially ordered. Its effects may go beyond the |
| CPU->Hardware interface and actually affect the hardware at some level. |
| |
| See the subsection "Locks vs I/O accesses" for more information. |
| |
| |
| =============================== |
| IMPLICIT KERNEL MEMORY BARRIERS |
| =============================== |
| |
| Some of the other functions in the linux kernel imply memory barriers, amongst |
| which are locking, scheduling and memory allocation functions. |
| |
| This specification is a _minimum_ guarantee; any particular architecture may |
| provide more substantial guarantees, but these may not be relied upon outside |
| of arch specific code. |
| |
| |
| LOCKING FUNCTIONS |
| ----------------- |
| |
| The Linux kernel has a number of locking constructs: |
| |
| (*) spin locks |
| (*) R/W spin locks |
| (*) mutexes |
| (*) semaphores |
| (*) R/W semaphores |
| (*) RCU |
| |
| In all cases there are variants on "LOCK" operations and "UNLOCK" operations |
| for each construct. These operations all imply certain barriers: |
| |
| (1) LOCK operation implication: |
| |
| Memory operations issued after the LOCK will be completed after the LOCK |
| operation has completed. |
| |
| Memory operations issued before the LOCK may be completed after the LOCK |
| operation has completed. |
| |
| (2) UNLOCK operation implication: |
| |
| Memory operations issued before the UNLOCK will be completed before the |
| UNLOCK operation has completed. |
| |
| Memory operations issued after the UNLOCK may be completed before the |
| UNLOCK operation has completed. |
| |
| (3) LOCK vs LOCK implication: |
| |
| All LOCK operations issued before another LOCK operation will be completed |
| before that LOCK operation. |
| |
| (4) LOCK vs UNLOCK implication: |
| |
| All LOCK operations issued before an UNLOCK operation will be completed |
| before the UNLOCK operation. |
| |
| All UNLOCK operations issued before a LOCK operation will be completed |
| before the LOCK operation. |
| |
| (5) Failed conditional LOCK implication: |
| |
| Certain variants of the LOCK operation may fail, either due to being |
| unable to get the lock immediately, or due to receiving an unblocked |
| signal whilst asleep waiting for the lock to become available. Failed |
| locks do not imply any sort of barrier. |
| |
| Therefore, from (1), (2) and (4) an UNLOCK followed by an unconditional LOCK is |
| equivalent to a full barrier, but a LOCK followed by an UNLOCK is not. |
| |
| [!] Note: one of the consequence of LOCKs and UNLOCKs being only one-way |
| barriers is that the effects instructions outside of a critical section may |
| seep into the inside of the critical section. |
| |
| Locks and semaphores may not provide any guarantee of ordering on UP compiled |
| systems, and so cannot be counted on in such a situation to actually achieve |
| anything at all - especially with respect to I/O accesses - unless combined |
| with interrupt disabling operations. |
| |
| See also the section on "Inter-CPU locking barrier effects". |
| |
| |
| As an example, consider the following: |
| |
| *A = a; |
| *B = b; |
| LOCK |
| *C = c; |
| *D = d; |
| UNLOCK |
| *E = e; |
| *F = f; |
| |
| The following sequence of events is acceptable: |
| |
| LOCK, {*F,*A}, *E, {*C,*D}, *B, UNLOCK |
| |
| [+] Note that {*F,*A} indicates a combined access. |
| |
| But none of the following are: |
| |
| {*F,*A}, *B, LOCK, *C, *D, UNLOCK, *E |
| *A, *B, *C, LOCK, *D, UNLOCK, *E, *F |
| *A, *B, LOCK, *C, UNLOCK, *D, *E, *F |
| *B, LOCK, *C, *D, UNLOCK, {*F,*A}, *E |
| |
| |
| |
| INTERRUPT DISABLING FUNCTIONS |
| ----------------------------- |
| |
| Functions that disable interrupts (LOCK equivalent) and enable interrupts |
| (UNLOCK equivalent) will act as compiler barriers only. So if memory or I/O |
| barriers are required in such a situation, they must be provided from some |
| other means. |
| |
| |
| MISCELLANEOUS FUNCTIONS |
| ----------------------- |
| |
| Other functions that imply barriers: |
| |
| (*) schedule() and similar imply full memory barriers. |
| |
| (*) Memory allocation and release functions imply full memory barriers. |
| |
| |
| ================================= |
| INTER-CPU LOCKING BARRIER EFFECTS |
| ================================= |
| |
| On SMP systems locking primitives give a more substantial form of barrier: one |
| that does affect memory access ordering on other CPUs, within the context of |
| conflict on any particular lock. |
| |
| |
| LOCKS VS MEMORY ACCESSES |
| ------------------------ |
| |
| Consider the following: the system has a pair of spinlocks (N) and (Q), and |
| three CPUs; then should the following sequence of events occur: |
| |
| CPU 1 CPU 2 |
| =============================== =============================== |
| *A = a; *E = e; |
| LOCK M LOCK Q |
| *B = b; *F = f; |
| *C = c; *G = g; |
| UNLOCK M UNLOCK Q |
| *D = d; *H = h; |
| |
| Then there is no guarantee as to what order CPU #3 will see the accesses to *A |
| through *H occur in, other than the constraints imposed by the separate locks |
| on the separate CPUs. It might, for example, see: |
| |
| *E, LOCK M, LOCK Q, *G, *C, *F, *A, *B, UNLOCK Q, *D, *H, UNLOCK M |
| |
| But it won't see any of: |
| |
| *B, *C or *D preceding LOCK M |
| *A, *B or *C following UNLOCK M |
| *F, *G or *H preceding LOCK Q |
| *E, *F or *G following UNLOCK Q |
| |
| |
| However, if the following occurs: |
| |
| CPU 1 CPU 2 |
| =============================== =============================== |
| *A = a; |
| LOCK M [1] |
| *B = b; |
| *C = c; |
| UNLOCK M [1] |
| *D = d; *E = e; |
| LOCK M [2] |
| *F = f; |
| *G = g; |
| UNLOCK M [2] |
| *H = h; |
| |
| CPU #3 might see: |
| |
| *E, LOCK M [1], *C, *B, *A, UNLOCK M [1], |
| LOCK M [2], *H, *F, *G, UNLOCK M [2], *D |
| |
| But assuming CPU #1 gets the lock first, it won't see any of: |
| |
| *B, *C, *D, *F, *G or *H preceding LOCK M [1] |
| *A, *B or *C following UNLOCK M [1] |
| *F, *G or *H preceding LOCK M [2] |
| *A, *B, *C, *E, *F or *G following UNLOCK M [2] |
| |
| |
| LOCKS VS I/O ACCESSES |
| --------------------- |
| |
| Under certain circumstances (especially involving NUMA), I/O accesses within |
| two spinlocked sections on two different CPUs may be seen as interleaved by the |
| PCI bridge, because the PCI bridge does not necessarily participate in the |
| cache-coherence protocol, and is therefore incapable of issuing the required |
| read memory barriers. |
| |
| For example: |
| |
| CPU 1 CPU 2 |
| =============================== =============================== |
| spin_lock(Q) |
| writel(0, ADDR) |
| writel(1, DATA); |
| spin_unlock(Q); |
| spin_lock(Q); |
| writel(4, ADDR); |
| writel(5, DATA); |
| spin_unlock(Q); |
| |
| may be seen by the PCI bridge as follows: |
| |
| STORE *ADDR = 0, STORE *ADDR = 4, STORE *DATA = 1, STORE *DATA = 5 |
| |
| which would probably cause the hardware to malfunction. |
| |
| |
| What is necessary here is to intervene with an mmiowb() before dropping the |
| spinlock, for example: |
| |
| CPU 1 CPU 2 |
| =============================== =============================== |
| spin_lock(Q) |
| writel(0, ADDR) |
| writel(1, DATA); |
| mmiowb(); |
| spin_unlock(Q); |
| spin_lock(Q); |
| writel(4, ADDR); |
| writel(5, DATA); |
| mmiowb(); |
| spin_unlock(Q); |
| |
| this will ensure that the two stores issued on CPU #1 appear at the PCI bridge |
| before either of the stores issued on CPU #2. |
| |
| |
| Furthermore, following a store by a load to the same device obviates the need |
| for an mmiowb(), because the load forces the store to complete before the load |
| is performed: |
| |
| CPU 1 CPU 2 |
| =============================== =============================== |
| spin_lock(Q) |
| writel(0, ADDR) |
| a = readl(DATA); |
| spin_unlock(Q); |
| spin_lock(Q); |
| writel(4, ADDR); |
| b = readl(DATA); |
| spin_unlock(Q); |
| |
| |
| See Documentation/DocBook/deviceiobook.tmpl for more information. |
| |
| |
| ================================= |
| WHERE ARE MEMORY BARRIERS NEEDED? |
| ================================= |
| |
| Under normal operation, memory operation reordering is generally not going to |
| be a problem as a single-threaded linear piece of code will still appear to |
| work correctly, even if it's in an SMP kernel. There are, however, three |
| circumstances in which reordering definitely _could_ be a problem: |
| |
| (*) Interprocessor interaction. |
| |
| (*) Atomic operations. |
| |
| (*) Accessing devices (I/O). |
| |
| (*) Interrupts. |
| |
| |
| INTERPROCESSOR INTERACTION |
| -------------------------- |
| |
| When there's a system with more than one processor, more than one CPU in the |
| system may be working on the same data set at the same time. This can cause |
| synchronisation problems, and the usual way of dealing with them is to use |
| locks. Locks, however, are quite expensive, and so it may be preferable to |
| operate without the use of a lock if at all possible. In such a case |
| operations that affect both CPUs may have to be carefully ordered to prevent |
| a malfunction. |
| |
| Consider, for example, the R/W semaphore slow path. Here a waiting process is |
| queued on the semaphore, by virtue of it having a piece of its stack linked to |
| the semaphore's list of waiting processes: |
| |
| struct rw_semaphore { |
| ... |
| spinlock_t lock; |
| struct list_head waiters; |
| }; |
| |
| struct rwsem_waiter { |
| struct list_head list; |
| struct task_struct *task; |
| }; |
| |
| To wake up a particular waiter, the up_read() or up_write() functions have to: |
| |
| (1) read the next pointer from this waiter's record to know as to where the |
| next waiter record is; |
| |
| (4) read the pointer to the waiter's task structure; |
| |
| (3) clear the task pointer to tell the waiter it has been given the semaphore; |
| |
| (4) call wake_up_process() on the task; and |
| |
| (5) release the reference held on the waiter's task struct. |
| |
| In otherwords, it has to perform this sequence of events: |
| |
| LOAD waiter->list.next; |
| LOAD waiter->task; |
| STORE waiter->task; |
| CALL wakeup |
| RELEASE task |
| |
| and if any of these steps occur out of order, then the whole thing may |
| malfunction. |
| |
| Once it has queued itself and dropped the semaphore lock, the waiter does not |
| get the lock again; it instead just waits for its task pointer to be cleared |
| before proceeding. Since the record is on the waiter's stack, this means that |
| if the task pointer is cleared _before_ the next pointer in the list is read, |
| another CPU might start processing the waiter and might clobber the waiter's |
| stack before the up*() function has a chance to read the next pointer. |
| |
| Consider then what might happen to the above sequence of events: |
| |
| CPU 1 CPU 2 |
| =============================== =============================== |
| down_xxx() |
| Queue waiter |
| Sleep |
| up_yyy() |
| LOAD waiter->task; |
| STORE waiter->task; |
| Woken up by other event |
| <preempt> |
| Resume processing |
| down_xxx() returns |
| call foo() |
| foo() clobbers *waiter |
| </preempt> |
| LOAD waiter->list.next; |
| --- OOPS --- |
| |
| This could be dealt with using the semaphore lock, but then the down_xxx() |
| function has to needlessly get the spinlock again after being woken up. |
| |
| The way to deal with this is to insert a general SMP memory barrier: |
| |
| LOAD waiter->list.next; |
| LOAD waiter->task; |
| smp_mb(); |
| STORE waiter->task; |
| CALL wakeup |
| RELEASE task |
| |
| In this case, the barrier makes a guarantee that all memory accesses before the |
| barrier will appear to happen before all the memory accesses after the barrier |
| with respect to the other CPUs on the system. It does _not_ guarantee that all |
| the memory accesses before the barrier will be complete by the time the barrier |
| instruction itself is complete. |
| |
| On a UP system - where this wouldn't be a problem - the smp_mb() is just a |
| compiler barrier, thus making sure the compiler emits the instructions in the |
| right order without actually intervening in the CPU. Since there there's only |
| one CPU, that CPU's dependency ordering logic will take care of everything |
| else. |
| |
| |
| ATOMIC OPERATIONS |
| ----------------- |
| |
| Whilst they are technically interprocessor interaction considerations, atomic |
| operations are noted specially as some of them imply full memory barriers and |
| some don't, but they're very heavily relied on as a group throughout the |
| kernel. |
| |
| Any atomic operation that modifies some state in memory and returns information |
| about the state (old or new) implies an SMP-conditional general memory barrier |
| (smp_mb()) on each side of the actual operation. These include: |
| |
| xchg(); |
| cmpxchg(); |
| atomic_cmpxchg(); |
| atomic_inc_return(); |
| atomic_dec_return(); |
| atomic_add_return(); |
| atomic_sub_return(); |
| atomic_inc_and_test(); |
| atomic_dec_and_test(); |
| atomic_sub_and_test(); |
| atomic_add_negative(); |
| atomic_add_unless(); |
| test_and_set_bit(); |
| test_and_clear_bit(); |
| test_and_change_bit(); |
| |
| These are used for such things as implementing LOCK-class and UNLOCK-class |
| operations and adjusting reference counters towards object destruction, and as |
| such the implicit memory barrier effects are necessary. |
| |
| |
| The following operation are potential problems as they do _not_ imply memory |
| barriers, but might be used for implementing such things as UNLOCK-class |
| operations: |
| |
| atomic_set(); |
| set_bit(); |
| clear_bit(); |
| change_bit(); |
| |
| With these the appropriate explicit memory barrier should be used if necessary |
| (smp_mb__before_clear_bit() for instance). |
| |
| |
| The following also do _not_ imply memory barriers, and so may require explicit |
| memory barriers under some circumstances (smp_mb__before_atomic_dec() for |
| instance)): |
| |
| atomic_add(); |
| atomic_sub(); |
| atomic_inc(); |
| atomic_dec(); |
| |
| If they're used for statistics generation, then they probably don't need memory |
| barriers, unless there's a coupling between statistical data. |
| |
| If they're used for reference counting on an object to control its lifetime, |
| they probably don't need memory barriers because either the reference count |
| will be adjusted inside a locked section, or the caller will already hold |
| sufficient references to make the lock, and thus a memory barrier unnecessary. |
| |
| If they're used for constructing a lock of some description, then they probably |
| do need memory barriers as a lock primitive generally has to do things in a |
| specific order. |
| |
| |
| Basically, each usage case has to be carefully considered as to whether memory |
| barriers are needed or not. |
| |
| [!] Note that special memory barrier primitives are available for these |
| situations because on some CPUs the atomic instructions used imply full memory |
| barriers, and so barrier instructions are superfluous in conjunction with them, |
| and in such cases the special barrier primitives will be no-ops. |
| |
| See Documentation/atomic_ops.txt for more information. |
| |
| |
| ACCESSING DEVICES |
| ----------------- |
| |
| Many devices can be memory mapped, and so appear to the CPU as if they're just |
| a set of memory locations. To control such a device, the driver usually has to |
| make the right memory accesses in exactly the right order. |
| |
| However, having a clever CPU or a clever compiler creates a potential problem |
| in that the carefully sequenced accesses in the driver code won't reach the |
| device in the requisite order if the CPU or the compiler thinks it is more |
| efficient to reorder, combine or merge accesses - something that would cause |
| the device to malfunction. |
| |
| Inside of the Linux kernel, I/O should be done through the appropriate accessor |
| routines - such as inb() or writel() - which know how to make such accesses |
| appropriately sequential. Whilst this, for the most part, renders the explicit |
| use of memory barriers unnecessary, there are a couple of situations where they |
| might be needed: |
| |
| (1) On some systems, I/O stores are not strongly ordered across all CPUs, and |
| so for _all_ general drivers locks should be used and mmiowb() must be |
| issued prior to unlocking the critical section. |
| |
| (2) If the accessor functions are used to refer to an I/O memory window with |
| relaxed memory access properties, then _mandatory_ memory barriers are |
| required to enforce ordering. |
| |
| See Documentation/DocBook/deviceiobook.tmpl for more information. |
| |
| |
| INTERRUPTS |
| ---------- |
| |
| A driver may be interrupted by its own interrupt service routine, and thus the |
| two parts of the driver may interfere with each other's attempts to control or |
| access the device. |
| |
| This may be alleviated - at least in part - by disabling local interrupts (a |
| form of locking), such that the critical operations are all contained within |
| the interrupt-disabled section in the driver. Whilst the driver's interrupt |
| routine is executing, the driver's core may not run on the same CPU, and its |
| interrupt is not permitted to happen again until the current interrupt has been |
| handled, thus the interrupt handler does not need to lock against that. |
| |
| However, consider a driver that was talking to an ethernet card that sports an |
| address register and a data register. If that driver's core talks to the card |
| under interrupt-disablement and then the driver's interrupt handler is invoked: |
| |
| LOCAL IRQ DISABLE |
| writew(ADDR, 3); |
| writew(DATA, y); |
| LOCAL IRQ ENABLE |
| <interrupt> |
| writew(ADDR, 4); |
| q = readw(DATA); |
| </interrupt> |
| |
| The store to the data register might happen after the second store to the |
| address register if ordering rules are sufficiently relaxed: |
| |
| STORE *ADDR = 3, STORE *ADDR = 4, STORE *DATA = y, q = LOAD *DATA |
| |
| |
| If ordering rules are relaxed, it must be assumed that accesses done inside an |
| interrupt disabled section may leak outside of it and may interleave with |
| accesses performed in an interrupt - and vice versa - unless implicit or |
| explicit barriers are used. |
| |
| Normally this won't be a problem because the I/O accesses done inside such |
| sections will include synchronous load operations on strictly ordered I/O |
| registers that form implicit I/O barriers. If this isn't sufficient then an |
| mmiowb() may need to be used explicitly. |
| |
| |
| A similar situation may occur between an interrupt routine and two routines |
| running on separate CPUs that communicate with each other. If such a case is |
| likely, then interrupt-disabling locks should be used to guarantee ordering. |
| |
| |
| ========================== |
| KERNEL I/O BARRIER EFFECTS |
| ========================== |
| |
| When accessing I/O memory, drivers should use the appropriate accessor |
| functions: |
| |
| (*) inX(), outX(): |
| |
| These are intended to talk to I/O space rather than memory space, but |
| that's primarily a CPU-specific concept. The i386 and x86_64 processors do |
| indeed have special I/O space access cycles and instructions, but many |
| CPUs don't have such a concept. |
| |
| The PCI bus, amongst others, defines an I/O space concept - which on such |
| CPUs as i386 and x86_64 cpus readily maps to the CPU's concept of I/O |
| space. However, it may also mapped as a virtual I/O space in the CPU's |
| memory map, particularly on those CPUs that don't support alternate |
| I/O spaces. |
| |
| Accesses to this space may be fully synchronous (as on i386), but |
| intermediary bridges (such as the PCI host bridge) may not fully honour |
| that. |
| |
| They are guaranteed to be fully ordered with respect to each other. |
| |
| They are not guaranteed to be fully ordered with respect to other types of |
| memory and I/O operation. |
| |
| (*) readX(), writeX(): |
| |
| Whether these are guaranteed to be fully ordered and uncombined with |
| respect to each other on the issuing CPU depends on the characteristics |
| defined for the memory window through which they're accessing. On later |
| i386 architecture machines, for example, this is controlled by way of the |
| MTRR registers. |
| |
| Ordinarily, these will be guaranteed to be fully ordered and uncombined,, |
| provided they're not accessing a prefetchable device. |
| |
| However, intermediary hardware (such as a PCI bridge) may indulge in |
| deferral if it so wishes; to flush a store, a load from the same location |
| is preferred[*], but a load from the same device or from configuration |
| space should suffice for PCI. |
| |
| [*] NOTE! attempting to load from the same location as was written to may |
| cause a malfunction - consider the 16550 Rx/Tx serial registers for |
| example. |
| |
| Used with prefetchable I/O memory, an mmiowb() barrier may be required to |
| force stores to be ordered. |
| |
| Please refer to the PCI specification for more information on interactions |
| between PCI transactions. |
| |
| (*) readX_relaxed() |
| |
| These are similar to readX(), but are not guaranteed to be ordered in any |
| way. Be aware that there is no I/O read barrier available. |
| |
| (*) ioreadX(), iowriteX() |
| |
| These will perform as appropriate for the type of access they're actually |
| doing, be it inX()/outX() or readX()/writeX(). |
| |
| |
| ======================================== |
| ASSUMED MINIMUM EXECUTION ORDERING MODEL |
| ======================================== |
| |
| It has to be assumed that the conceptual CPU is weakly-ordered but that it will |
| maintain the appearance of program causality with respect to itself. Some CPUs |
| (such as i386 or x86_64) are more constrained than others (such as powerpc or |
| frv), and so the most relaxed case (namely DEC Alpha) must be assumed outside |
| of arch-specific code. |
| |
| This means that it must be considered that the CPU will execute its instruction |
| stream in any order it feels like - or even in parallel - provided that if an |
| instruction in the stream depends on the an earlier instruction, then that |
| earlier instruction must be sufficiently complete[*] before the later |
| instruction may proceed; in other words: provided that the appearance of |
| causality is maintained. |
| |
| [*] Some instructions have more than one effect - such as changing the |
| condition codes, changing registers or changing memory - and different |
| instructions may depend on different effects. |
| |
| A CPU may also discard any instruction sequence that winds up having no |
| ultimate effect. For example, if two adjacent instructions both load an |
| immediate value into the same register, the first may be discarded. |
| |
| |
| Similarly, it has to be assumed that compiler might reorder the instruction |
| stream in any way it sees fit, again provided the appearance of causality is |
| maintained. |
| |
| |
| ============================ |
| THE EFFECTS OF THE CPU CACHE |
| ============================ |
| |
| The way cached memory operations are perceived across the system is affected to |
| a certain extent by the caches that lie between CPUs and memory, and by the |
| memory coherence system that maintains the consistency of state in the system. |
| |
| As far as the way a CPU interacts with another part of the system through the |
| caches goes, the memory system has to include the CPU's caches, and memory |
| barriers for the most part act at the interface between the CPU and its cache |
| (memory barriers logically act on the dotted line in the following diagram): |
| |
| <--- CPU ---> : <----------- Memory -----------> |
| : |
| +--------+ +--------+ : +--------+ +-----------+ |
| | | | | : | | | | +--------+ |
| | CPU | | Memory | : | CPU | | | | | |
| | Core |--->| Access |----->| Cache |<-->| | | | |
| | | | Queue | : | | | |--->| Memory | |
| | | | | : | | | | | | |
| +--------+ +--------+ : +--------+ | | | | |
| : | Cache | +--------+ |
| : | Coherency | |
| : | Mechanism | +--------+ |
| +--------+ +--------+ : +--------+ | | | | |
| | | | | : | | | | | | |
| | CPU | | Memory | : | CPU | | |--->| Device | |
| | Core |--->| Access |----->| Cache |<-->| | | | |
| | | | Queue | : | | | | | | |
| | | | | : | | | | +--------+ |
| +--------+ +--------+ : +--------+ +-----------+ |
| : |
| : |
| |
| Although any particular load or store may not actually appear outside of the |
| CPU that issued it since it may have been satisfied within the CPU's own cache, |
| it will still appear as if the full memory access had taken place as far as the |
| other CPUs are concerned since the cache coherency mechanisms will migrate the |
| cacheline over to the accessing CPU and propagate the effects upon conflict. |
| |
| The CPU core may execute instructions in any order it deems fit, provided the |
| expected program causality appears to be maintained. Some of the instructions |
| generate load and store operations which then go into the queue of memory |
| accesses to be performed. The core may place these in the queue in any order |
| it wishes, and continue execution until it is forced to wait for an instruction |
| to complete. |
| |
| What memory barriers are concerned with is controlling the order in which |
| accesses cross from the CPU side of things to the memory side of things, and |
| the order in which the effects are perceived to happen by the other observers |
| in the system. |
| |
| [!] Memory barriers are _not_ needed within a given CPU, as CPUs always see |
| their own loads and stores as if they had happened in program order. |
| |
| [!] MMIO or other device accesses may bypass the cache system. This depends on |
| the properties of the memory window through which devices are accessed and/or |
| the use of any special device communication instructions the CPU may have. |
| |
| |
| CACHE COHERENCY |
| --------------- |
| |
| Life isn't quite as simple as it may appear above, however: for while the |
| caches are expected to be coherent, there's no guarantee that that coherency |
| will be ordered. This means that whilst changes made on one CPU will |
| eventually become visible on all CPUs, there's no guarantee that they will |
| become apparent in the same order on those other CPUs. |
| |
| |
| Consider dealing with a system that has pair of CPUs (1 & 2), each of which has |
| a pair of parallel data caches (CPU 1 has A/B, and CPU 2 has C/D): |
| |
| : |
| : +--------+ |
| : +---------+ | | |
| +--------+ : +--->| Cache A |<------->| | |
| | | : | +---------+ | | |
| | CPU 1 |<---+ | | |
| | | : | +---------+ | | |
| +--------+ : +--->| Cache B |<------->| | |
| : +---------+ | | |
| : | Memory | |
| : +---------+ | System | |
| +--------+ : +--->| Cache C |<------->| | |
| | | : | +---------+ | | |
| | CPU 2 |<---+ | | |
| | | : | +---------+ | | |
| +--------+ : +--->| Cache D |<------->| | |
| : +---------+ | | |
| : +--------+ |
| : |
| |
| Imagine the system has the following properties: |
| |
| (*) an odd-numbered cache line may be in cache A, cache C or it may still be |
| resident in memory; |
| |
| (*) an even-numbered cache line may be in cache B, cache D or it may still be |
| resident in memory; |
| |
| (*) whilst the CPU core is interrogating one cache, the other cache may be |
| making use of the bus to access the rest of the system - perhaps to |
| displace a dirty cacheline or to do a speculative load; |
| |
| (*) each cache has a queue of operations that need to be applied to that cache |
| to maintain coherency with the rest of the system; |
| |
| (*) the coherency queue is not flushed by normal loads to lines already |
| present in the cache, even though the contents of the queue may |
| potentially effect those loads. |
| |
| Imagine, then, that two writes are made on the first CPU, with a write barrier |
| between them to guarantee that they will appear to reach that CPU's caches in |
| the requisite order: |
| |
| CPU 1 CPU 2 COMMENT |
| =============== =============== ======================================= |
| u == 0, v == 1 and p == &u, q == &u |
| v = 2; |
| smp_wmb(); Make sure change to v visible before |
| change to p |
| <A:modify v=2> v is now in cache A exclusively |
| p = &v; |
| <B:modify p=&v> p is now in cache B exclusively |
| |
| The write memory barrier forces the other CPUs in the system to perceive that |
| the local CPU's caches have apparently been updated in the correct order. But |
| now imagine that the second CPU that wants to read those values: |
| |
| CPU 1 CPU 2 COMMENT |
| =============== =============== ======================================= |
| ... |
| q = p; |
| x = *q; |
| |
| The above pair of reads may then fail to happen in expected order, as the |
| cacheline holding p may get updated in one of the second CPU's caches whilst |
| the update to the cacheline holding v is delayed in the other of the second |
| CPU's caches by some other cache event: |
| |
| CPU 1 CPU 2 COMMENT |
| =============== =============== ======================================= |
| u == 0, v == 1 and p == &u, q == &u |
| v = 2; |
| smp_wmb(); |
| <A:modify v=2> <C:busy> |
| <C:queue v=2> |
| p = &b; q = p; |
| <D:request p> |
| <B:modify p=&v> <D:commit p=&v> |
| <D:read p> |
| x = *q; |
| <C:read *q> Reads from v before v updated in cache |
| <C:unbusy> |
| <C:commit v=2> |
| |
| Basically, whilst both cachelines will be updated on CPU 2 eventually, there's |
| no guarantee that, without intervention, the order of update will be the same |
| as that committed on CPU 1. |
| |
| |
| To intervene, we need to interpolate a data dependency barrier or a read |
| barrier between the loads. This will force the cache to commit its coherency |
| queue before processing any further requests: |
| |
| CPU 1 CPU 2 COMMENT |
| =============== =============== ======================================= |
| u == 0, v == 1 and p == &u, q == &u |
| v = 2; |
| smp_wmb(); |
| <A:modify v=2> <C:busy> |
| <C:queue v=2> |
| p = &b; q = p; |
| <D:request p> |
| <B:modify p=&v> <D:commit p=&v> |
| <D:read p> |
| smp_read_barrier_depends() |
| <C:unbusy> |
| <C:commit v=2> |
| x = *q; |
| <C:read *q> Reads from v after v updated in cache |
| |
| |
| This sort of problem can be encountered on DEC Alpha processors as they have a |
| split cache that improves performance by making better use of the data bus. |
| Whilst most CPUs do imply a data dependency barrier on the read when a memory |
| access depends on a read, not all do, so it may not be relied on. |
| |
| Other CPUs may also have split caches, but must coordinate between the various |
| cachelets for normal memory accesss. The semantics of the Alpha removes the |
| need for coordination in absence of memory barriers. |
| |
| |
| CACHE COHERENCY VS DMA |
| ---------------------- |
| |
| Not all systems maintain cache coherency with respect to devices doing DMA. In |
| such cases, a device attempting DMA may obtain stale data from RAM because |
| dirty cache lines may be resident in the caches of various CPUs, and may not |
| have been written back to RAM yet. To deal with this, the appropriate part of |
| the kernel must flush the overlapping bits of cache on each CPU (and maybe |
| invalidate them as well). |
| |
| In addition, the data DMA'd to RAM by a device may be overwritten by dirty |
| cache lines being written back to RAM from a CPU's cache after the device has |
| installed its own data, or cache lines simply present in a CPUs cache may |
| simply obscure the fact that RAM has been updated, until at such time as the |
| cacheline is discarded from the CPU's cache and reloaded. To deal with this, |
| the appropriate part of the kernel must invalidate the overlapping bits of the |
| cache on each CPU. |
| |
| See Documentation/cachetlb.txt for more information on cache management. |
| |
| |
| CACHE COHERENCY VS MMIO |
| ----------------------- |
| |
| Memory mapped I/O usually takes place through memory locations that are part of |
| a window in the CPU's memory space that have different properties assigned than |
| the usual RAM directed window. |
| |
| Amongst these properties is usually the fact that such accesses bypass the |
| caching entirely and go directly to the device buses. This means MMIO accesses |
| may, in effect, overtake accesses to cached memory that were emitted earlier. |
| A memory barrier isn't sufficient in such a case, but rather the cache must be |
| flushed between the cached memory write and the MMIO access if the two are in |
| any way dependent. |
| |
| |
| ========================= |
| THE THINGS CPUS GET UP TO |
| ========================= |
| |
| A programmer might take it for granted that the CPU will perform memory |
| operations in exactly the order specified, so that if a CPU is, for example, |
| given the following piece of code to execute: |
| |
| a = *A; |
| *B = b; |
| c = *C; |
| d = *D; |
| *E = e; |
| |
| They would then expect that the CPU will complete the memory operation for each |
| instruction before moving on to the next one, leading to a definite sequence of |
| operations as seen by external observers in the system: |
| |
| LOAD *A, STORE *B, LOAD *C, LOAD *D, STORE *E. |
| |
| |
| Reality is, of course, much messier. With many CPUs and compilers, the above |
| assumption doesn't hold because: |
| |
| (*) loads are more likely to need to be completed immediately to permit |
| execution progress, whereas stores can often be deferred without a |
| problem; |
| |
| (*) loads may be done speculatively, and the result discarded should it prove |
| to have been unnecessary; |
| |
| (*) loads may be done speculatively, leading to the result having being |
| fetched at the wrong time in the expected sequence of events; |
| |
| (*) the order of the memory accesses may be rearranged to promote better use |
| of the CPU buses and caches; |
| |
| (*) loads and stores may be combined to improve performance when talking to |
| memory or I/O hardware that can do batched accesses of adjacent locations, |
| thus cutting down on transaction setup costs (memory and PCI devices may |
| both be able to do this); and |
| |
| (*) the CPU's data cache may affect the ordering, and whilst cache-coherency |
| mechanisms may alleviate this - once the store has actually hit the cache |
| - there's no guarantee that the coherency management will be propagated in |
| order to other CPUs. |
| |
| So what another CPU, say, might actually observe from the above piece of code |
| is: |
| |
| LOAD *A, ..., LOAD {*C,*D}, STORE *E, STORE *B |
| |
| (Where "LOAD {*C,*D}" is a combined load) |
| |
| |
| However, it is guaranteed that a CPU will be self-consistent: it will see its |
| _own_ accesses appear to be correctly ordered, without the need for a memory |
| barrier. For instance with the following code: |
| |
| U = *A; |
| *A = V; |
| *A = W; |
| X = *A; |
| *A = Y; |
| Z = *A; |
| |
| and assuming no intervention by an external influence, it can be assumed that |
| the final result will appear to be: |
| |
| U == the original value of *A |
| X == W |
| Z == Y |
| *A == Y |
| |
| The code above may cause the CPU to generate the full sequence of memory |
| accesses: |
| |
| U=LOAD *A, STORE *A=V, STORE *A=W, X=LOAD *A, STORE *A=Y, Z=LOAD *A |
| |
| in that order, but, without intervention, the sequence may have almost any |
| combination of elements combined or discarded, provided the program's view of |
| the world remains consistent. |
| |
| The compiler may also combine, discard or defer elements of the sequence before |
| the CPU even sees them. |
| |
| For instance: |
| |
| *A = V; |
| *A = W; |
| |
| may be reduced to: |
| |
| *A = W; |
| |
| since, without a write barrier, it can be assumed that the effect of the |
| storage of V to *A is lost. Similarly: |
| |
| *A = Y; |
| Z = *A; |
| |
| may, without a memory barrier, be reduced to: |
| |
| *A = Y; |
| Z = Y; |
| |
| and the LOAD operation never appear outside of the CPU. |
| |
| |
| AND THEN THERE'S THE ALPHA |
| -------------------------- |
| |
| The DEC Alpha CPU is one of the most relaxed CPUs there is. Not only that, |
| some versions of the Alpha CPU have a split data cache, permitting them to have |
| two semantically related cache lines updating at separate times. This is where |
| the data dependency barrier really becomes necessary as this synchronises both |
| caches with the memory coherence system, thus making it seem like pointer |
| changes vs new data occur in the right order. |
| |
| The Alpha defines the Linux's kernel's memory barrier model. |
| |
| See the subsection on "Cache Coherency" above. |
| |
| |
| ========== |
| REFERENCES |
| ========== |
| |
| Alpha AXP Architecture Reference Manual, Second Edition (Sites & Witek, |
| Digital Press) |
| Chapter 5.2: Physical Address Space Characteristics |
| Chapter 5.4: Caches and Write Buffers |
| Chapter 5.5: Data Sharing |
| Chapter 5.6: Read/Write Ordering |
| |
| AMD64 Architecture Programmer's Manual Volume 2: System Programming |
| Chapter 7.1: Memory-Access Ordering |
| Chapter 7.4: Buffering and Combining Memory Writes |
| |
| IA-32 Intel Architecture Software Developer's Manual, Volume 3: |
| System Programming Guide |
| Chapter 7.1: Locked Atomic Operations |
| Chapter 7.2: Memory Ordering |
| Chapter 7.4: Serializing Instructions |
| |
| The SPARC Architecture Manual, Version 9 |
| Chapter 8: Memory Models |
| Appendix D: Formal Specification of the Memory Models |
| Appendix J: Programming with the Memory Models |
| |
| UltraSPARC Programmer Reference Manual |
| Chapter 5: Memory Accesses and Cacheability |
| Chapter 15: Sparc-V9 Memory Models |
| |
| UltraSPARC III Cu User's Manual |
| Chapter 9: Memory Models |
| |
| UltraSPARC IIIi Processor User's Manual |
| Chapter 8: Memory Models |
| |
| UltraSPARC Architecture 2005 |
| Chapter 9: Memory |
| Appendix D: Formal Specifications of the Memory Models |
| |
| UltraSPARC T1 Supplement to the UltraSPARC Architecture 2005 |
| Chapter 8: Memory Models |
| Appendix F: Caches and Cache Coherency |
| |
| Solaris Internals, Core Kernel Architecture, p63-68: |
| Chapter 3.3: Hardware Considerations for Locks and |
| Synchronization |
| |
| Unix Systems for Modern Architectures, Symmetric Multiprocessing and Caching |
| for Kernel Programmers: |
| Chapter 13: Other Memory Models |
| |
| Intel Itanium Architecture Software Developer's Manual: Volume 1: |
| Section 2.6: Speculation |
| Section 4.4: Memory Access |