| Peter Zijlstra | a560aa4 | 2007-10-07 00:24:33 -0700 | [diff] [blame] | 1 | |
| 2 | LOCK STATISTICS |
| 3 | |
| 4 | - WHAT |
| 5 | |
| 6 | As the name suggests, it provides statistics on locks. |
| 7 | |
| 8 | - WHY |
| 9 | |
| 10 | Because things like lock contention can severely impact performance. |
| 11 | |
| 12 | - HOW |
| 13 | |
| 14 | Lockdep already has hooks in the lock functions and maps lock instances to |
| 15 | lock classes. We build on that. The graph below shows the relation between |
| 16 | the lock functions and the various hooks therein. |
| 17 | |
| 18 | __acquire |
| 19 | | |
| 20 | lock _____ |
| 21 | | \ |
| 22 | | __contended |
| 23 | | | |
| 24 | | <wait> |
| 25 | | _______/ |
| 26 | |/ |
| 27 | | |
| 28 | __acquired |
| 29 | | |
| 30 | . |
| 31 | <hold> |
| 32 | . |
| 33 | | |
| 34 | __release |
| 35 | | |
| 36 | unlock |
| 37 | |
| 38 | lock, unlock - the regular lock functions |
| 39 | __* - the hooks |
| 40 | <> - states |
| 41 | |
| 42 | With these hooks we provide the following statistics: |
| 43 | |
| 44 | con-bounces - number of lock contention that involved x-cpu data |
| 45 | contentions - number of lock acquisitions that had to wait |
| 46 | wait time min - shortest (non-0) time we ever had to wait for a lock |
| 47 | max - longest time we ever had to wait for a lock |
| 48 | total - total time we spend waiting on this lock |
| 49 | acq-bounces - number of lock acquisitions that involved x-cpu data |
| 50 | acquisitions - number of times we took the lock |
| 51 | hold time min - shortest (non-0) time we ever held the lock |
| 52 | max - longest time we ever held the lock |
| 53 | total - total time this lock was held |
| 54 | |
| 55 | From these number various other statistics can be derived, such as: |
| 56 | |
| 57 | hold time average = hold time total / acquisitions |
| 58 | |
| 59 | These numbers are gathered per lock class, per read/write state (when |
| 60 | applicable). |
| 61 | |
| 62 | It also tracks 4 contention points per class. A contention point is a call site |
| 63 | that had to wait on lock acquisition. |
| 64 | |
| 65 | - USAGE |
| 66 | |
| 67 | Look at the current lock statistics: |
| 68 | |
| 69 | ( line numbers not part of actual output, done for clarity in the explanation |
| 70 | below ) |
| 71 | |
| 72 | # less /proc/lock_stat |
| 73 | |
| Peter Zijlstra | c7e78cf | 2008-10-16 23:17:09 +0200 | [diff] [blame] | 74 | 01 lock_stat version 0.3 |
| Peter Zijlstra | a560aa4 | 2007-10-07 00:24:33 -0700 | [diff] [blame] | 75 | 02 ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 76 | 03 class name con-bounces contentions waittime-min waittime-max waittime-total acq-bounces acquisitions holdtime-min holdtime-max holdtime-total |
| 77 | 04 ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 78 | 05 |
| Peter Zijlstra | c7e78cf | 2008-10-16 23:17:09 +0200 | [diff] [blame] | 79 | 06 &mm->mmap_sem-W: 233 538 18446744073708 22924.27 607243.51 1342 45806 1.71 8595.89 1180582.34 |
| 80 | 07 &mm->mmap_sem-R: 205 587 18446744073708 28403.36 731975.00 1940 412426 0.58 187825.45 6307502.88 |
| 81 | 08 --------------- |
| 82 | 09 &mm->mmap_sem 487 [<ffffffff8053491f>] do_page_fault+0x466/0x928 |
| 83 | 10 &mm->mmap_sem 179 [<ffffffff802a6200>] sys_mprotect+0xcd/0x21d |
| 84 | 11 &mm->mmap_sem 279 [<ffffffff80210a57>] sys_mmap+0x75/0xce |
| 85 | 12 &mm->mmap_sem 76 [<ffffffff802a490b>] sys_munmap+0x32/0x59 |
| 86 | 13 --------------- |
| 87 | 14 &mm->mmap_sem 270 [<ffffffff80210a57>] sys_mmap+0x75/0xce |
| 88 | 15 &mm->mmap_sem 431 [<ffffffff8053491f>] do_page_fault+0x466/0x928 |
| 89 | 16 &mm->mmap_sem 138 [<ffffffff802a490b>] sys_munmap+0x32/0x59 |
| 90 | 17 &mm->mmap_sem 145 [<ffffffff802a6200>] sys_mprotect+0xcd/0x21d |
| 91 | 18 |
| 92 | 19 ............................................................................................................................................................................................... |
| 93 | 20 |
| 94 | 21 dcache_lock: 621 623 0.52 118.26 1053.02 6745 91930 0.29 316.29 118423.41 |
| 95 | 22 ----------- |
| 96 | 23 dcache_lock 179 [<ffffffff80378274>] _atomic_dec_and_lock+0x34/0x54 |
| 97 | 24 dcache_lock 113 [<ffffffff802cc17b>] d_alloc+0x19a/0x1eb |
| 98 | 25 dcache_lock 99 [<ffffffff802ca0dc>] d_rehash+0x1b/0x44 |
| 99 | 26 dcache_lock 104 [<ffffffff802cbca0>] d_instantiate+0x36/0x8a |
| 100 | 27 ----------- |
| 101 | 28 dcache_lock 192 [<ffffffff80378274>] _atomic_dec_and_lock+0x34/0x54 |
| 102 | 29 dcache_lock 98 [<ffffffff802ca0dc>] d_rehash+0x1b/0x44 |
| 103 | 30 dcache_lock 72 [<ffffffff802cc17b>] d_alloc+0x19a/0x1eb |
| 104 | 31 dcache_lock 112 [<ffffffff802cbca0>] d_instantiate+0x36/0x8a |
| Peter Zijlstra | a560aa4 | 2007-10-07 00:24:33 -0700 | [diff] [blame] | 105 | |
| 106 | This excerpt shows the first two lock class statistics. Line 01 shows the |
| 107 | output version - each time the format changes this will be updated. Line 02-04 |
| Peter Zijlstra | c7e78cf | 2008-10-16 23:17:09 +0200 | [diff] [blame] | 108 | show the header with column descriptions. Lines 05-18 and 20-31 show the actual |
| Peter Zijlstra | a560aa4 | 2007-10-07 00:24:33 -0700 | [diff] [blame] | 109 | statistics. These statistics come in two parts; the actual stats separated by a |
| Peter Zijlstra | c7e78cf | 2008-10-16 23:17:09 +0200 | [diff] [blame] | 110 | short separator (line 08, 13) from the contention points. |
| Peter Zijlstra | a560aa4 | 2007-10-07 00:24:33 -0700 | [diff] [blame] | 111 | |
| Peter Zijlstra | c7e78cf | 2008-10-16 23:17:09 +0200 | [diff] [blame] | 112 | The first lock (05-18) is a read/write lock, and shows two lines above the |
| Peter Zijlstra | a560aa4 | 2007-10-07 00:24:33 -0700 | [diff] [blame] | 113 | short separator. The contention points don't match the column descriptors, |
| Peter Zijlstra | c7e78cf | 2008-10-16 23:17:09 +0200 | [diff] [blame] | 114 | they have two: contentions and [<IP>] symbol. The second set of contention |
| 115 | points are the points we're contending with. |
| Peter Zijlstra | a560aa4 | 2007-10-07 00:24:33 -0700 | [diff] [blame] | 116 | |
| Peter Zijlstra | cdad722 | 2008-09-03 16:00:57 +0200 | [diff] [blame] | 117 | The integer part of the time values is in us. |
| Peter Zijlstra | a560aa4 | 2007-10-07 00:24:33 -0700 | [diff] [blame] | 118 | |
| 119 | View the top contending locks: |
| 120 | |
| 121 | # grep : /proc/lock_stat | head |
| 122 | &inode->i_data.tree_lock-W: 15 21657 0.18 1093295.30 11547131054.85 58 10415 0.16 87.51 6387.60 |
| 123 | &inode->i_data.tree_lock-R: 0 0 0.00 0.00 0.00 23302 231198 0.25 8.45 98023.38 |
| 124 | dcache_lock: 1037 1161 0.38 45.32 774.51 6611 243371 0.15 306.48 77387.24 |
| 125 | &inode->i_mutex: 161 286 18446744073709 62882.54 1244614.55 3653 20598 18446744073709 62318.60 1693822.74 |
| 126 | &zone->lru_lock: 94 94 0.53 7.33 92.10 4366 32690 0.29 59.81 16350.06 |
| 127 | &inode->i_data.i_mmap_lock: 79 79 0.40 3.77 53.03 11779 87755 0.28 116.93 29898.44 |
| 128 | &q->__queue_lock: 48 50 0.52 31.62 86.31 774 13131 0.17 113.08 12277.52 |
| 129 | &rq->rq_lock_key: 43 47 0.74 68.50 170.63 3706 33929 0.22 107.99 17460.62 |
| 130 | &rq->rq_lock_key#2: 39 46 0.75 6.68 49.03 2979 32292 0.17 125.17 17137.63 |
| 131 | tasklist_lock-W: 15 15 1.45 10.87 32.70 1201 7390 0.58 62.55 13648.47 |
| 132 | |
| 133 | Clear the statistics: |
| 134 | |
| 135 | # echo 0 > /proc/lock_stat |