blob: 079305640790ee0436ab7ae46e49624498c333ce [file] [log] [blame]
Akinobu Mitade1ba092006-12-08 02:39:42 -08001Fault injection capabilities infrastructure
2===========================================
3
4See also drivers/md/faulty.c and "every_nth" module option for scsi_debug.
5
6
7Available fault injection capabilities
8--------------------------------------
9
10o failslab
11
12 injects slab allocation failures. (kmalloc(), kmem_cache_alloc(), ...)
13
14o fail_page_alloc
15
16 injects page allocation failures. (alloc_pages(), get_free_pages(), ...)
17
18o fail_make_request
19
Don Mullis5d0ffa22006-12-08 02:39:50 -080020 injects disk IO errors on devices permitted by setting
Akinobu Mitade1ba092006-12-08 02:39:42 -080021 /sys/block/<device>/make-it-fail or
22 /sys/block/<device>/<partition>/make-it-fail. (generic_make_request())
23
24Configure fault-injection capabilities behavior
25-----------------------------------------------
26
27o debugfs entries
28
29fault-inject-debugfs kernel module provides some debugfs entries for runtime
30configuration of fault-injection capabilities.
31
GeunSik Lim156f5a72009-06-02 15:01:37 +090032- /sys/kernel/debug/fail*/probability:
Akinobu Mitade1ba092006-12-08 02:39:42 -080033
34 likelihood of failure injection, in percent.
35 Format: <percent>
36
Don Mullis5d0ffa22006-12-08 02:39:50 -080037 Note that one-failure-per-hundred is a very high error rate
38 for some testcases. Consider setting probability=100 and configure
GeunSik Lim156f5a72009-06-02 15:01:37 +090039 /sys/kernel/debug/fail*/interval for such testcases.
Akinobu Mitade1ba092006-12-08 02:39:42 -080040
GeunSik Lim156f5a72009-06-02 15:01:37 +090041- /sys/kernel/debug/fail*/interval:
Akinobu Mitade1ba092006-12-08 02:39:42 -080042
43 specifies the interval between failures, for calls to
44 should_fail() that pass all the other tests.
45
46 Note that if you enable this, by setting interval>1, you will
47 probably want to set probability=100.
48
GeunSik Lim156f5a72009-06-02 15:01:37 +090049- /sys/kernel/debug/fail*/times:
Akinobu Mitade1ba092006-12-08 02:39:42 -080050
51 specifies how many times failures may happen at most.
52 A value of -1 means "no limit".
53
GeunSik Lim156f5a72009-06-02 15:01:37 +090054- /sys/kernel/debug/fail*/space:
Akinobu Mitade1ba092006-12-08 02:39:42 -080055
56 specifies an initial resource "budget", decremented by "size"
57 on each call to should_fail(,size). Failure injection is
58 suppressed until "space" reaches zero.
59
GeunSik Lim156f5a72009-06-02 15:01:37 +090060- /sys/kernel/debug/fail*/verbose
Akinobu Mitade1ba092006-12-08 02:39:42 -080061
62 Format: { 0 | 1 | 2 }
Don Mullis5d0ffa22006-12-08 02:39:50 -080063 specifies the verbosity of the messages when failure is
64 injected. '0' means no messages; '1' will print only a single
65 log line per failure; '2' will print a call trace too -- useful
66 to debug the problems revealed by fault injection.
Akinobu Mitade1ba092006-12-08 02:39:42 -080067
GeunSik Lim156f5a72009-06-02 15:01:37 +090068- /sys/kernel/debug/fail*/task-filter:
Akinobu Mitade1ba092006-12-08 02:39:42 -080069
Don Mullis5d0ffa22006-12-08 02:39:50 -080070 Format: { 'Y' | 'N' }
71 A value of 'N' disables filtering by process (default).
Akinobu Mitade1ba092006-12-08 02:39:42 -080072 Any positive value limits failures to only processes indicated by
73 /proc/<pid>/make-it-fail==1.
74
GeunSik Lim156f5a72009-06-02 15:01:37 +090075- /sys/kernel/debug/fail*/require-start:
76- /sys/kernel/debug/fail*/require-end:
77- /sys/kernel/debug/fail*/reject-start:
78- /sys/kernel/debug/fail*/reject-end:
Akinobu Mitade1ba092006-12-08 02:39:42 -080079
80 specifies the range of virtual addresses tested during
81 stacktrace walking. Failure is injected only if some caller
Akinobu Mita329409a2006-12-08 02:39:48 -080082 in the walked stacktrace lies within the required range, and
83 none lies within the rejected range.
84 Default required range is [0,ULONG_MAX) (whole of virtual address space).
85 Default rejected range is [0,0).
Akinobu Mitade1ba092006-12-08 02:39:42 -080086
GeunSik Lim156f5a72009-06-02 15:01:37 +090087- /sys/kernel/debug/fail*/stacktrace-depth:
Akinobu Mitade1ba092006-12-08 02:39:42 -080088
89 specifies the maximum stacktrace depth walked during search
Don Mullis5d0ffa22006-12-08 02:39:50 -080090 for a caller within [require-start,require-end) OR
91 [reject-start,reject-end).
Akinobu Mitade1ba092006-12-08 02:39:42 -080092
GeunSik Lim156f5a72009-06-02 15:01:37 +090093- /sys/kernel/debug/fail_page_alloc/ignore-gfp-highmem:
Akinobu Mitade1ba092006-12-08 02:39:42 -080094
Don Mullis5d0ffa22006-12-08 02:39:50 -080095 Format: { 'Y' | 'N' }
96 default is 'N', setting it to 'Y' won't inject failures into
Akinobu Mitade1ba092006-12-08 02:39:42 -080097 highmem/user allocations.
98
GeunSik Lim156f5a72009-06-02 15:01:37 +090099- /sys/kernel/debug/failslab/ignore-gfp-wait:
100- /sys/kernel/debug/fail_page_alloc/ignore-gfp-wait:
Akinobu Mitade1ba092006-12-08 02:39:42 -0800101
Don Mullis5d0ffa22006-12-08 02:39:50 -0800102 Format: { 'Y' | 'N' }
103 default is 'N', setting it to 'Y' will inject failures
Akinobu Mitade1ba092006-12-08 02:39:42 -0800104 only into non-sleep allocations (GFP_ATOMIC allocations).
105
GeunSik Lim156f5a72009-06-02 15:01:37 +0900106- /sys/kernel/debug/fail_page_alloc/min-order:
Akinobu Mita54114992007-07-15 23:40:23 -0700107
108 specifies the minimum page allocation order to be injected
109 failures.
110
Akinobu Mitade1ba092006-12-08 02:39:42 -0800111o Boot option
112
113In order to inject faults while debugfs is not available (early boot time),
114use the boot option:
115
116 failslab=
117 fail_page_alloc=
118 fail_make_request=<interval>,<probability>,<space>,<times>
119
120How to add new fault injection capability
121-----------------------------------------
122
123o #include <linux/fault-inject.h>
124
125o define the fault attributes
126
127 DECLARE_FAULT_INJECTION(name);
128
129 Please see the definition of struct fault_attr in fault-inject.h
130 for details.
131
Don Mullis5d0ffa22006-12-08 02:39:50 -0800132o provide a way to configure fault attributes
Akinobu Mitade1ba092006-12-08 02:39:42 -0800133
134- boot option
135
136 If you need to enable the fault injection capability from boot time, you can
Don Mullis5d0ffa22006-12-08 02:39:50 -0800137 provide boot option to configure it. There is a helper function for it:
Akinobu Mitade1ba092006-12-08 02:39:42 -0800138
Don Mullis5d0ffa22006-12-08 02:39:50 -0800139 setup_fault_attr(attr, str);
Akinobu Mitade1ba092006-12-08 02:39:42 -0800140
141- debugfs entries
142
143 failslab, fail_page_alloc, and fail_make_request use this way.
Don Mullis5d0ffa22006-12-08 02:39:50 -0800144 Helper functions:
Akinobu Mitade1ba092006-12-08 02:39:42 -0800145
Don Mullis5d0ffa22006-12-08 02:39:50 -0800146 init_fault_attr_entries(entries, attr, name);
147 void cleanup_fault_attr_entries(entries);
Akinobu Mitade1ba092006-12-08 02:39:42 -0800148
149- module parameters
150
151 If the scope of the fault injection capability is limited to a
152 single kernel module, it is better to provide module parameters to
153 configure the fault attributes.
154
155o add a hook to insert failures
156
Don Mullis5d0ffa22006-12-08 02:39:50 -0800157 Upon should_fail() returning true, client code should inject a failure.
Akinobu Mitade1ba092006-12-08 02:39:42 -0800158
Don Mullis5d0ffa22006-12-08 02:39:50 -0800159 should_fail(attr, size);
Akinobu Mitade1ba092006-12-08 02:39:42 -0800160
161Application Examples
162--------------------
163
Akinobu Mita18584872007-07-15 23:40:24 -0700164o Inject slab allocation failures into module init/exit code
Akinobu Mitade1ba092006-12-08 02:39:42 -0800165
Akinobu Mitade1ba092006-12-08 02:39:42 -0800166#!/bin/bash
167
Akinobu Mita18584872007-07-15 23:40:24 -0700168FAILTYPE=failslab
GeunSik Lim156f5a72009-06-02 15:01:37 +0900169echo Y > /sys/kernel/debug/$FAILTYPE/task-filter
170echo 10 > /sys/kernel/debug/$FAILTYPE/probability
171echo 100 > /sys/kernel/debug/$FAILTYPE/interval
172echo -1 > /sys/kernel/debug/$FAILTYPE/times
173echo 0 > /sys/kernel/debug/$FAILTYPE/space
174echo 2 > /sys/kernel/debug/$FAILTYPE/verbose
175echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait
Akinobu Mitade1ba092006-12-08 02:39:42 -0800176
Akinobu Mita18584872007-07-15 23:40:24 -0700177faulty_system()
Akinobu Mitade1ba092006-12-08 02:39:42 -0800178{
Akinobu Mita18584872007-07-15 23:40:24 -0700179 bash -c "echo 1 > /proc/self/make-it-fail && exec $*"
Akinobu Mitade1ba092006-12-08 02:39:42 -0800180}
181
Akinobu Mita18584872007-07-15 23:40:24 -0700182if [ $# -eq 0 ]
183then
184 echo "Usage: $0 modulename [ modulename ... ]"
185 exit 1
186fi
Akinobu Mitade1ba092006-12-08 02:39:42 -0800187
Akinobu Mita18584872007-07-15 23:40:24 -0700188for m in $*
189do
190 echo inserting $m...
191 faulty_system modprobe $m
Akinobu Mitade1ba092006-12-08 02:39:42 -0800192
Akinobu Mita18584872007-07-15 23:40:24 -0700193 echo removing $m...
194 faulty_system modprobe -r $m
195done
Akinobu Mitade1ba092006-12-08 02:39:42 -0800196
197------------------------------------------------------------------------------
198
Akinobu Mita18584872007-07-15 23:40:24 -0700199o Inject page allocation failures only for a specific module
Akinobu Mitade1ba092006-12-08 02:39:42 -0800200
Akinobu Mitade1ba092006-12-08 02:39:42 -0800201#!/bin/bash
202
Akinobu Mita18584872007-07-15 23:40:24 -0700203FAILTYPE=fail_page_alloc
204module=$1
Akinobu Mitade1ba092006-12-08 02:39:42 -0800205
Akinobu Mita18584872007-07-15 23:40:24 -0700206if [ -z $module ]
207then
208 echo "Usage: $0 <modulename>"
209 exit 1
210fi
Akinobu Mitade1ba092006-12-08 02:39:42 -0800211
Akinobu Mita18584872007-07-15 23:40:24 -0700212modprobe $module
Akinobu Mitade1ba092006-12-08 02:39:42 -0800213
Akinobu Mita18584872007-07-15 23:40:24 -0700214if [ ! -d /sys/module/$module/sections ]
215then
216 echo Module $module is not loaded
217 exit 1
218fi
219
GeunSik Lim156f5a72009-06-02 15:01:37 +0900220cat /sys/module/$module/sections/.text > /sys/kernel/debug/$FAILTYPE/require-start
221cat /sys/module/$module/sections/.data > /sys/kernel/debug/$FAILTYPE/require-end
Akinobu Mita18584872007-07-15 23:40:24 -0700222
GeunSik Lim156f5a72009-06-02 15:01:37 +0900223echo N > /sys/kernel/debug/$FAILTYPE/task-filter
224echo 10 > /sys/kernel/debug/$FAILTYPE/probability
225echo 100 > /sys/kernel/debug/$FAILTYPE/interval
226echo -1 > /sys/kernel/debug/$FAILTYPE/times
227echo 0 > /sys/kernel/debug/$FAILTYPE/space
228echo 2 > /sys/kernel/debug/$FAILTYPE/verbose
229echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait
230echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-highmem
231echo 10 > /sys/kernel/debug/$FAILTYPE/stacktrace-depth
Akinobu Mita18584872007-07-15 23:40:24 -0700232
GeunSik Lim156f5a72009-06-02 15:01:37 +0900233trap "echo 0 > /sys/kernel/debug/$FAILTYPE/probability" SIGINT SIGTERM EXIT
Akinobu Mita18584872007-07-15 23:40:24 -0700234
235echo "Injecting errors into the module $module... (interrupt to stop)"
236sleep 1000000
Akinobu Mitade1ba092006-12-08 02:39:42 -0800237