blob: 4cf1a2a6bd7257125e824f930d46c0e70b8fb8e6 [file] [log] [blame]
Akinobu Mitade1ba092006-12-08 02:39:42 -08001Fault injection capabilities infrastructure
2===========================================
3
4See also drivers/md/faulty.c and "every_nth" module option for scsi_debug.
5
6
7Available fault injection capabilities
8--------------------------------------
9
10o failslab
11
12 injects slab allocation failures. (kmalloc(), kmem_cache_alloc(), ...)
13
14o fail_page_alloc
15
16 injects page allocation failures. (alloc_pages(), get_free_pages(), ...)
17
18o fail_make_request
19
Don Mullis5d0ffa22006-12-08 02:39:50 -080020 injects disk IO errors on devices permitted by setting
Akinobu Mitade1ba092006-12-08 02:39:42 -080021 /sys/block/<device>/make-it-fail or
22 /sys/block/<device>/<partition>/make-it-fail. (generic_make_request())
23
Per Forlin1e4cb222011-08-19 14:52:38 +020024o fail_mmc_request
25
26 injects MMC data errors on devices permitted by setting
27 debugfs entries under /sys/kernel/debug/mmc0/fail_mmc_request
28
Akinobu Mitade1ba092006-12-08 02:39:42 -080029Configure fault-injection capabilities behavior
30-----------------------------------------------
31
32o debugfs entries
33
34fault-inject-debugfs kernel module provides some debugfs entries for runtime
35configuration of fault-injection capabilities.
36
GeunSik Lim156f5a72009-06-02 15:01:37 +090037- /sys/kernel/debug/fail*/probability:
Akinobu Mitade1ba092006-12-08 02:39:42 -080038
39 likelihood of failure injection, in percent.
40 Format: <percent>
41
Don Mullis5d0ffa22006-12-08 02:39:50 -080042 Note that one-failure-per-hundred is a very high error rate
43 for some testcases. Consider setting probability=100 and configure
GeunSik Lim156f5a72009-06-02 15:01:37 +090044 /sys/kernel/debug/fail*/interval for such testcases.
Akinobu Mitade1ba092006-12-08 02:39:42 -080045
GeunSik Lim156f5a72009-06-02 15:01:37 +090046- /sys/kernel/debug/fail*/interval:
Akinobu Mitade1ba092006-12-08 02:39:42 -080047
48 specifies the interval between failures, for calls to
49 should_fail() that pass all the other tests.
50
51 Note that if you enable this, by setting interval>1, you will
52 probably want to set probability=100.
53
GeunSik Lim156f5a72009-06-02 15:01:37 +090054- /sys/kernel/debug/fail*/times:
Akinobu Mitade1ba092006-12-08 02:39:42 -080055
56 specifies how many times failures may happen at most.
57 A value of -1 means "no limit".
58
GeunSik Lim156f5a72009-06-02 15:01:37 +090059- /sys/kernel/debug/fail*/space:
Akinobu Mitade1ba092006-12-08 02:39:42 -080060
61 specifies an initial resource "budget", decremented by "size"
62 on each call to should_fail(,size). Failure injection is
63 suppressed until "space" reaches zero.
64
GeunSik Lim156f5a72009-06-02 15:01:37 +090065- /sys/kernel/debug/fail*/verbose
Akinobu Mitade1ba092006-12-08 02:39:42 -080066
67 Format: { 0 | 1 | 2 }
Don Mullis5d0ffa22006-12-08 02:39:50 -080068 specifies the verbosity of the messages when failure is
69 injected. '0' means no messages; '1' will print only a single
70 log line per failure; '2' will print a call trace too -- useful
71 to debug the problems revealed by fault injection.
Akinobu Mitade1ba092006-12-08 02:39:42 -080072
GeunSik Lim156f5a72009-06-02 15:01:37 +090073- /sys/kernel/debug/fail*/task-filter:
Akinobu Mitade1ba092006-12-08 02:39:42 -080074
Don Mullis5d0ffa22006-12-08 02:39:50 -080075 Format: { 'Y' | 'N' }
76 A value of 'N' disables filtering by process (default).
Akinobu Mitade1ba092006-12-08 02:39:42 -080077 Any positive value limits failures to only processes indicated by
78 /proc/<pid>/make-it-fail==1.
79
GeunSik Lim156f5a72009-06-02 15:01:37 +090080- /sys/kernel/debug/fail*/require-start:
81- /sys/kernel/debug/fail*/require-end:
82- /sys/kernel/debug/fail*/reject-start:
83- /sys/kernel/debug/fail*/reject-end:
Akinobu Mitade1ba092006-12-08 02:39:42 -080084
85 specifies the range of virtual addresses tested during
86 stacktrace walking. Failure is injected only if some caller
Akinobu Mita329409a2006-12-08 02:39:48 -080087 in the walked stacktrace lies within the required range, and
88 none lies within the rejected range.
89 Default required range is [0,ULONG_MAX) (whole of virtual address space).
90 Default rejected range is [0,0).
Akinobu Mitade1ba092006-12-08 02:39:42 -080091
GeunSik Lim156f5a72009-06-02 15:01:37 +090092- /sys/kernel/debug/fail*/stacktrace-depth:
Akinobu Mitade1ba092006-12-08 02:39:42 -080093
94 specifies the maximum stacktrace depth walked during search
Don Mullis5d0ffa22006-12-08 02:39:50 -080095 for a caller within [require-start,require-end) OR
96 [reject-start,reject-end).
Akinobu Mitade1ba092006-12-08 02:39:42 -080097
GeunSik Lim156f5a72009-06-02 15:01:37 +090098- /sys/kernel/debug/fail_page_alloc/ignore-gfp-highmem:
Akinobu Mitade1ba092006-12-08 02:39:42 -080099
Don Mullis5d0ffa22006-12-08 02:39:50 -0800100 Format: { 'Y' | 'N' }
101 default is 'N', setting it to 'Y' won't inject failures into
Akinobu Mitade1ba092006-12-08 02:39:42 -0800102 highmem/user allocations.
103
GeunSik Lim156f5a72009-06-02 15:01:37 +0900104- /sys/kernel/debug/failslab/ignore-gfp-wait:
105- /sys/kernel/debug/fail_page_alloc/ignore-gfp-wait:
Akinobu Mitade1ba092006-12-08 02:39:42 -0800106
Don Mullis5d0ffa22006-12-08 02:39:50 -0800107 Format: { 'Y' | 'N' }
108 default is 'N', setting it to 'Y' will inject failures
Akinobu Mitade1ba092006-12-08 02:39:42 -0800109 only into non-sleep allocations (GFP_ATOMIC allocations).
110
GeunSik Lim156f5a72009-06-02 15:01:37 +0900111- /sys/kernel/debug/fail_page_alloc/min-order:
Akinobu Mita54114992007-07-15 23:40:23 -0700112
113 specifies the minimum page allocation order to be injected
114 failures.
115
Akinobu Mitade1ba092006-12-08 02:39:42 -0800116o Boot option
117
118In order to inject faults while debugfs is not available (early boot time),
119use the boot option:
120
121 failslab=
122 fail_page_alloc=
Per Forlin1e4cb222011-08-19 14:52:38 +0200123 fail_make_request=
Per Forlin199e3f42011-09-13 23:03:30 +0200124 mmc_core.fail_request=<interval>,<probability>,<space>,<times>
Akinobu Mitade1ba092006-12-08 02:39:42 -0800125
126How to add new fault injection capability
127-----------------------------------------
128
129o #include <linux/fault-inject.h>
130
131o define the fault attributes
132
133 DECLARE_FAULT_INJECTION(name);
134
135 Please see the definition of struct fault_attr in fault-inject.h
136 for details.
137
Don Mullis5d0ffa22006-12-08 02:39:50 -0800138o provide a way to configure fault attributes
Akinobu Mitade1ba092006-12-08 02:39:42 -0800139
140- boot option
141
142 If you need to enable the fault injection capability from boot time, you can
Don Mullis5d0ffa22006-12-08 02:39:50 -0800143 provide boot option to configure it. There is a helper function for it:
Akinobu Mitade1ba092006-12-08 02:39:42 -0800144
Don Mullis5d0ffa22006-12-08 02:39:50 -0800145 setup_fault_attr(attr, str);
Akinobu Mitade1ba092006-12-08 02:39:42 -0800146
147- debugfs entries
148
149 failslab, fail_page_alloc, and fail_make_request use this way.
Don Mullis5d0ffa22006-12-08 02:39:50 -0800150 Helper functions:
Akinobu Mitade1ba092006-12-08 02:39:42 -0800151
Akinobu Mitadd48c082011-08-03 16:21:01 -0700152 fault_create_debugfs_attr(name, parent, attr);
Akinobu Mitade1ba092006-12-08 02:39:42 -0800153
154- module parameters
155
156 If the scope of the fault injection capability is limited to a
157 single kernel module, it is better to provide module parameters to
158 configure the fault attributes.
159
160o add a hook to insert failures
161
Don Mullis5d0ffa22006-12-08 02:39:50 -0800162 Upon should_fail() returning true, client code should inject a failure.
Akinobu Mitade1ba092006-12-08 02:39:42 -0800163
Don Mullis5d0ffa22006-12-08 02:39:50 -0800164 should_fail(attr, size);
Akinobu Mitade1ba092006-12-08 02:39:42 -0800165
166Application Examples
167--------------------
168
Akinobu Mita18584872007-07-15 23:40:24 -0700169o Inject slab allocation failures into module init/exit code
Akinobu Mitade1ba092006-12-08 02:39:42 -0800170
Akinobu Mitade1ba092006-12-08 02:39:42 -0800171#!/bin/bash
172
Akinobu Mita18584872007-07-15 23:40:24 -0700173FAILTYPE=failslab
GeunSik Lim156f5a72009-06-02 15:01:37 +0900174echo Y > /sys/kernel/debug/$FAILTYPE/task-filter
175echo 10 > /sys/kernel/debug/$FAILTYPE/probability
176echo 100 > /sys/kernel/debug/$FAILTYPE/interval
177echo -1 > /sys/kernel/debug/$FAILTYPE/times
178echo 0 > /sys/kernel/debug/$FAILTYPE/space
179echo 2 > /sys/kernel/debug/$FAILTYPE/verbose
180echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait
Akinobu Mitade1ba092006-12-08 02:39:42 -0800181
Akinobu Mita18584872007-07-15 23:40:24 -0700182faulty_system()
Akinobu Mitade1ba092006-12-08 02:39:42 -0800183{
Akinobu Mita18584872007-07-15 23:40:24 -0700184 bash -c "echo 1 > /proc/self/make-it-fail && exec $*"
Akinobu Mitade1ba092006-12-08 02:39:42 -0800185}
186
Akinobu Mita18584872007-07-15 23:40:24 -0700187if [ $# -eq 0 ]
188then
189 echo "Usage: $0 modulename [ modulename ... ]"
190 exit 1
191fi
Akinobu Mitade1ba092006-12-08 02:39:42 -0800192
Akinobu Mita18584872007-07-15 23:40:24 -0700193for m in $*
194do
195 echo inserting $m...
196 faulty_system modprobe $m
Akinobu Mitade1ba092006-12-08 02:39:42 -0800197
Akinobu Mita18584872007-07-15 23:40:24 -0700198 echo removing $m...
199 faulty_system modprobe -r $m
200done
Akinobu Mitade1ba092006-12-08 02:39:42 -0800201
202------------------------------------------------------------------------------
203
Akinobu Mita18584872007-07-15 23:40:24 -0700204o Inject page allocation failures only for a specific module
Akinobu Mitade1ba092006-12-08 02:39:42 -0800205
Akinobu Mitade1ba092006-12-08 02:39:42 -0800206#!/bin/bash
207
Akinobu Mita18584872007-07-15 23:40:24 -0700208FAILTYPE=fail_page_alloc
209module=$1
Akinobu Mitade1ba092006-12-08 02:39:42 -0800210
Akinobu Mita18584872007-07-15 23:40:24 -0700211if [ -z $module ]
212then
213 echo "Usage: $0 <modulename>"
214 exit 1
215fi
Akinobu Mitade1ba092006-12-08 02:39:42 -0800216
Akinobu Mita18584872007-07-15 23:40:24 -0700217modprobe $module
Akinobu Mitade1ba092006-12-08 02:39:42 -0800218
Akinobu Mita18584872007-07-15 23:40:24 -0700219if [ ! -d /sys/module/$module/sections ]
220then
221 echo Module $module is not loaded
222 exit 1
223fi
224
GeunSik Lim156f5a72009-06-02 15:01:37 +0900225cat /sys/module/$module/sections/.text > /sys/kernel/debug/$FAILTYPE/require-start
226cat /sys/module/$module/sections/.data > /sys/kernel/debug/$FAILTYPE/require-end
Akinobu Mita18584872007-07-15 23:40:24 -0700227
GeunSik Lim156f5a72009-06-02 15:01:37 +0900228echo N > /sys/kernel/debug/$FAILTYPE/task-filter
229echo 10 > /sys/kernel/debug/$FAILTYPE/probability
230echo 100 > /sys/kernel/debug/$FAILTYPE/interval
231echo -1 > /sys/kernel/debug/$FAILTYPE/times
232echo 0 > /sys/kernel/debug/$FAILTYPE/space
233echo 2 > /sys/kernel/debug/$FAILTYPE/verbose
234echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait
235echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-highmem
236echo 10 > /sys/kernel/debug/$FAILTYPE/stacktrace-depth
Akinobu Mita18584872007-07-15 23:40:24 -0700237
GeunSik Lim156f5a72009-06-02 15:01:37 +0900238trap "echo 0 > /sys/kernel/debug/$FAILTYPE/probability" SIGINT SIGTERM EXIT
Akinobu Mita18584872007-07-15 23:40:24 -0700239
240echo "Injecting errors into the module $module... (interrupt to stop)"
241sleep 1000000
Akinobu Mitade1ba092006-12-08 02:39:42 -0800242
Akinobu Mitac24aa642012-07-30 14:43:20 -0700243Tool to run command with failslab or fail_page_alloc
244----------------------------------------------------
245In order to make it easier to accomplish the tasks mentioned above, we can use
246tools/testing/fault-injection/failcmd.sh. Please run a command
247"./tools/testing/fault-injection/failcmd.sh --help" for more information and
248see the following examples.
249
250Examples:
251
252Run a command "make -C tools/testing/selftests/ run_tests" with injecting slab
253allocation failure.
254
255 # ./tools/testing/fault-injection/failcmd.sh \
256 -- make -C tools/testing/selftests/ run_tests
257
258Same as above except to specify 100 times failures at most instead of one time
259at most by default.
260
261 # ./tools/testing/fault-injection/failcmd.sh --times=100 \
262 -- make -C tools/testing/selftests/ run_tests
263
264Same as above except to inject page allocation failure instead of slab
265allocation failure.
266
267 # env FAILCMD_TYPE=fail_page_alloc \
268 ./tools/testing/fault-injection/failcmd.sh --times=100 \
269 -- make -C tools/testing/selftests/ run_tests