Howard McLauchlan | ef4154b | 2018-03-16 16:50:26 -0700 | [diff] [blame] | 1 | Some examples for inject |
| 2 | |
| 3 | inject guarantees the appropriate erroneous return of the specified injection |
Howard McLauchlan | ac7c154 | 2018-03-21 15:53:24 -0700 | [diff] [blame] | 4 | mode (kmalloc,bio,etc) given a call chain and an optional set of predicates. You |
Howard McLauchlan | ef4154b | 2018-03-16 16:50:26 -0700 | [diff] [blame] | 5 | can also optionally print out the generated BPF program for |
| 6 | modification/debugging purposes. |
| 7 | |
Howard McLauchlan | 4c9305c | 2018-04-10 13:22:00 -0700 | [diff] [blame] | 8 | As a simple example, let's say you wanted to fail all mounts. As of 4.17 we can |
| 9 | fail syscalls directly, so let's do that: |
Howard McLauchlan | ef4154b | 2018-03-16 16:50:26 -0700 | [diff] [blame] | 10 | |
Howard McLauchlan | 4c9305c | 2018-04-10 13:22:00 -0700 | [diff] [blame] | 11 | # ./inject.py kmalloc -v 'SyS_mount()' |
Howard McLauchlan | ef4154b | 2018-03-16 16:50:26 -0700 | [diff] [blame] | 12 | |
Howard McLauchlan | ac7c154 | 2018-03-21 15:53:24 -0700 | [diff] [blame] | 13 | The first argument indicates the mode (or what to fail). Appropriate headers are |
Howard McLauchlan | 4c9305c | 2018-04-10 13:22:00 -0700 | [diff] [blame] | 14 | specified, if necessary. The verbosity flag prints the generated program. Note |
| 15 | that some syscalls will be available as 'SyS_xyz' and some will be available as |
| 16 | 'sys_xyz'. This is largely dependent on the number of arguments each syscall |
| 17 | takes. |
Howard McLauchlan | ef4154b | 2018-03-16 16:50:26 -0700 | [diff] [blame] | 18 | |
Howard McLauchlan | ac7c154 | 2018-03-21 15:53:24 -0700 | [diff] [blame] | 19 | Trying to mount various filesystems will fail and report an inability to |
| 20 | allocate memory, as expected. |
| 21 | |
| 22 | Whenever a predicate is missing, an implicit "(true)" is inserted. The example |
| 23 | above can be explicitly written as: |
| 24 | |
Howard McLauchlan | 4c9305c | 2018-04-10 13:22:00 -0700 | [diff] [blame] | 25 | # ./inject.py kmalloc -v '(true) => SyS_mount()(true)' |
Howard McLauchlan | ac7c154 | 2018-03-21 15:53:24 -0700 | [diff] [blame] | 26 | |
| 27 | The "(true)" without an associated function is a predicate for the error |
| 28 | injection mechanism of the current mode. In the case of kmalloc, the predicate |
| 29 | would have access to the arguments of: |
| 30 | |
| 31 | int should_failslab(struct kmem_cache *s, gfp_t gfpflags); |
| 32 | |
| 33 | The bio mode works similarly, with access to the arguments of: |
| 34 | |
| 35 | static noinline int should_fail_bio(struct bio *bio) |
| 36 | |
| 37 | We also note that it's unnecessary to state the arguments of the function if you |
| 38 | have no intention to reference them in the associated predicate. |
| 39 | |
| 40 | Now let's say we want to be a bit more specific; suppose you want to fail |
| 41 | kmalloc() from mount_subtree() when called from btrfs_mount(). This will fail |
| 42 | only btrfs mounts: |
| 43 | |
Howard McLauchlan | 601d75d | 2018-03-21 16:27:18 -0700 | [diff] [blame] | 44 | # ./inject.py kmalloc -v 'mount_subtree() => btrfs_mount()' |
Howard McLauchlan | ac7c154 | 2018-03-21 15:53:24 -0700 | [diff] [blame] | 45 | |
| 46 | Attempting to mount btrfs filesystem during the execution of this command will |
| 47 | yield an error, but other filesystems will be fine. |
Howard McLauchlan | ef4154b | 2018-03-16 16:50:26 -0700 | [diff] [blame] | 48 | |
| 49 | Next, lets say we want to hit one of the BUG_ONs in fs/btrfs. As of 4.16-rc3, |
| 50 | there is a BUG_ON in btrfs_prepare_close_one_device() at fs/btrfs/volumes.c:1002 |
| 51 | |
| 52 | To hit this, we can use the following: |
| 53 | |
Howard McLauchlan | 601d75d | 2018-03-21 16:27:18 -0700 | [diff] [blame] | 54 | # ./inject.py kmalloc -v 'btrfs_alloc_device() => btrfs_close_devices()' |
Howard McLauchlan | ef4154b | 2018-03-16 16:50:26 -0700 | [diff] [blame] | 55 | |
| 56 | While the script was executing, I mounted and unmounted btrfs, causing a |
| 57 | segfault on umount(since that satisfied the call path indicated). A look at |
Howard McLauchlan | ac7c154 | 2018-03-21 15:53:24 -0700 | [diff] [blame] | 58 | dmesg will confirm that the erroneous return value injected by the script |
| 59 | tripped the BUG_ON, causing a segfault down the line. |
Howard McLauchlan | ef4154b | 2018-03-16 16:50:26 -0700 | [diff] [blame] | 60 | |
| 61 | In general, it's worth noting that the required specificity of the call chain is |
| 62 | dependent on how much granularity you need. The example above might have |
| 63 | performed as expected without the intermediate btrfs_alloc_device, but might |
| 64 | have also done something unexpected(an earlier kmalloc could have failed before |
| 65 | the one we were targetting). |
| 66 | |
| 67 | For hot paths, the approach outlined above isn't enough. If a path is traversed |
| 68 | very often, we can distinguish distinct calls with function arguments. Let's say |
| 69 | we want to fail the dentry allocation of a file creatively named 'bananas'. We |
| 70 | can do the following: |
| 71 | |
Howard McLauchlan | 601d75d | 2018-03-21 16:27:18 -0700 | [diff] [blame] | 72 | # ./inject.py kmalloc -v 'd_alloc_parallel(struct dentry *parent, const struct |
| 73 | qstr *name)(STRCMP(name->name, 'bananas'))' |
Howard McLauchlan | ef4154b | 2018-03-16 16:50:26 -0700 | [diff] [blame] | 74 | |
| 75 | While this script is executing, any operation that would cause a dentry |
| 76 | allocation where the name is 'bananas' fails, as expected. |
| 77 | |
Howard McLauchlan | ac7c154 | 2018-03-21 15:53:24 -0700 | [diff] [blame] | 78 | Here, since we're referencing a function argument in our predicate, we need to |
| 79 | provide the function signature up to the argument we're using. |
| 80 | |
Howard McLauchlan | ef4154b | 2018-03-16 16:50:26 -0700 | [diff] [blame] | 81 | To note, STRCMP is a workaround for some rewriter issues. It will take input of |
| 82 | the form (x->...->z, 'literal'), and generate some equivalent code that the |
| 83 | verifier is more friendly about. It's not horribly robust, but works for the |
| 84 | purposes of making string comparisons a bit easier. |
| 85 | |
| 86 | Finally, we briefly demonstrate how to inject bio failures. The mechanism is |
| 87 | identical, so any information from above will apply. |
| 88 | |
| 89 | Let's say we want to fail bio requests when the request is to some specific |
| 90 | sector. An example use case would be to fail superblock writes in btrfs. For |
| 91 | btrfs, we know that there must be a superblock at 65536 bytes, or sector 128. |
| 92 | This allows us to run the following: |
| 93 | |
| 94 | # ./inject.py bio -v -I 'linux/blkdev.h' '(({struct gendisk *d = bio->bi_disk; |
| 95 | struct disk_part_tbl *tbl = d->part_tbl; struct hd_struct **parts = (void *)tbl + |
| 96 | sizeof(struct disk_part_tbl); struct hd_struct **partp = parts + bio->bi_partno; |
| 97 | struct hd_struct *p = *partp; dev_t disk = p->__dev.devt; disk == |
| 98 | MKDEV(254,16);}) && bio->bi_iter.bi_sector == 128)' |
| 99 | |
| 100 | The predicate in the command above has two parts. The first is a compound |
| 101 | statement which shortens to "only if the system is btrfs", but is long due |
| 102 | to rewriter/verifier shenanigans. The major/minor information can be found |
| 103 | however; I used Python. The second part simply checks the starting |
| 104 | address of bi_iter. While executing, this script effectively fails superblock |
| 105 | writes to the superblock at sector 128 without affecting other filesystems. |
| 106 | |
| 107 | As an extension to the above, one could easily fail all btrfs superblock writes |
| 108 | (we only fail the primary) by calculating the sector number of the mirrors and |
| 109 | amending the predicate accordingly. |
Howard McLauchlan | ac7c154 | 2018-03-21 15:53:24 -0700 | [diff] [blame] | 110 | |
Howard McLauchlan | 4c9305c | 2018-04-10 13:22:00 -0700 | [diff] [blame] | 111 | Inject also provides a probability option; this allows you to fail the |
| 112 | path+predicates some percentage of the time. For example, let's say we want to |
| 113 | fail our mounts half the time: |
Howard McLauchlan | ac7c154 | 2018-03-21 15:53:24 -0700 | [diff] [blame] | 114 | |
Howard McLauchlan | 4c9305c | 2018-04-10 13:22:00 -0700 | [diff] [blame] | 115 | # ./inject.py kmalloc -v -P 0.01 'SyS_mount()' |
| 116 | |
| 117 | USAGE message: |
| 118 | usage: inject.py [-h] [-I header] [-P probability] [-v] mode spec |
Howard McLauchlan | ac7c154 | 2018-03-21 15:53:24 -0700 | [diff] [blame] | 119 | |
| 120 | Fail specified kernel functionality when call chain and predicates are met |
| 121 | |
| 122 | positional arguments: |
| 123 | mode indicate which base kernel function to fail |
| 124 | spec specify call chain |
| 125 | |
| 126 | optional arguments: |
| 127 | -h, --help show this help message and exit |
| 128 | -I header, --include header |
| 129 | additional header files to include in the BPF program |
Howard McLauchlan | 4c9305c | 2018-04-10 13:22:00 -0700 | [diff] [blame] | 130 | -P probability, --probability probability |
| 131 | probability that this call chain will fail |
Howard McLauchlan | ac7c154 | 2018-03-21 15:53:24 -0700 | [diff] [blame] | 132 | -v, --verbose print BPF program |