Wyatt Hepler | f9fb90f | 2020-09-30 18:59:33 -0700 | [diff] [blame] | 1 | .. _module-pw_fuzzer: |
Alexei Frolov | 199045a | 2020-08-28 13:02:30 -0700 | [diff] [blame] | 2 | |
Aaron Green | f3c3d2b | 2020-04-02 23:12:31 -0700 | [diff] [blame] | 3 | --------- |
| 4 | pw_fuzzer |
| 5 | --------- |
| 6 | ``pw_fuzzer`` provides developers with tools to write `libFuzzer`_ based |
| 7 | fuzzers. |
| 8 | |
| 9 | Fuzzing or fuzz testing is style of testing that stochastically generates inputs |
| 10 | to targeted interfaces in order to automatically find defects and/or |
| 11 | vulnerabilities. In other words, fuzzing is simply an automated way of testing |
| 12 | APIs with generated data. |
| 13 | |
| 14 | A fuzzer is a program that is used to fuzz a interface. It typically has three |
| 15 | steps that it executes repeatedly: |
| 16 | |
| 17 | #. Generate a new, context-free input. This is the *fuzzing engine*. For |
| 18 | ``pw_fuzzer``, this is `libFuzzer`_. |
| 19 | #. Use the input to exercise the targeted interface, or code being tested. This |
| 20 | is the *fuzz target function*. For ``pw_fuzzer``, these are the GN |
| 21 | ``sources`` and/or ``deps`` that define `LLVMFuzzerTestOneInput`_. |
| 22 | #. Monitor the code being tested for any abnormal conditions. This is the |
| 23 | *instrumentation*. For ``pw_fuzzer``, these are sanitizer runtimes from |
| 24 | LLVM's `compiler_rt`_. |
| 25 | |
| 26 | .. note:: |
| 27 | |
Ali Zhang | 5876558 | 2021-01-29 12:04:06 -0800 | [diff] [blame] | 28 | ``pw_fuzzer`` is currently only supported on Linux and MacOS using clang. |
Aaron Green | f3c3d2b | 2020-04-02 23:12:31 -0700 | [diff] [blame] | 29 | |
| 30 | .. image:: doc_resources/pw_fuzzer_coverage_guided.png |
| 31 | :alt: Coverage Guided Fuzzing with libFuzzer |
| 32 | :align: left |
| 33 | |
| 34 | Writing fuzzers |
| 35 | =============== |
| 36 | |
| 37 | To write a fuzzer, a developer needs to write a fuzz target function follwing |
| 38 | the `fuzz target function`__ guidelines given by libFuzzer: |
| 39 | |
Ted Pudlik | 1b69a4e | 2021-11-13 00:30:12 +0000 | [diff] [blame] | 40 | .. code:: cpp |
Aaron Green | f3c3d2b | 2020-04-02 23:12:31 -0700 | [diff] [blame] | 41 | |
| 42 | extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) { |
| 43 | DoSomethingInterestingWithMyAPI(data, size); |
| 44 | return 0; // Non-zero return values are reserved for future use. |
| 45 | } |
| 46 | |
| 47 | .. __: LLVMFuzzerTestOneInput_ |
| 48 | |
| 49 | When writing you fuzz target function, you may want to consider: |
| 50 | |
Shiva Rajagopal | 9e51656 | 2021-05-11 17:04:15 -0700 | [diff] [blame] | 51 | - It is acceptable to return early if the input doesn't meet some constraints, |
Aaron Green | f3c3d2b | 2020-04-02 23:12:31 -0700 | [diff] [blame] | 52 | e.g. it is too short. |
| 53 | - If your fuzzer accepts data with a well-defined format, you can bootstrap |
| 54 | coverage by crafting examples and adding them to a `corpus`_. |
| 55 | - There are tools to `split a fuzzing input`_ into multiple fields if needed; |
| 56 | the `FuzzedDataProvider`_ is particularly easy to use. |
| 57 | - If your code acts on "transformed" inputs, such as encoded or compressed |
| 58 | inputs, you may want to try `structure aware fuzzing`. |
| 59 | - You can do `startup initialization`_ if you need to. |
| 60 | - If your code is non-deterministic or uses checksums, you may want to disable |
| 61 | those **only** when fuzzing by using LLVM's |
| 62 | `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION`_ |
| 63 | |
Aaron Green | e8449d1 | 2020-04-06 19:24:30 -0700 | [diff] [blame] | 64 | .. _build: |
| 65 | |
Nathaniel Brough | b6b90d0 | 2021-05-11 11:32:17 +0800 | [diff] [blame] | 66 | Building fuzzers with GN |
| 67 | ======================== |
Aaron Green | f3c3d2b | 2020-04-02 23:12:31 -0700 | [diff] [blame] | 68 | |
| 69 | To build a fuzzer, do the following: |
| 70 | |
| 71 | 1. Add the GN target using ``pw_fuzzer`` GN template, and add it to your the |
| 72 | test group of the module: |
| 73 | |
| 74 | .. code:: |
| 75 | |
| 76 | # In $dir_my_module/BUILD.gn |
| 77 | import("$dir_pw_fuzzer/fuzzer.gni") |
| 78 | |
| 79 | pw_fuzzer("my_fuzzer") { |
| 80 | sources = [ "my_fuzzer.cc" ] |
| 81 | deps = [ ":my_lib" ] |
| 82 | } |
| 83 | |
| 84 | pw_test_group("tests") { |
| 85 | tests = [ |
| 86 | ":existing_tests", ... |
| 87 | ":my_fuzzer", # <- Added! |
| 88 | ] |
| 89 | } |
| 90 | |
Ali Zhang | 5876558 | 2021-01-29 12:04:06 -0800 | [diff] [blame] | 91 | 2. Select your choice of sanitizers ("address" is also the current default). |
| 92 | See LLVM for `valid options`_. |
Aaron Green | f3c3d2b | 2020-04-02 23:12:31 -0700 | [diff] [blame] | 93 | |
| 94 | .. code:: sh |
| 95 | |
Ali Zhang | 5876558 | 2021-01-29 12:04:06 -0800 | [diff] [blame] | 96 | $ gn gen out --args='pw_toolchain_SANITIZERS=["address"]' |
Aaron Green | f3c3d2b | 2020-04-02 23:12:31 -0700 | [diff] [blame] | 97 | |
| 98 | 3. Build normally, e.g. using ``pw watch``. |
| 99 | |
Aaron Green | e8449d1 | 2020-04-06 19:24:30 -0700 | [diff] [blame] | 100 | .. _run: |
| 101 | |
Nathaniel Brough | b6b90d0 | 2021-05-11 11:32:17 +0800 | [diff] [blame] | 102 | Building and running fuzzers with Bazel |
| 103 | ======================================= |
| 104 | To build a fuzzer, do the following: |
| 105 | |
| 106 | 1. Add the Bazel target using ``pw_cc_fuzz_test`` macro. |
| 107 | |
| 108 | .. code:: py |
| 109 | |
| 110 | load("@pigweed//pw_fuzzer:fuzzer.bzl", "pw_cc_fuzz_test") |
| 111 | |
| 112 | pw_cc_fuzz_test( |
| 113 | name = "my_fuzz_test", |
| 114 | srcs = ["my_fuzzer.cc"], |
| 115 | deps = [ |
| 116 | "@pigweed//pw_fuzzer", |
| 117 | ":my_lib", |
| 118 | ], |
| 119 | ) |
| 120 | |
| 121 | 2. Build and run the fuzzer. |
| 122 | |
| 123 | .. code:: sh |
| 124 | |
| 125 | bazel test //my_module:my_fuzz_test |
| 126 | |
| 127 | 3. Swap fuzzer backend to use ASAN fuzzing engine. |
| 128 | |
| 129 | .. code:: |
| 130 | |
| 131 | # .bazelrc |
| 132 | # Define the --config=asan-libfuzzer configuration. |
| 133 | build:asan-libfuzzer \ |
| 134 | --@rules_fuzzing//fuzzing:cc_engine=@rules_fuzzing//fuzzing/engines:libfuzzer |
| 135 | build:asan-libfuzzer \ |
| 136 | --@rules_fuzzing//fuzzing:cc_engine_instrumentation=libfuzzer |
| 137 | build:asan-libfuzzer --@rules_fuzzing//fuzzing:cc_engine_sanitizer=asan |
| 138 | |
| 139 | 4. Re-run fuzz tests. |
| 140 | |
| 141 | .. code:: |
| 142 | |
| 143 | bazel test //my_module:my_fuzz_test --config asan-libfuzzer |
| 144 | |
Aaron Green | e8449d1 | 2020-04-06 19:24:30 -0700 | [diff] [blame] | 145 | Running fuzzers locally |
| 146 | ======================= |
Aaron Green | f3c3d2b | 2020-04-02 23:12:31 -0700 | [diff] [blame] | 147 | |
| 148 | Based on the example above, the fuzzer output will be at |
| 149 | ``out/host/obj/my_module/my_fuzzer``. It can be invoked using the normal |
| 150 | `libFuzzer options`_ and `sanitizer runtime flags`_. For even more details, see |
| 151 | the libFuzzer section on `running a fuzzer`_. |
| 152 | |
| 153 | For example, the following invocation disables "one definition rule" detection, |
| 154 | saves failing inputs to ``artifacts/``, treats any input that takes longer than |
| 155 | 10 seconds as a failure, and stores the working corpus in ``corpus/``. |
| 156 | |
| 157 | .. code:: |
| 158 | |
Keir Mierle | 6a10636 | 2021-01-14 16:27:44 -0800 | [diff] [blame] | 159 | $ mkdir -p corpus |
| 160 | $ ASAN_OPTIONS=detect_odr_violation=0 \ |
Ali Zhang | 5876558 | 2021-01-29 12:04:06 -0800 | [diff] [blame] | 161 | out/host_clang_fuzz/obj/pw_fuzzer/bin/toy_fuzzer \ |
Keir Mierle | 6a10636 | 2021-01-14 16:27:44 -0800 | [diff] [blame] | 162 | -artifact_prefix=artifacts/ \ |
| 163 | -timeout=10 \ |
| 164 | corpus |
Aaron Green | f3c3d2b | 2020-04-02 23:12:31 -0700 | [diff] [blame] | 165 | INFO: Seed: 305325345 |
| 166 | INFO: Loaded 1 modules (46 inline 8-bit counters): 46 [0x38dfc0, 0x38dfee), |
| 167 | INFO: Loaded 1 PC tables (46 PCs): 46 [0x23aaf0,0x23add0), |
| 168 | INFO: 0 files found in corpus |
| 169 | INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes |
| 170 | INFO: A corpus is not provided, starting from an empty corpus |
| 171 | #2 INITED cov: 2 ft: 3 corp: 1/1b exec/s: 0 rss: 27Mb |
| 172 | #4 NEW cov: 3 ft: 4 corp: 2/3b lim: 4 exec/s: 0 rss: 27Mb L: 2/2 MS: 2 ShuffleBytes-InsertByte- |
| 173 | #11 NEW cov: 7 ft: 8 corp: 3/7b lim: 4 exec/s: 0 rss: 27Mb L: 4/4 MS: 2 EraseBytes-CrossOver- |
| 174 | #27 REDUCE cov: 7 ft: 8 corp: 3/6b lim: 4 exec/s: 0 rss: 27Mb L: 3/3 MS: 1 EraseBytes- |
| 175 | #29 REDUCE cov: 7 ft: 8 corp: 3/5b lim: 4 exec/s: 0 rss: 27Mb L: 2/2 MS: 2 ChangeBit-EraseBytes- |
| 176 | #445 REDUCE cov: 9 ft: 10 corp: 4/13b lim: 8 exec/s: 0 rss: 27Mb L: 8/8 MS: 1 InsertRepeatedBytes- |
| 177 | #12104 NEW cov: 11 ft: 12 corp: 5/24b lim: 122 exec/s: 0 rss: 28Mb L: 11/11 MS: 4 CMP-InsertByte-ShuffleBytes-ChangeByte- DE: "\xff\xff"- |
| 178 | #12321 NEW cov: 12 ft: 13 corp: 6/31b lim: 122 exec/s: 0 rss: 28Mb L: 7/11 MS: 2 CopyPart-EraseBytes- |
| 179 | #12459 REDUCE cov: 12 ft: 13 corp: 6/28b lim: 122 exec/s: 0 rss: 28Mb L: 8/8 MS: 3 CMP-InsertByte-EraseBytes- DE: "\x00\x00"- |
| 180 | #12826 REDUCE cov: 12 ft: 13 corp: 6/26b lim: 122 exec/s: 0 rss: 28Mb L: 5/8 MS: 2 ShuffleBytes-EraseBytes- |
| 181 | #14824 REDUCE cov: 12 ft: 13 corp: 6/25b lim: 135 exec/s: 0 rss: 28Mb L: 4/8 MS: 3 PersAutoDict-ShuffleBytes-EraseBytes- DE: "\x00\x00"- |
| 182 | #15106 REDUCE cov: 12 ft: 13 corp: 6/24b lim: 135 exec/s: 0 rss: 28Mb L: 3/8 MS: 2 ChangeByte-EraseBytes- |
| 183 | ... |
| 184 | #197809 REDUCE cov: 35 ft: 36 corp: 22/129b lim: 1800 exec/s: 0 rss: 79Mb L: 9/9 MS: 1 InsertByte- |
| 185 | #216250 REDUCE cov: 35 ft: 36 corp: 22/128b lim: 1980 exec/s: 0 rss: 87Mb L: 8/8 MS: 1 EraseBytes- |
| 186 | #242761 REDUCE cov: 35 ft: 36 corp: 22/127b lim: 2237 exec/s: 0 rss: 101Mb L: 7/8 MS: 1 EraseBytes- |
| 187 | ==126148== ERROR: libFuzzer: deadly signal |
| 188 | #0 0x35b981 in __sanitizer_print_stack_trace ../recipe_cleanup/clangFu99hg/llvm_build_dir/tools/clang/stage2-bins/runtimes/runtimes-x86_64-unknown-linux-gnu-bins/compiler-rt/lib/asan/asan_stack.cpp:86:3 |
| 189 | #1 0x2bcdb5 in fuzzer::PrintStackTrace() (/home/aarongreen/src/pigweed/out/host/obj/pw_fuzzer/toy_fuzzer+0x2bcdb5) |
| 190 | #2 0x2a2ac9 in fuzzer::Fuzzer::CrashCallback() (/home/aarongreen/src/pigweed/out/host/obj/pw_fuzzer/toy_fuzzer+0x2a2ac9) |
| 191 | #3 0x7f866684151f (/lib/x86_64-linux-gnu/libpthread.so.0+0x1351f) |
| 192 | #4 0x3831df in (anonymous namespace)::toy_example(char const*, char const*) /home/aarongreen/src/pigweed/out/host/../../pw_fuzzer/examples/toy_fuzzer.cc:49:15 |
| 193 | #5 0x3831df in LLVMFuzzerTestOneInput /home/aarongreen/src/pigweed/out/host/../../pw_fuzzer/examples/toy_fuzzer.cc:80:3 |
| 194 | #6 0x2a4025 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) (/home/aarongreen/src/pigweed/out/host/obj/pw_fuzzer/toy_fuzzer+0x2a4025) |
| 195 | #7 0x2a3774 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool*) (/home/aarongreen/src/pigweed/out/host/obj/pw_fuzzer/toy_fuzzer+0x2a3774) |
| 196 | #8 0x2a5769 in fuzzer::Fuzzer::MutateAndTestOne() (/home/aarongreen/src/pigweed/out/host/obj/pw_fuzzer/toy_fuzzer+0x2a5769) |
| 197 | #9 0x2a6185 in fuzzer::Fuzzer::Loop(std::__Fuzzer::vector<fuzzer::SizedFile, fuzzer::fuzzer_allocator<fuzzer::SizedFile> >&) (/home/aarongreen/src/pigweed/out/host/obj/pw_fuzzer/toy_fuzzer+0x2a6185) |
| 198 | #10 0x294c8a in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/home/aarongreen/src/pigweed/out/host/obj/pw_fuzzer/toy_fuzzer+0x294c8a) |
| 199 | #11 0x2bd422 in main ../recipe_cleanup/clangFu99hg/llvm_build_dir/tools/clang/stage2-bins/runtimes/runtimes-x86_64-unknown-linux-gnu-bins/compiler-rt/lib/fuzzer/FuzzerMain.cpp:19:10 |
| 200 | #12 0x7f8666684bba in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26bba) |
| 201 | #13 0x26ae19 in _start (/home/aarongreen/src/pigweed/out/host/obj/pw_fuzzer/toy_fuzzer+0x26ae19) |
| 202 | |
| 203 | NOTE: libFuzzer has rudimentary signal handlers. |
| 204 | Combine libFuzzer with AddressSanitizer or similar for better crash reports. |
| 205 | SUMMARY: libFuzzer: deadly signal |
| 206 | MS: 1 CrossOver-; base unit: 9f479f7a6e3a21363397a25da3168218ba182a16 |
| 207 | 0x68,0x65,0x6c,0x6c,0x6f,0x0,0x77,0x6f,0x72,0x6c,0x64,0x0,0x0,0x0, |
| 208 | hello\x00world\x00\x00\x00 |
Aaron Green | e8449d1 | 2020-04-06 19:24:30 -0700 | [diff] [blame] | 209 | artifact_prefix='artifacts'; Test unit written to artifacts/crash-6e4fdc7ffd04131ea15dd243a0890b1b606f4831 |
Aaron Green | f3c3d2b | 2020-04-02 23:12:31 -0700 | [diff] [blame] | 210 | Base64: aGVsbG8Ad29ybGQAAAA= |
| 211 | |
Aaron Green | e8449d1 | 2020-04-06 19:24:30 -0700 | [diff] [blame] | 212 | Running fuzzers on OSS-Fuzz |
| 213 | =========================== |
| 214 | |
| 215 | Pigweed is integrated with `OSS-Fuzz`_, a continuous fuzzing infrastructure for |
| 216 | open source software. Fuzzers listed in in ``pw_test_groups`` will automatically |
| 217 | start being run within a day or so of appearing in the git repository. |
| 218 | |
| 219 | Bugs produced by OSS-Fuzz can be found in its `Monorail instance`_. These bugs |
| 220 | include: |
| 221 | |
| 222 | * A detailed report, including a symbolized backtrace. |
| 223 | * A revision range indicating when the bug has been detected. |
| 224 | * A minimized testcase, which is a fuzzer input that can be used to reproduce |
| 225 | the bug. |
| 226 | |
| 227 | To reproduce a bug: |
| 228 | |
| 229 | #. Build_ the fuzzers as described above. |
| 230 | #. Download the minimized testcase. |
| 231 | #. Run_ the fuzzer with the testcase as an argument. |
| 232 | |
| 233 | For example, if the testcase is saved as "~/Downloads/testcase" |
| 234 | and the fuzzer is the same as in the examples above, you could run: |
| 235 | |
| 236 | .. code:: |
| 237 | |
| 238 | $ ./out/host/obj/pw_fuzzer/toy_fuzzer ~/Downloads/testcase |
| 239 | |
| 240 | If you need to recreate the OSS-Fuzz environment locally, you can use its |
| 241 | documentation on `reproducing`_ issues. |
| 242 | |
| 243 | In particular, you can recreate the OSS-Fuzz environment using: |
| 244 | |
| 245 | .. code:: |
| 246 | |
| 247 | $ python infra/helper.py pull_images |
| 248 | $ python infra/helper.py build_image pigweed |
| 249 | $ python infra/helper.py build_fuzzers --sanitizer <address/undefined> pigweed |
| 250 | |
| 251 | With that environment, you can run the reproduce bugs using: |
| 252 | |
| 253 | .. code:: |
| 254 | |
| 255 | python infra/helper.py reproduce pigweed <pw_module>_<fuzzer_name> ~/Downloads/testcase |
| 256 | |
| 257 | You can even verify fixes in your local source checkout: |
| 258 | |
| 259 | .. code:: |
| 260 | |
| 261 | $ python infra/helper.py build_fuzzers --sanitizer <address/undefined> pigweed $PW_ROOT |
| 262 | $ python infra/helper.py reproduce pigweed <pw_module>_<fuzzer_name> ~/Downloads/testcase |
| 263 | |
Aaron Green | f3c3d2b | 2020-04-02 23:12:31 -0700 | [diff] [blame] | 264 | .. _compiler_rt: https://compiler-rt.llvm.org/ |
| 265 | .. _corpus: https://llvm.org/docs/LibFuzzer.html#corpus |
| 266 | .. _FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION: https://llvm.org/docs/LibFuzzer.html#fuzzer-friendly-build-mode |
Rob Mohr | 55bb0ad | 2021-05-22 10:59:52 -0700 | [diff] [blame] | 267 | .. _FuzzedDataProvider: https://github.com/llvm/llvm-project/blob/HEAD/compiler-rt/include/fuzzer/FuzzedDataProvider.h |
Aaron Green | f3c3d2b | 2020-04-02 23:12:31 -0700 | [diff] [blame] | 268 | .. _libFuzzer: https://llvm.org/docs/LibFuzzer.html |
| 269 | .. _libFuzzer options: https://llvm.org/docs/LibFuzzer.html#options |
| 270 | .. _LLVMFuzzerTestOneInput: https://llvm.org/docs/LibFuzzer.html#fuzz-target |
Aaron Green | e8449d1 | 2020-04-06 19:24:30 -0700 | [diff] [blame] | 271 | .. _monorail instance: https://bugs.chromium.org/p/oss-fuzz |
| 272 | .. _oss-fuzz: https://github.com/google/oss-fuzz |
| 273 | .. _reproducing: https://google.github.io/oss-fuzz/advanced-topics/reproducing/ |
Aaron Green | f3c3d2b | 2020-04-02 23:12:31 -0700 | [diff] [blame] | 274 | .. _running a fuzzer: https://llvm.org/docs/LibFuzzer.html#running |
| 275 | .. _sanitizer runtime flags: https://github.com/google/sanitizers/wiki/SanitizerCommonFlags |
Rob Mohr | 55bb0ad | 2021-05-22 10:59:52 -0700 | [diff] [blame] | 276 | .. _split a fuzzing input: https://github.com/google/fuzzing/blob/HEAD/docs/split-inputs.md |
Aaron Green | f3c3d2b | 2020-04-02 23:12:31 -0700 | [diff] [blame] | 277 | .. _startup initialization: https://llvm.org/docs/LibFuzzer.html#startup-initialization |
Rob Mohr | 55bb0ad | 2021-05-22 10:59:52 -0700 | [diff] [blame] | 278 | .. _structure aware fuzzing: https://github.com/google/fuzzing/blob/HEAD/docs/structure-aware-fuzzing.md |
Aaron Green | f3c3d2b | 2020-04-02 23:12:31 -0700 | [diff] [blame] | 279 | .. _valid options: https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html |