diff --git a/src/SUMMARY.md b/src/SUMMARY.md index b558fb2b..3cd6989a 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -46,6 +46,7 @@ - [Stabilizing Features](./stabilization_guide.md) - [Feature Gates](./feature-gates.md) - [Coding conventions](./conventions.md) +- [Fuzzing](./fuzzing.md) - [Notification groups](notification-groups/about.md) - [ARM](notification-groups/arm.md) - [Cleanup Crew](notification-groups/cleanup-crew.md) diff --git a/src/fuzzing.md b/src/fuzzing.md new file mode 100644 index 00000000..061ed16a --- /dev/null +++ b/src/fuzzing.md @@ -0,0 +1,105 @@ +# Fuzzing + + + +For the purposes of this guide, *fuzzing* is any testing methodology that +involves compiling a wide variety of programs in an attempt to uncover bugs in +rustc. Fuzzing is often used to find internal compiler errors (ICEs). Fuzzing +can be beneficial, because it can find bugs before users run into them and +provide small, self-contained programs that make the bug easier to track down. +However, some common mistakes can reduce the helpfulness of fuzzing and end up +making contributors' lives harder. To maximize your positive impact on the Rust +project, please read this guide before reporting fuzzer-generated bugs! + +## Guidelines + +### In a nutshell + +*Please do:* + +- Ensure the bug is still present on the latest nightly rustc +- Include a reasonably minimal, standalone example along with any bug report +- Include all of the information requested in the bug report template +- Search for existing reports with the same message and query stack +- Format the test case with `rustfmt`, if it maintains the bug + +*Please don't:* + +- Report lots of bugs that use internal features, including but not limited to + `custom_mir`, `lang_items`, `no_std`, and `rustc_attrs`. +- Seed your fuzzer with inputs that are known to crash rustc (details below). + +### Discussion + +If you're not sure whether or not an ICE is a duplicate of one that's already +been reported, please go ahead and report it and link to issues you think might +be related. In general, ICEs on the same line but with different *query stacks* +are usually distinct bugs. + +## Building a corpus + +When building a corpus, be sure to avoid collecting tests that are already +known to crash rustc. A fuzzer that is seeded with such tests is more likely to +generate bugs with the same root cause, wasting everyone's time. The simplest +way to avoid this is to loop over each file in the corpus, see if it causes an +ICE, and remove it if so. + +To build a corpus, you may want to use: + +- The rustc/rust-analyzer/clippy test suites (or even source code) --- though avoid + tests that are already known to cause failures, which often begin with comments + like `// failure-status: 101` or `// known-bug: #NNN`. +- The already-fixed ICEs in [Glacier][glacier] --- though avoid the unfixed + ones in `ices/`! + +## Extra credit + +Here are a few things you can do to help the Rust project after filing an ICE. + +- Add the minimal test case to [Glacier][glacier] +- [Bisect][bisect] the bug to figure out when it was introduced +- Fix unrelated problems with the test case (things like syntax errors or + borrow-checking errors) +- Minimize the test case (see below) + +[bisect]: https://github.com/rust-lang/cargo-bisect-rustc/blob/master/TUTORIAL.md + +## Minimization + +It can be helpful to *minimize* the fuzzer-generated input. When minimizing, be +careful to preserve the original error, and avoid introducing distracting +problems such as syntax, type-checking, or borrow-checking errors. + +There are some tools that can help with minimization. If you're not sure how +to avoid introducing syntax, type-, and borrow-checking errors while using +these tools, post both the complete and minimized test cases. Generally, +*syntax-aware* tools give the best results in the least amount of time. +[`treereduce-rust`][treereduce] and [picireny][picireny] are syntax-aware. +`halfempty` is not, but is generally a high-quality tool. + +[halfempty]: https://github.com/googleprojectzero/halfempty +[picireny]: https://github.com/renatahodovan/picireny +[treereduce]: https://github.com/langston-barrett/treereduce + +## Effective fuzzing + +When fuzzing rustc, you may want to avoid generating code, since this is mostly +done by LLVM. Try `--emit=mir` instead. + +A variety of compiler flags can uncover different issues. + +If you're fuzzing a compiler you built, you may want to build it with `-C +target-cpu=native` to squeeze out a few more executions per second. + +## Existing projects + +- [fuzz-rustc][fuzz-rustc] demonstrates how to fuzz rustc with libfuzzer +- [icemaker][icemaker] runs rustc and other tools on a large number of source + files with a variety of flags to catch ICEs +- [tree-splicer][tree-splicer] generates new source files by combining existing + ones while maintaining correct syntax + +[glacier]: https://github.com/rust-lang/glacier +[fuzz-rustc]: https://github.com/dwrensha/fuzz-rustc +[icemaker]: https://github.com/matthiaskrgr/icemaker/ +[tree-splicer]: https://github.com/langston-barrett/tree-splicer/ \ No newline at end of file