diff --git a/src/SUMMARY.md b/src/SUMMARY.md index a2f40ed3..21976fbf 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -2,7 +2,9 @@ - [About this guide](./about-this-guide.md) - [How to build the compiler and run what you built](./how-to-build-and-run.md) -- [Using the compiler testing framework](./running-tests.md) +- [The compiler testing framework](./tests/intro.md) + - [Running tests](./tests/running.md) + - [Adding new tests](./tests/adding.md) - [Walkthrough: a typical contribution](./walkthrough.md) - [High-level overview of the compiler source](./high-level-overview.md) - [Queries: demand-driven compilation](./query.md) diff --git a/src/running-tests.md b/src/running-tests.md deleted file mode 100644 index 02c9de84..00000000 --- a/src/running-tests.md +++ /dev/null @@ -1 +0,0 @@ -# Using the compiler testing framework diff --git a/src/tests/adding.md b/src/tests/adding.md new file mode 100644 index 00000000..9ce8318d --- /dev/null +++ b/src/tests/adding.md @@ -0,0 +1,273 @@ +# Adding new tests + +**In general, we expect every PR that fixes a bug in rustc to come +accompanied with a regression test of some kind.** This test should +fail in master but pass after the PR. These tests are really useful +for preventing us from repeating the mistakes of the past. + +To add a new test, the first thing you generally do is to create a +file, typically a Rust source file. Test files have a particular +structure: + +- They always begin with the copyright notice; +- then they should have some kind of + [comment explaining what the test is about](#explanatory_comment); +- next, they can have one or more [header commands](#header_commands), which are special + comments that the test interpreter knows how to interpret. +- finally, they have the Rust source. This may have various [error + annotations](#error_annotations) which indicate expected compilation errors or + warnings. + +Depending on the test suite, there may be some other details to be aware of: + - For [the `ui` test suite](#ui), you need to generate reference output files. + +## Naming your test + +We have not traditionally had a lot of structure in the names of +tests. Moreover, for a long time, the rustc test runner did not +support subdirectories (it now does), so test suites like +`src/test/run-pass` have a huge mess of files in them. This is not +considered an ideal setup. + +For regression tests -- basically, some random snippet of code that +came in from the internet -- we often just name the test after the +issue. For example, `src/test/run-pass/issue-1234.rs`. If possible, +though, it is better if you can put the test into a directory that +helps identify what piece of code is being tested here (e.g., +`borrowck/issue-12345.rs` is much better), or perhaps give it a more +meaningful name. Still, **do include the issue number somewhere**. + +When writing a new feature, **create a subdirectory to store your +tests**. For example, if you are implementing RFC 1234 ("Widgets"), +then it might make sense to put the tests in directories like: + +- `src/test/ui/rfc1234-widgets/` +- `src/test/run-pass/rfc1234-widgets/` +- etc + +In other cases, there may already be a suitable directory. (The proper +directory structure to use is actually an area of active debate.) + + + +## Comment explaining what the test is about + +When you create a test file, **include a comment summarizing the point +of the test immediately after the copyright notice**. This should +highlight which parts of the test are more important, and what the bug +was that the test is fixing. Citing an issue number is often very +helpful. + +This comment doesn't have to be super extensive. Just something like +"Regression test for #18060: match arms were matching in the wrong +order." might already be enough. + +These comments are very useful to others later on when your test +breaks, since they often can highlight what the problem is. They are +also useful if for some reason the tests need to be refactored, since +they let others know which parts of the test were important (often a +test must be rewritten because it no longer tests what is was meant to +test, and then it's useful to know what it *was* meant to test +exactly). + + + +## Header commands: configuring rustc + +Header commands are special comments that the test runner knows how to +interpret. They must appear before the Rust source in the test. They +are normally put after the short comment that explains the point of +this test. For example, this test uses the `// compile-flags` command +to specify a custom flag to give to rustc when the test is compiled: + +```rust +// Copyright 2017 The Rust Project Developers. blah blah blah. +// ... +// except according to those terms. + +// Test the behavior of `0 - 1` when overflow checks are disabled. + +// compile-flags: -Coverflow-checks=off + +fn main() { + let x = 0 - 1; + ... +} +``` + +### Ignoring tests + +These are used to ignore the test in some situations, which means the test won't +be compiled or run. + +* `ignore-X` where `X` is a target detail or stage will ignore the test accordingly (see below) +* `ignore-pretty` will not compile the pretty-printed test (this is done to test the pretty-printer, but might not always work) +* `ignore-test` always ignores the test +* `ignore-lldb` and `ignore-gdb` will skip a debuginfo test on that debugger. + +Some examples of `X` in `ignore-X`: + +* Architecture: `aarch64`, `arm`, `asmjs`, `mips`, `wasm32`, `x86_64`, `x86`, ... +* OS: `android`, `emscripten`, `freebsd`, `ios`, `linux`, `macos`, `windows`, ... +* Environment (fourth word of the target triple): `gnu`, `msvc`, `musl`. +* Pointer width: `32bit`, `64bit`. +* Stage: `stage0`, `stage1`, `stage2`. + +### Other Header Commands + +Here is a list of other header commands. This list is not +exhaustive. Header commands can generally be found by browsing the +`TestProps` structure found in [`header.rs`] from the compiletest +source. + +* `min-{gdb,lldb}-version` +* `min-llvm-version` +* `must-compile-successfully` for UI tests, indicates that the test is supposed + to compile, as opposed to the default where the test is supposed to error out. +* `compile-flags` passes extra command-line args to the compiler, + e.g. `compile-flags -g` which forces debuginfo to be enabled. +* `should-fail` indicates that the test should fail; used for "meta testing", + where we test the compiletest program itself to check that it will generate + errors in appropriate scenarios. This header is ignored for pretty-printer tests. +* `gate-test-X` where `X` is a feature marks the test as "gate test" for feature X. + Such tests are supposed to ensure that the compiler errors when usage of a gated + feature is attempted without the proper `#![feature(X)]` tag. + Each unstable lang feature is required to have a gate test. + +[`header.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/header.rs + + + +## Error annotations + +Error annotations specify the errors that the compiler is expected to +emit. They are "attached" to the line in source where the error is +located. + +* `~`: Associates the following error level and message with the + current line +* `~|`: Associates the following error level and message with the same + line as the previous comment +* `~^`: Associates the following error level and message with the + previous line. Each caret (`^`) that you add adds a line to this, so + `~^^^^^^^` is seven lines up. + +The error levels that you can have are: + +1. `ERROR` +2. `WARNING` +3. `NOTE` +4. `HELP` and `SUGGESTION`* + +\* **Note**: `SUGGESTION` must follow immediately after `HELP`. + +## Revisions + +Certain classes of tests support "revisions" (as of the time of this +writing, this includes run-pass, compile-fail, run-fail, and +incremental, though incremental tests are somewhat +different). Revisions allow a single test file to be used for multiple +tests. This is done by adding a special header at the top of the file: + +``` +// revisions: foo bar baz +``` + +This will result in the test being compiled (and tested) three times, +once with `--cfg foo`, once with `--cfg bar`, and once with `--cfg +baz`. You can therefore use `#[cfg(foo)]` etc within the test to tweak +each of these results. + +You can also customize headers and expected error messages to a particular +revision. To do this, add `[foo]` (or `bar`, `baz`, etc) after the `//` +comment, like so: + +``` +// A flag to pass in only for cfg `foo`: +//[foo]compile-flags: -Z verbose + +#[cfg(foo)] +fn test_foo() { + let x: usize = 32_u32; //[foo]~ ERROR mismatched types +} +``` + +Note that not all headers have meaning when customized to a revision. +For example, the `ignore-test` header (and all "ignore" headers) +currently only apply to the test as a whole, not to particular +revisions. The only headers that are intended to really work when +customized to a revision are error patterns and compiler flags. + + + +## Guide to the UI tests + +The UI tests are intended to capture the compiler's complete output, +so that we can test all aspects of the presentation. They work by +compiling a file (e.g., `ui/hello_world/main.rs`), capturing the output, +and then applying some normalization (see below). This normalized +result is then compared against reference files named +`ui/hello_world/main.stderr` and `ui/hello_world/main.stdout`. If either of +those files doesn't exist, the output must be empty. If the test run +fails, we will print out the current output, but it is also saved in +`build//test/ui/hello_world/main.stdout` (this path is +printed as part of the test failure message), so you can run `diff` and +so forth. + +Normally, the test-runner checks that UI tests fail compilation. If you want +to do a UI test for code that *compiles* (e.g. to test warnings, or if you +have a collection of tests, only some of which error out), you can use the +`// must-compile-successfully` header command to have the test runner instead +check that the test compiles successfully. + +### Editing and updating the reference files + +If you have changed the compiler's output intentionally, or you are +making a new test, you can use the script `ui/update-references.sh` to +update the references. When you run the test framework, it will report +various errors: in those errors is a command you can use to run the +`ui/update-references.sh` script, which will then copy over the files +from the build directory and use them as the new reference. You can +also just run `ui/update-all-references.sh`. In both cases, you can run +the script with `--help` to get a help message. + +### Normalization + +The normalization applied is aimed at eliminating output difference +between platforms, mainly about filenames: + +- the test directory is replaced with `$DIR` +- all backslashes (`\`) are converted to forward slashes (`/`) (for Windows) +- all CR LF newlines are converted to LF + +Sometimes these built-in normalizations are not enough. In such cases, you +may provide custom normalization rules using the header commands, e.g. + +``` +// normalize-stdout-test: "foo" -> "bar" +// normalize-stderr-32bit: "fn\(\) \(32 bits\)" -> "fn\(\) \($$PTR bits\)" +// normalize-stderr-64bit: "fn\(\) \(64 bits\)" -> "fn\(\) \($$PTR bits\)" +``` + +This tells the test, on 32-bit platforms, whenever the compiler writes +`fn() (32 bits)` to stderr, it should be normalized to read `fn() ($PTR bits)` +instead. Similar for 64-bit. The replacement is performed by regexes using +default regex flavor provided by `regex` crate. + +The corresponding reference file will use the normalized output to test both +32-bit and 64-bit platforms: + +``` +... + | + = note: source type: fn() ($PTR bits) + = note: target type: u16 (16 bits) +... +``` + +Please see `ui/transmute/main.rs` and `.stderr` for a concrete usage example. + +Besides `normalize-stderr-32bit` and `-64bit`, one may use any target +information or stage supported by `ignore-X` here as well (e.g. +`normalize-stderr-windows` or simply `normalize-stderr-test` for unconditional +replacement). diff --git a/src/tests/intro.md b/src/tests/intro.md new file mode 100644 index 00000000..c22916dc --- /dev/null +++ b/src/tests/intro.md @@ -0,0 +1,48 @@ +# Using the compiler testing framework + +The compiler has an extensive testing framework, masterminded by the +compiletest tool (sources in the [`src/tools/compiletest`]). This +section gives a brief overview of how the testing framework is setup, +and then gets into some of the details on +[how to run tests](running.html) as well as +[how to add new tests](adding.html). + +[`src/tools/compiletest`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest + +## Test suites + +The tests are located in the tree in the [`src/test`] +directory. Immediately within you will see a series of subdirectories +(e.g. `ui`, `run-make`, and so forth). Each of those directories is +called a **test suite** -- they house a group of tests that are run in +a distinct mode. + +[`src/test`]: https://github.com/rust-lang/rust/tree/master/src/test + +Here is a brief summary of the test suites as of this writing and what +they mean. In some cases, the test suites are linked to parts of the manual +that give more details. + +- [`ui`](ui.html) -- tests that check the exact stdout/stderr from compilation + and/or running the test +- `run-pass` -- tests that are expected to compile and execute successfully (no panics) + - `run-pass-valgrind` -- tests that ought to run with valrind +- `run-fail` -- tests that are expected to compile but then panic during execution +- `compile-fail` -- tests that are expected to fail compilation. +- `parse-fail` -- tests that are expected to fail to parse +- `pretty` -- tests targeting the Rust "pretty printer", which + generates valid Rust code from the AST +- `debuginfo` -- tests that run in gdb or lldb and query the debug info +- `codegen` -- tests that compile and then test the generated LLVM + code to make sure that optimizing we want are kicking in etc +- `mir-opt` -- tests that check parts of the generated MIR to make sure we are optimizing + etc. +- `incremental` -- tests for incremental compilation, checking that + when certain modifications are performed, we are able to reuse the + results from previous compilations. +- `run-make` -- tests that basically just execute a `Makefile`; the ultimate in flexibility + but annoying as all get out to write. +- `rustdoc` -- tests for rustdoc, making sure that the generated files contain + documentation for various entities etc +- `*-fulldeps` -- same as above, but indicates that the test depends on things other + than `libstd` (and hence those things must be built) diff --git a/src/tests/running.md b/src/tests/running.md new file mode 100644 index 00000000..8813862d --- /dev/null +++ b/src/tests/running.md @@ -0,0 +1,65 @@ +# Running tests + +You can run the tests using `x.py`. The most basic command -- which +you will almost never want to use! -- is as follows: + +``` +./x.py test +``` + +This will build the full stage 2 compiler and then run the whole test +suite. You probably don't want to do this very often, because it takes +a very long time, and anyway bors / travis will do it for you. (Often, +I will run this command in the background after opening a PR that I +think is done, but rarely otherwise. -nmatsakis) + +## Running a subset of the test suites + +When working on a specific PR, you will usually want to run a smaller +set of tests, and with a stage 1 build. For example, a good "smoke +test" that can be used after modifying rustc to see if things are +generally working correctly would be the following: + +```bash +./x.py test --stage 1 src/test/{ui,compile-fail,run-pass} +``` + +This will run the `ui`, `compile-fail`, and `run-pass` test suites, and +only with the stage 1 build. Of course, the choice of test suites is somewhat +arbitrary, and may not suit the task you are doing. For example, if you are hacking +on debuginfo, you may be better off with the debuginfo test suite: + +```bash +./x.py test --stage 1 src/test/debuginfo +``` + +**Warning:** Note that bors only runs the tests with the full stage 2 +build; therefore, while the tests **usually** work fine with stage 1, +there are some limitations. In particular, `./x.py test --stage 1` +(ie., + +## Running an individual test + +Another common thing that people want to do is to run an **individual +test**, often the test they are trying to fix. One way to do this is +to invoke `x.py` with the `--test-args` option: + +``` +./x.py test --stage 1 src/test/ui --test-args issue-1234 +``` + +Under the hood, the test runner invokes the standard rust test runner +(the same one you get with `#[test]`), so this command would wind up +filtering for tests that include "issue-1234" in the name. + +Often, though, it's easier to just run the test by hand. More tests are +just `rs` files, so you can do something like + +``` +rustc +stage1 src/test/ui/issue-1234.rs +``` + +This is much faster, but doesn't always work. For example, some tests +include directives that specify specific compiler flags, or which rely +on other crates, and they may not run the same without those options. +