mirror
/
rustc-dev-guide
mirror of https://github.com/rust-lang/rustc-dev-guide.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Reorganize and expand the testing chapters. (#1281) 

* Reorganize and expand the testing chapters.

* Update tests chapters for review comments.

* Fix typo.
This commit is contained in:
Eric Huss 2022-01-18 09:44:26 -08:00 committed by GitHub
parent 99beaada7c
commit 121b57d498
14 changed files with 2051 additions and 1096 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
book.toml
View File

@ -24,3 +24,6 @@ follow-web-links = true
exclude = [ "crates\\.io", "gcc\\.godbolt\\.org", "youtube\\.com", "youtu\\.be", "dl\\.acm\\.org", "cs\\.bgu\\.ac\\.il", "www\\.amazon\\.com", "www\\.rustaceans\\.org", "play\\.rust-lang\\.org" ]
cache-timeout = 86400
warning-policy = "error"
[output.html.redirect]
"/compiletest.html" = "tests/compiletest.html"

10
src/SUMMARY.md
View File

@ -15,10 +15,16 @@
- [Documenting Compiler](./building/compiler-documenting.md)
- [Rustdoc overview](./rustdoc.md)
- [Adding a new target](./building/new-target.md)
- [The compiler testing framework](./tests/intro.md)
- [Testing the compiler](./tests/intro.md)
- [Running tests](./tests/running.md)
- [Testing with Docker](./tests/docker.md)
- [Testing with CI](./tests/ci.md)
- [Adding new tests](./tests/adding.md)
- [Using `compiletest` commands to control test execution](./compiletest.md)
- [Compiletest](./tests/compiletest.md)
- [UI tests](./tests/ui.md)
- [Test headers](./tests/headers.md)
- [Performance testing](./tests/perf.md)
- [Crater](./tests/crater.md)
- [Debugging the Compiler](./compiler-debugging.md)
- [Using the tracing/logging instrumentation](./tracing.md)
- [Profiling the compiler](./profiling.md)

223
src/compiletest.md
View File

@ -1,223 +0,0 @@
# Using `compiletest` commands to control test execution
## Introduction
`compiletest` is the main test harness of the Rust test suite. It allows
test authors to organize large numbers of tests (the Rust compiler has many
thousands), efficient test execution (parallel execution is supported), and
allows the test author to configure behavior and expected results of both
individual and groups of tests.
`compiletest` tests may check test code for success, for runtime failure, or for
compile-time failure. Tests are typically organized as a Rust source file with
annotations in comments before and/or within the test code, which serve to
direct `compiletest` on if or how to run the test, what behavior to expect,
and more. If you are unfamiliar with the compiler testing framework,
see [this chapter](./tests/intro.md) for additional background.
The tests themselves are typically (but not always) organized into
"suites" – for example, `incremental`, a folder holding tests that check
incremental compilation behavior, `codegen`,
a folder holding tests that check code generation, and many more. The various
suites are defined in [`src/tools/compiletest/src/common.rs`] in the
`pub enum Mode` declaration. And a good introduction to the different
suites of compiler tests along with details about them can be found in
[Adding new tests](./tests/adding.md).
## Adding a new test file
Briefly, simply create your new test in the appropriate location under
[`src/test`]. No registration of test files is necessary as `compiletest`
will scan the [`src/test`] subfolder recursively, and will execute any
Rust source files it finds as tests.
See [Adding new tests](./tests/adding.md) for a complete guide on how to add
new tests.
## Header Commands
Source file annotations which appear in comments near the top of the source
file *before* any test code are known as header commands. These commands can
instruct `compiletest` to ignore this test, set expectations on whether it is
expected to succeed at compiling, or what the test's return code is expected to
be. Header commands and inline `//~ ERROR` commands are described more fully
[here](./tests/adding.md#header-commands-configuring-rustc).
### Adding a new header command
Header commands are defined in the `TestProps` struct in
[`src/tools/compiletest/src/header.rs`]. At a high level, there are
dozens of test properties defined here, all set to default values in the
`TestProp` struct's `impl` block. Any test can override this default value by
specifying the property in question as header command as a comment (`//`) in
the test source file, before any source code.
#### Using a header command
Here is an example, specifying the `must-compile-successfully` header command,
which takes no arguments, followed by the `failure-status` header command,
which takes a single argument (which, in this case is a value of 1).
`failure-status` is instructing `compiletest` to expect a failure status of 1
(rather than the current Rust default of 101). The header command and
the argument list (if present) are typically separated by a colon:
```rust,ignore
// must-compile-successfully
// failure-status: 1
#![feature(termination_trait)]
use std::io::{Error, ErrorKind};
fn main() -> Result<(), Box<Error>> {
Err(Box::new(Error::new(ErrorKind::Other, "returned Box<Error> from main()")))
}
```
#### Adding a new header command property
One would add a new header command if there is a need to define some test
property or behavior on an individual, test-by-test basis. A header command
property serves as the header command's backing store (holds the command's
current value) at runtime.
To add a new header command property:
1. Look for the `pub struct TestProps` declaration in
[`src/tools/compiletest/src/header.rs`] and add the new public property to
the end of the declaration.
2. Look for the `impl TestProps` implementation block immediately following
the struct declaration and initialize the new property to its default
value.
#### Adding a new header command parser
When `compiletest` encounters a test file, it parses the file a line at a time
by calling every parser defined in the `Config` struct's implementation block,
also in [`src/tools/compiletest/src/header.rs`][] (note that the `Config`
struct's declaration block is found in [`src/tools/compiletest/src/common.rs`]).
`TestProps`'s `load_from()` method will try passing the current line of text to
each parser, which, in turn typically checks to see if the line begins with a
particular commented (`//`) header command such as `// must-compile-successfully`
or `// failure-status`. Whitespace after the comment marker is optional.
Parsers will override a given header command property's default value merely by
being specified in the test file as a header command or by having a parameter
value specified in the test file, depending on the header command.
Parsers defined in `impl Config` are typically named `parse_<header_command>`
(note kebab-case `<header-command>` transformed to snake-case
`<header_command>`). `impl Config` also defines several 'low-level' parsers
which make it simple to parse common patterns like simple presence or not
(`parse_name_directive()`), header-command:parameter(s)
(`parse_name_value_directive()`), optional parsing only if a particular `cfg`
attribute is defined (`has_cfg_prefix()`) and many more. The low-level parsers
are found near the end of the `impl Config` block; be sure to look through them
and their associated parsers immediately above to see how they are used to
avoid writing additional parsing code unnecessarily.
As a concrete example, here is the implementation for the
`parse_failure_status()` parser, in [`src/tools/compiletest/src/header.rs`]:
```diff
@@ -232,6 +232,7 @@ pub struct TestProps {
// customized normalization rules
pub normalize_stdout: Vec<(String, String)>,
pub normalize_stderr: Vec<(String, String)>,
+ pub failure_status: i32,
}
impl TestProps {
@@ -260,6 +261,7 @@ impl TestProps {
run_pass: false,
normalize_stdout: vec![],
normalize_stderr: vec![],
+ failure_status: 101,
}
}
@@ -383,6 +385,10 @@ impl TestProps {
if let Some(rule) = config.parse_custom_normalization(ln, "normalize-stderr") {
self.normalize_stderr.push(rule);
}
+
+ if let Some(code) = config.parse_failure_status(ln) {
+ self.failure_status = code;
+ }
});
for key in &["RUST_TEST_NOCAPTURE", "RUST_TEST_THREADS"] {
@@ -488,6 +494,13 @@ impl Config {
self.parse_name_directive(line, "pretty-compare-only")
}
+ fn parse_failure_status(&self, line: &str) -> Option<i32> {
+ match self.parse_name_value_directive(line, "failure-status") {
+ Some(code) => code.trim().parse::<i32>().ok(),
+ _ => None,
+ }
+ }
```
## Implementing the behavior change
When a test invokes a particular header command, it is expected that some
behavior will change as a result. What behavior, obviously, will depend on the
purpose of the header command. In the case of `failure-status`, the behavior
that changes is that `compiletest` expects the failure code defined by the
header command invoked in the test, rather than the default value.
Although specific to `failure-status` (as every header command will have a
different implementation in order to invoke behavior change) perhaps it is
helpful to see the behavior change implementation of one case, simply as an
example. To implement `failure-status`, the `check_correct_failure_status()`
function found in the `TestCx` implementation block, located in
[`src/tools/compiletest/src/runtest.rs`], was modified as per below:
```diff
@@ -295,11 +295,14 @@ impl<'test> TestCx<'test> {
}
fn check_correct_failure_status(&self, proc_res: &ProcRes) {
- // The value the Rust runtime returns on failure
- const RUST_ERR: i32 = 101;
- if proc_res.status.code() != Some(RUST_ERR) {
+ let expected_status = Some(self.props.failure_status);
+ let received_status = proc_res.status.code();
+
+ if expected_status != received_status {
self.fatal_proc_rec(
- &format!("failure produced the wrong error: {}", proc_res.status),
+ &format!("Error: expected failure status ({:?}) but received status {:?}.",
+ expected_status,
+ received_status),
proc_res,
);
}
@@ -320,7 +323,6 @@ impl<'test> TestCx<'test> {
);
let proc_res = self.exec_compiled_test();
-
if !proc_res.status.success() {
self.fatal_proc_rec("test run failed!", &proc_res);
}
@@ -499,7 +501,6 @@ impl<'test> TestCx<'test> {
expected,
actual
);
- panic!();
}
}
```
Note the use of `self.props.failure_status` to access the header command
property. In tests which do not specify the failure status header command,
`self.props.failure_status` will evaluate to the default value of 101 at the
time of this writing. But for a test which specifies a header command of, for
example, `// failure-status: 1`, `self.props.failure_status` will evaluate to
1, as `parse_failure_status()` will have overridden the `TestProps` default
value, for that test specifically.
[`src/test`]: https://github.com/rust-lang/rust/tree/master/src/test
[`src/tools/compiletest/src/header.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/header.rs
[`src/tools/compiletest/src/common.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/common.rs
[`src/tools/compiletest/src/runtest.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/runtest.rs

7
src/profiling.md
View File

@ -4,11 +4,8 @@ This section talks about how to profile the compiler and find out where it spend
Depending on what you're trying to measure, there are several different approaches:
- If you want to see if a PR improves or regresses compiler performance:
- The [rustc-perf](https://github.com/rust-lang/rustc-perf) project makes this easy and can be triggered to run on a PR via the `@rust-timer` bot.
The `@bors try @rust-timer queue` command, in a comment on the PR, will queue a try build and a
benchmarking run.
Note: you need `try` privileges to be able to do this. More details are available in the [perf collector documentation](https://github.com/rust-lang/rustc-perf/blob/master/collector/README.md).
- If you want to see if a PR improves or regresses compiler performance,
see the [rustc-perf chapter](tests/perf.md) for requesting a benchmarking run.
- If you want a medium-to-high level overview of where `rustc` is spending its time:
- The `-Z self-profile` flag and [measureme](https://github.com/rust-lang/measureme) tools offer a query-based approach to profiling.

711
src/tests/adding.md
View File

@ -7,85 +7,180 @@ accompanied by a regression test of some kind.** This test should fail
in master but pass after the PR. These tests are really useful for
preventing us from repeating the mistakes of the past.
To add a new test, the first thing you generally do is to create a
file, typically a Rust source file. Test files have a particular
structure:
The first thing to decide is which kind of test to add.
This will depend on the nature of the change and what you want to exercise.
Here are some rough guidelines:
- They should have some kind of
[comment explaining what the test is about](#explanatory_comment);
- next, they can have one or more [header commands](#header_commands), which
are special comments that the test interpreter knows how to interpret.
- finally, they have the Rust source. This may have various [error
annotations](#error_annotations) which indicate expected compilation errors or
warnings.
Depending on the test suite, there may be some other details to be aware of:
- For [the `ui` test suite](#ui), you need to generate reference output files.
## What kind of test should I add?
It can be difficult to know what kind of test to use. Here are some
rough heuristics:
- Some tests have specialized needs:
- need to run gdb or lldb? use the `debuginfo` test suite
- need to inspect LLVM IR or MIR IR? use the `codegen` or `mir-opt` test
suites
- need to run rustdoc? Prefer a `rustdoc` or `rustdoc-ui` test.
Occasionally you'll need `rustdoc-js` as well.
- need to inspect the resulting binary in some way? Then use `run-make`
- Library tests should go in `library/${crate}/tests` (where `${crate}` is
usually `core`, `alloc`, or `std`). Library tests include:
- tests that an API behaves properly, including accepting various types or
having some runtime behavior
- tests where any compiler warnings are not relevant to the test
- tests that a use of an API gives a compile error, where the exact error
message is not relevant to the test. These should have an
[error number] (`E0XXX`) in the code block to make sure it's the correct error.
- For most other things, [a `ui` (or `ui-fulldeps`) test](#ui) is to be preferred:
- in the case of warnings or errors, `ui` tests capture the full output,
which makes it easier to review but also helps prevent "hidden" regressions
in the output
[error number]: https://doc.rust-lang.org/rustdoc/unstable-features.html#error-numbers-for-compile-fail-doctests
## Naming your test
We have not traditionally had a lot of structure in the names of
tests. Moreover, for a long time, the rustc test runner did not
support subdirectories (it now does), so test suites like
[`src/test/ui`] have a huge mess of files in them. This is not
considered an ideal setup.
- The majority of compiler tests are done with [compiletest].
- The majority of compiletest tests are [UI](ui.md) tests in the [`src/test/ui`] directory.
- Changes to the standard library are usually tested within the standard library itself.
- The majority of standard library tests are written as doctests,
which illustrate and exercise typical API behavior.
- Additional [unit tests](intro.md#package-tests) should go in
`library/${crate}/tests` (where `${crate}` is usually `core`, `alloc`, or `std`).
- If the code is part of an isolated system, and you are not testing compiler output,
consider using a [unit or integration test](intro.md#package-tests).
- Need to run rustdoc? Prefer a `rustdoc` or `rustdoc-ui` test.
Occasionally you'll need `rustdoc-js` as well.
- Other compiletest test suites are generally used for special purposes:
- Need to run gdb or lldb? Use the `debuginfo` test suite.
- Need to inspect LLVM IR or MIR IR? Use the `codegen` or `mir-opt` test suites.
- Need to inspect the resulting binary in some way?
Then use `run-make`.
- Check out the [compiletest] chapter for more specialized test suites.
[compiletest]: compiletest.md
[`src/test/ui`]: https://github.com/rust-lang/rust/tree/master/src/test/ui/
For regression tests – basically, some random snippet of code that
came in from the internet – we often name the test after the issue
plus a short description. Ideally, the test should be added to a
directory that helps identify what piece of code is being tested here
(e.g., `src/test/ui/borrowck/issue-54597-reject-move-out-of-borrow-via-pat.rs`)
If you've tried and cannot find a more relevant place,
the test may be added to `src/test/ui/issues/`.
Still, **do include the issue number somewhere**.
But please avoid putting your test there as possible since that
directory has too many tests and it causes poor semantic organization.
## UI test walkthrough
When writing a new feature, **create a subdirectory to store your
tests**. For example, if you are implementing RFC 1234 ("Widgets"),
then it might make sense to put the tests in a directory like
`src/test/ui/rfc1234-widgets/`.
The following is a basic guide for creating a [UI test](ui.md), which is one
of the most common compiler tests.
For this tutorial, we'll be adding a test for an async error message.
In other cases, there may already be a suitable directory. (The proper
directory structure to use is actually an area of active debate.)
### Step 1. Add a test file
The first step is to create a Rust source file somewhere in the
[`src/test/ui`] tree.
When creating a test, do your best to find a good location and name (see [Test
organization](ui.md#test-organization) for more).
Since naming is the hardest part of development, everything should be downhill
from here!
Let's place our async test at `src/test/ui/async-await/await-without-async.rs`:
```rust,ignore
// Check what happens when using await in a non-async fn.
// edition:2018
async fn foo() {}
fn bar() {
foo().await
}
fn main() {}
```
A few things to notice about our test:
* The top should start with a short comment that [explains what the test is
for](#explanatory_comment).
* The `// edition:2018` comment is called a [header](headers.md) which provides
instructions to compiletest on how to build the test.
Here we need to set the edition for `async` to work (the default is 2015).
* Following that is the source of the test.
Try to keep it succinct and to the point.
This may require some effort if you are trying to minimize an example from a
bug report.
* We end this test with an empty `fn main` function.
This is because the default for UI tests is a `bin` crate-type,
and we don't want the "main not found" error in our test.
Alternatively, you could add `#![crate_type="lib"]`.
### Step 2. Generate the expected output
The next step is to create the expected output from the compiler.
This can be done with the `--bless` option:
```sh
./x.py test src/test/ui/async-await/await-without-async.rs --bless
```
This will build the compiler (if it hasn't already been built), compile the
test, and place the output of the compiler in a file called
`src/test/ui/async-await/await-without-async.stderr`.
However, this step will fail!
You should see an error message, something like this:
> error: /rust/src/test/ui/async-await/await-without-async.rs:7: unexpected
> error: '7:10: 7:16: `await` is only allowed inside `async` functions and
> blocks [E0728]'
### Step 3. Add error annotations
Every error needs to be annotated with a comment in the source with the text
of the error.
In this case, we can add the following comment to our test file:
```rust,ignore
fn bar() {
foo().await
//~^ ERROR `await` is only allowed inside `async` functions and blocks
}
```
The `//~^` squiggle caret comment tells compiletest that the error belongs to
the previous line (more on this in the [Error
annotations](ui.md#error-annotations) section).
Save that, and run the test again:
```sh
./x.py test src/test/ui/async-await/await-without-async.rs
```
It should now pass, yay!
### Step 4. Review the output
Somewhat hand-in-hand with the previous step, you should inspect the `.stderr`
file that was created to see if it looks like how you expect.
If you are adding a new diagnostic message, now would be a good time to
also consider how readable the message looks overall, particularly for
people new to Rust.
Our example `src/test/ui/async-await/await-without-async.stderr` file should
look like this:
```text
error[E0728]: `await` is only allowed inside `async` functions and blocks
--> $DIR/await-without-async.rs:7:10
|
LL | fn bar() {
| --- this is not `async`
LL | foo().await
| ^^^^^^ only allowed inside `async` functions and blocks
error: aborting due to previous error
For more information about this error, try `rustc --explain E0728`.
```
You may notice some things look a little different than the regular
compiler output.
The `$DIR` removes the path information which will differ between systems.
The `LL` values replace the line numbers.
That helps avoid small changes in the source from triggering large diffs.
See the [Normalization](ui.md#normalization) section for more.
Around this stage, you may need to iterate over the last few steps a few times
to tweak your test, re-bless the test, and re-review the output.
### Step 5. Check other tests
Sometimes when adding or changing a diagnostic message, this will affect
other tests in the test suite.
The final step before posting a PR is to check if you have affected anything else.
Running the UI suite is usually a good start:
```sh
./x.py test src/test/ui
```
If other tests start failing, you may need to investigate what has changed
and if the new output makes sense.
You may also need to re-bless the output with the `--bless` flag.
<a name="explanatory_comment"></a>
## Comment explaining what the test is about
When you create a test file, **include a comment summarizing the point
of the test at the start of the file**. This should highlight which
parts of the test are more important, and what the bug was that the
test is fixing. Citing an issue number is often very helpful.
The first comment of a test file should **summarize the point
of the test**, and highlight what is important about it.
If there is an issue number associated with the test, include
the issue number.
This comment doesn't have to be super extensive. Just something like
"Regression test for #18060: match arms were matching in the wrong
@ -98,483 +193,3 @@ they let others know which parts of the test were important (often a
test must be rewritten because it no longer tests what is was meant to
test, and then it's useful to know what it *was* meant to test
exactly).
<a name="header_commands"></a>
## Header commands: configuring rustc
Header commands are special comments that the test runner knows how to
interpret. They must appear before the Rust source in the test. They
are normally put after the short comment that explains the point of
this test. For example, this test uses the `// compile-flags` command
to specify a custom flag to give to rustc when the test is compiled:
```rust,ignore
// Test the behavior of `0 - 1` when overflow checks are disabled.
// compile-flags: -C overflow-checks=off
fn main() {
let x = 0 - 1;
...
}
```
### Ignoring tests
These are used to ignore the test in some situations, which means the test won't
be compiled or run.
* `ignore-X` where `X` is a target detail or stage will ignore the
test accordingly (see below)
* `only-X` is like `ignore-X`, but will *only* run the test on that
target or stage
* `ignore-pretty` will not compile the pretty-printed test (this is
done to test the pretty-printer, but might not always work)
* `ignore-test` always ignores the test
* `ignore-lldb` and `ignore-gdb` will skip a debuginfo test on that
debugger.
* `ignore-gdb-version` can be used to ignore the test when certain gdb
versions are used
Some examples of `X` in `ignore-X`:
* Architecture: `aarch64`, `arm`, `asmjs`, `mips`, `wasm32`, `x86_64`,
`x86`, ...
* OS: `android`, `emscripten`, `freebsd`, `ios`, `linux`, `macos`,
`windows`, ...
* Environment (fourth word of the target triple): `gnu`, `msvc`,
`musl`.
* Pointer width: `32bit`, `64bit`.
* Stage: `stage0`, `stage1`, `stage2`.
* When cross compiling: `cross-compile`
* When remote testing is used: `remote`
* When debug-assertions are enabled: `debug`
* When particular debuggers are being tested: `cdb`, `gdb`, `lldb`
* Specific compare modes: `compare-mode-nll`, `compare-mode-polonius`
### Other Header Commands
Here is a list of other header commands. This list is not
exhaustive. Header commands can generally be found by browsing the
`TestProps` structure found in [`header.rs`] from the compiletest
source.
* `run-rustfix` for UI tests, indicates that the test produces
structured suggestions. The test writer should create a `.fixed`
file, which contains the source with the suggestions applied.
When the test is run, compiletest first checks that the correct
lint/warning is generated. Then, it applies the suggestion and
compares against `.fixed` (they must match). Finally, the fixed
source is compiled, and this compilation is required to succeed.
The `.fixed` file can also be generated automatically with the
`--bless` option, described in [this section][bless].
* `rustfix-only-machine-applicable` is equivalent to `run-rustfix` except it
will only apply [`MachineApplicable`](../diagnostics.md#suggestions)
suggestions. `run-rustfix` will apply *all* suggestions. This should be used
if there is a mixture of different suggestion levels, and some of the
non-machine-applicable ones do not apply cleanly.
* `min-gdb-version` specifies the minimum gdb version required for
this test; see also `ignore-gdb-version`
* `min-lldb-version` specifies the minimum lldb version required for
this test
* `rust-lldb` causes the lldb part of the test to only be run if the
lldb in use contains the Rust plugin
* `no-system-llvm` causes the test to be ignored if the system llvm is used
* `min-llvm-version` specifies the minimum llvm version required for
this test
* `min-system-llvm-version` specifies the minimum system llvm version
required for this test; the test is ignored if the system llvm is in
use and it doesn't meet the minimum version. This is useful when an
llvm feature has been backported to rust-llvm
* `ignore-llvm-version` can be used to skip the test when certain LLVM
versions are used. This takes one or two arguments; the first
argument is the first version to ignore. If no second argument is
given, all subsequent versions are ignored; otherwise, the second
argument is the last version to ignore.
* `build-pass` for UI tests, indicates that the test is supposed to
successfully compile and link, as opposed to the default where the test is
supposed to error out.
* `compile-flags` passes extra command-line args to the compiler,
e.g. `compile-flags -g` which forces debuginfo to be enabled.
* `edition` controls the edition the test should be compiled with
(defaults to 2015). Example usage: `// edition:2018`.
* `should-fail` indicates that the test should fail; used for "meta
testing", where we test the compiletest program itself to check that
it will generate errors in appropriate scenarios. This header is
ignored for pretty-printer tests.
* `gate-test-X` where `X` is a feature marks the test as "gate test"
for feature X. Such tests are supposed to ensure that the compiler
errors when usage of a gated feature is attempted without the proper
`#![feature(X)]` tag. Each unstable lang feature is required to
have a gate test.
* `needs-profiler-support` - a profiler runtime is required, i.e.,
`profiler = true` in rustc's `config.toml`.
* `needs-sanitizer-support` - a sanitizer runtime is required, i.e.,
`sanitizers = true` in rustc's `config.toml`.
* `needs-sanitizer-{address,hwaddress,leak,memory,thread}` - indicates that
test requires a target with a support for AddressSanitizer, hardware-assisted
AddressSanitizer, LeakSanitizer, MemorySanitizer or ThreadSanitizer
respectively.
* `error-pattern` checks the diagnostics just like the `ERROR` annotation
without specifying error line. This is useful when the error doesn't give
any span.
* `incremental` runs the test with the `-C incremental` flag and an empty
incremental directory. This should be avoided when possible; you should use
an *incremental mode* test instead. Incremental mode tests support running
the compiler multiple times and verifying that it can load the generated
incremental cache. This flag is for specialized circumstances, like checking
the interaction of codegen unit partitioning with generating an incremental
cache.
* `aux-build` is used to compile additional crates to link. Just pass it the
name of the source file. The source file should be in a directory called
`auxiliary` beside the test file. The aux crate will be built as a dylib if
possible (unless on a platform that does not support them, or
`no-prefer-dynamic` is specified in the aux file). The `-L` flag is used to
find the extern crates.
* `aux-crate` is very similar to `aux-build`; however, it uses the `--extern`
flag to link to the extern crate. That allows you to specify the additional
syntax of the `--extern` flag, such as renaming a dependency. For example,
`// aux-crate:foo=bar.rs` will compile `auxiliary/bar.rs` and make it
available under then name `foo` within the test. This is similar to how
Cargo does dependency renaming.
* `no-prefer-dynamic` will force an auxiliary crate to be built as an rlib
instead of a dylib. When specified in a test, it will remove the use of `-C
prefer-dynamic`. This can be useful in a variety of circumstances. For
example, it can prevent a proc-macro from being built with the wrong crate
type. Or if your test is specifically targeting behavior of other crate
types, it can be used to prevent building with the wrong crate type.
* `force-host` will force the test to build for the host platform instead of
the target. This is useful primarily for auxiliary proc-macros, which need
to be loaded by the host compiler.
* `pretty-mode` specifies the mode pretty-print tests should run in.
The default is `normal` if not specified.
* `pretty-compare-only` causes a pretty test to only compare the
pretty-printed output. It will not try to compile the expanded output to
typecheck it. This is needed for a pretty-mode that does not expand to valid
Rust, or for other situations where the expanded output cannot be compiled.
* `pretty-expanded` allows a pretty test to also run with
`-Zunpretty=expanded` as a final step. It will also try to compile the
resulting output (without codegen). This is needed because not all code can
be compiled after being expanded. Pretty tests should specify this if they
can. An example where this cannot be used is if the test includes
`println!`. That macro expands to reference private internal functions of
the standard library that cannot be called directly without the
`fmt_internals` feature gate.
More history about this may be found in [#23616].
* `pp-exact` is used to ensure a pretty-print test results in specific output.
If specified without a value, then it means the pretty-print output should
match the original source. If specified with a value, as in `//
pp-exact:foo.pp`, it will ensure that the pretty-printed output matches the
contents of the given file. Otherwise, if `pp-exact` is not specified, then
the pretty-printed output will be pretty-printed one more time, and the
output of the two pretty-printing rounds will be compared to ensure that the
pretty-printed output converges to a steady state.
[`header.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/header.rs
[bless]: ./running.md#editing-and-updating-the-reference-files
[#23616]: https://github.com/rust-lang/rust/issues/23616#issuecomment-484999901
<a name="error_annotations"></a>
## Error annotations
Error annotations specify the errors that the compiler is expected to
emit. They are "attached" to the line in source where the error is
located. Error annotations are considered during tidy lints of line
length and should be formatted according to tidy requirements. You may
use an error message prefix sub-string if necessary to meet line length
requirements. Make sure that the text is long enough for the error
message to be self-documenting.
The error annotation definition and source line definition association
is defined with the following set of idioms:
* `~`: Associates the following error level and message with the
current line
* `~|`: Associates the following error level and message with the same
line as the previous comment
* `~^`: Associates the following error level and message with the
previous error annotation line. Each caret (`^`) that you add adds
a line to this, so `~^^^` is three lines above the error annotation
line.
### Error annotation examples
Here are examples of error annotations on different lines of UI test
source.
#### Positioned on error line
Use the `//~ ERROR` idiom:
```rust,ignore
fn main() {
let x = (1, 2, 3);
match x {
(_a, _x @ ..) => {} //~ ERROR `_x @` is not allowed in a tuple
_ => {}
}
}
```
#### Positioned below error line
Use the `//~^` idiom with number of carets in the string to indicate the
number of lines above. In the example below, the error line is four
lines above the error annotation line so four carets are included in
the annotation.
```rust,ignore
fn main() {
let x = (1, 2, 3);
match x {
(_a, _x @ ..) => {} // <- the error is on this line
_ => {}
}
}
//~^^^^ ERROR `_x @` is not allowed in a tuple
```
#### Use same error line as defined on error annotation line above
Use the `//~|` idiom to define the same error line as
the error annotation line above:
```rust,ignore
struct Binder(i32, i32, i32);
fn main() {
let x = Binder(1, 2, 3);
match x {
Binder(_a, _x @ ..) => {} // <- the error is on this line
_ => {}
}
}
//~^^^^ ERROR `_x @` is not allowed in a tuple struct
//~| ERROR this pattern has 1 field, but the corresponding tuple struct has 3 fields [E0023]
```
#### When error line cannot be specified
Let's think about this test:
```rust,ignore
fn main() {
let a: *const [_] = &[1, 2, 3];
unsafe {
let _b = (*a)[3];
}
}
```
We want to ensure this shows "index out of bounds" but we cannot use the `ERROR` annotation
since the error doesn't have any span. Then it's time to use the `error-pattern`:
```rust,ignore
// error-pattern: index out of bounds
fn main() {
let a: *const [_] = &[1, 2, 3];
unsafe {
let _b = (*a)[3];
}
}
```
But for strict testing, try to use the `ERROR` annotation as much as possible.
#### Error levels
The error levels that you can have are:
1. `ERROR`
2. `WARNING`
3. `NOTE`
4. `HELP` and `SUGGESTION`[^sugg-placement]
[^sugg-placement]: **Note**: `SUGGESTION` must follow immediately after `HELP`.
## Revisions
Certain classes of tests support "revisions" (as of <!-- date: 2021-02 --> February 2021,
this includes compile-fail, run-fail, and incremental, though
incremental tests are somewhat different). Revisions allow a single test file to
be used for multiple tests. This is done by adding a special header at the top
of the file:
```rust
// revisions: foo bar baz
```
This will result in the test being compiled (and tested) three times,
once with `--cfg foo`, once with `--cfg bar`, and once with `--cfg
baz`. You can therefore use `#[cfg(foo)]` etc within the test to tweak
each of these results.
You can also customize headers and expected error messages to a particular
revision. To do this, add `[foo]` (or `bar`, `baz`, etc) after the `//`
comment, like so:
```rust
// A flag to pass in only for cfg `foo`:
//[foo]compile-flags: -Z verbose
#[cfg(foo)]
fn test_foo() {
let x: usize = 32_u32; //[foo]~ ERROR mismatched types
}
```
Note that not all headers have meaning when customized to a revision.
For example, the `ignore-test` header (and all "ignore" headers)
currently only apply to the test as a whole, not to particular
revisions. The only headers that are intended to really work when
customized to a revision are error patterns and compiler flags.
<a name="ui"></a>
## Guide to the UI tests
The UI tests are intended to capture the compiler's complete output,
so that we can test all aspects of the presentation. They work by
compiling a file (e.g., [`ui/hello_world/main.rs`][hw-main]),
capturing the output, and then applying some normalization (see
below). This normalized result is then compared against reference
files named `ui/hello_world/main.stderr` and
`ui/hello_world/main.stdout`. If either of those files doesn't exist,
the output must be empty (that is actually the case for
[this particular test][hw]). If the test run fails, we will print out
the current output, but it is also saved in
`build/<target-triple>/test/ui/hello_world/main.stdout` (this path is
printed as part of the test failure message), so you can run `diff`
and so forth.
[hw-main]: https://github.com/rust-lang/rust/blob/master/src/test/ui/hello_world/main.rs
[hw]: https://github.com/rust-lang/rust/blob/master/src/test/ui/hello_world/
We now have a ton of UI tests and some directories have too many entries.
This is a problem because it isn't editor/IDE friendly and GitHub UI won't
show more than 1000 entries. To resolve it and organize semantic structure,
we have a tidy check to ensure the number of entries is less than 1000.
However, since `src/test/ui` (UI test root directory) and
`src/test/ui/issues` directories have more than 1000 entries,
we set a different limit for each directories. So, please
avoid putting a new test there and try to find a more relevant place.
For example, if your test is related to closures, you should put it in
`src/test/ui/closures`. If you're not sure where is the best place,
it's still okay to add to `src/test/ui/issues/`. When you reach the limit,
you could increase it by tweaking [here][ui test tidy].
[ui test tidy]: https://github.com/rust-lang/rust/blob/master/src/tools/tidy/src/ui_tests.rs
### Tests that do not result in compile errors
By default, a UI test is expected **not to compile** (in which case,
it should contain at least one `//~ ERROR` annotation). However, you
can also make UI tests where compilation is expected to succeed, and
you can even run the resulting program. Just add one of the following
[header commands](#header_commands):
- `// check-pass` - compilation should succeed but skip codegen
(which is expensive and isn't supposed to fail in most cases)
- `// build-pass` – compilation and linking should succeed but do
not run the resulting binary
- `// run-pass` – compilation should succeed and we should run the
resulting binary
### Output Normalization
The compiler output is normalized to eliminate output difference between
platforms, mainly about filenames.
The following strings replace their corresponding values:
- `$DIR`: The directory where the test is defined.
- Example: `/path/to/rust/src/test/ui/error-codes`
- `$SRC_DIR`: The root source directory.
- Example: `/path/to/rust/src`
- `$TEST_BUILD_DIR`: The base directory where the test's output goes.
- Example: `/path/to/rust/build/x86_64-unknown-linux-gnu/test/ui`
Additionally, the following changes are made:
- Line and column numbers for paths in `$SRC_DIR` are replaced with `LL:CC`.
For example, `/path/to/rust/library/core/src/clone.rs:122:8` is replaced with
`$SRC_DIR/core/src/clone.rs:LL:COL`.
Note: The line and column numbers for `-->` lines pointing to the test are
*not* normalized, and left as-is. This ensures that the compiler continues
to point to the correct location, and keeps the stderr files readable.
Ideally all line/column information would be retained, but small changes to
the source causes large diffs, and more frequent merge conflicts and test
errors. See also `-Z ui-testing` below which applies additional line number
normalization.
- `\t` is replaced with an actual tab character.
- Error line annotations like `// ~ERROR some message` are removed.
- Backslashes (`\`) are converted to forward slashes (`/`) within paths (using
a heuristic). This helps normalize differences with Windows-style paths.
- CRLF newlines are converted to LF.
Additionally, the compiler is run with the `-Z ui-testing` flag which causes
the compiler itself to apply some changes to the diagnostic output to make it
more suitable for UI testing. For example, it will anonymize line numbers in
the output (line numbers prefixing each source line are replaced with `LL`).
In extremely rare situations, this mode can be disabled with the header
command `// compile-flags: -Z ui-testing=no`.
Sometimes these built-in normalizations are not enough. In such cases, you
may provide custom normalization rules using the header commands, e.g.
```rust
// normalize-stdout-test: "foo" -> "bar"
// normalize-stderr-32bit: "fn\(\) \(32 bits\)" -> "fn\(\) \($$PTR bits\)"
// normalize-stderr-64bit: "fn\(\) \(64 bits\)" -> "fn\(\) \($$PTR bits\)"
```
This tells the test, on 32-bit platforms, whenever the compiler writes
`fn() (32 bits)` to stderr, it should be normalized to read `fn() ($PTR bits)`
instead. Similar for 64-bit. The replacement is performed by regexes using
default regex flavor provided by `regex` crate.
The corresponding reference file will use the normalized output to test both
32-bit and 64-bit platforms:
```text
...
|
= note: source type: fn() ($PTR bits)
= note: target type: u16 (16 bits)
...
```
Please see [`ui/transmute/main.rs`][mrs] and [`main.stderr`][] for a
concrete usage example.
[mrs]: https://github.com/rust-lang/rust/blob/master/src/test/ui/transmute/main.rs
[`main.stderr`]: https://github.com/rust-lang/rust/blob/master/src/test/ui/transmute/main.stderr
Besides `normalize-stderr-32bit` and `-64bit`, one may use any target
information or stage supported by [`ignore-X`](#ignoring-tests) here as well (e.g.
`normalize-stderr-windows` or simply `normalize-stderr-test` for unconditional
replacement).
## Input Normalization
Sometimes, you want to normalize the inputs to a test. For example, you may
want to pass `// compile-flags: --x=y.rs`, where y.rs is some file in the test
directory. In this case you can use input normalization. The following strings
are replaced in header inputs:
- {{cwd}}: The directory where compiletest is run from. This may not be the
root of the checkout, so you should avoid using it where possible.
- Examples: `/path/to/rust`, `/path/to/build/root`
- {{src-base}}: The directory where the test is defined. This is equivalent to
`$DIR` for output normalization.
- Example: `/path/to/rust/src/test/ui/error-codes`
- {{build-base}}: The base directory where the test's output goes. This is
equivalent to `$TEST_BUILD_DIR` for output normalization.
- Example: `/path/to/rust/build/x86_64-unknown-linux-gnu/test/ui`
See [`src/test/ui/commandline-argfile.rs`](https://github.com/rust-lang/rust/blob/a5029ac0ab372aec515db2e718da6d7787f3d122/src/test/ui/commandline-argfile.rs)
for an example of a test that uses input normalization.

73
src/tests/ci.md Normal file
View File

@ -0,0 +1,73 @@
# Testing with CI
## Testing infrastructure
When a Pull Request is opened on GitHub, [GitHub Actions] will automatically
launch a build that will run all tests on some configurations
(x86_64-gnu-llvm-12 linux. x86_64-gnu-tools linux, mingw-check linux).
In essence, each runs `./x.py test` with various different options.
The integration bot [bors] is used for coordinating merges to the master branch.
When a PR is approved, it goes into a [queue] where merges are tested one at a
time on a wide set of platforms using GitHub Actions. Due to the limit on the
number of parallel jobs, we run CI under the [rust-lang-ci] organization except
for PRs.
Most platforms only run the build steps, some run a restricted set of tests,
only a subset run the full suite of tests (see Rust's [platform tiers]).
If everything passes, then all of the distribution artifacts that were
generated during the CI run are published.
[GitHub Actions]: https://github.com/rust-lang/rust/actions
[rust-lang-ci]: https://github.com/rust-lang-ci/rust/actions
[bors]: https://github.com/servo/homu
[queue]: https://bors.rust-lang.org/queue/rust
[platform tiers]: https://forge.rust-lang.org/release/platform-support.html#rust-platform-support
## Using CI to test
In some cases, a PR may run into problems with running tests on a particular
platform or configuration.
If you can't run those tests locally, don't hesitate to use CI resources to
try out a fix.
As mentioned above, opening or updating a PR will only run on a small subset
of configurations.
Only when a PR is approved will it go through the full set of test configurations.
However, you can try one of those configurations in your PR before it is approved.
For example, if a Windows build fails, but you don't have access to a Windows
machine, you can try running the Windows job that failed on CI within your PR
after pushing a possible fix.
To do this, you'll need to edit [`src/ci/github-actions/ci.yml`].
The `jobs` section defines the jobs that will run.
The `jobs.pr` section defines everything that will run in a push to a PR.
The `jobs.auto` section defines the full set of tests that are run after a PR is approved.
You can copy one of the definitions from the `auto` section up to the `pr` section.
For example, the `x86_64-msvc-1` and `x86_64-msvc-2` jobs are responsible for
running the 64-bit MSVC tests.
You can copy those up to the `jobs.pr.strategy.matrix.include` section with
the other jobs.
The comment at the top of `ci.yml` will tell you to run this command:
```sh
./x.py run src/tools/expand-yaml-anchors
````
This will generate the true [`.github/workflows/ci.yml`] which is what GitHub
Actions uses.
Then, you can commit those two files and push to GitHub.
GitHub Actions should launch the tests.
After you have finished, don't forget to remove any changes you have made to `ci.yml`.
Although you are welcome to use CI, just be conscientious that this is a shared
resource with limited concurrency.
Try not to enable too many jobs at once (one or two should be sufficient in
most cases).
[`src/ci/github-actions/ci.yml`]: https://github.com/rust-lang/rust/blob/master/src/ci/github-actions/ci.yml
[`.github/workflows/ci.yml`]: https://github.com/rust-lang/rust/blob/master/.github/workflows/ci.yml#L1

526
src/tests/compiletest.md Normal file
View File

@ -0,0 +1,526 @@
# Compiletest
<!-- toc -->
## Introduction
`compiletest` is the main test harness of the Rust test suite.
It allows test authors to organize large numbers of tests
(the Rust compiler has many thousands),
efficient test execution (parallel execution is supported),
and allows the test author to configure behavior and expected results of both
individual and groups of tests.
`compiletest` may check test code for success, for runtime failure,
or for compile-time failure.
Tests are typically organized as a Rust source file with annotations in
comments before and/or within the test code.
These comments serve to direct `compiletest` on if or how to run the test,
what behavior to expect, and more.
See [header commands](headers.md) and the test suite documentation below
for more details on these annotations.
See the [Adding new tests](adding.md) chapter for a tutorial on creating a new
test, and the [Running tests](running.md) chapter on how to run the test
suite.
## Test suites
All of the tests are in the [`src/test`] directory.
The tests are organized into "suites", with each suite in a separate subdirectory.
Each test suite behaves a little differently, with different compiler behavior
and different checks for correctness.
For example, the [`src/test/incremental`] directory contains tests for
incremental compilation.
The various suites are defined in [`src/tools/compiletest/src/common.rs`] in
the `pub enum Mode` declaration.
The following test suites are available, with links for more information:
- [`ui`](ui.md) — tests that check the stdout/stderr from the compilation
and/or running the resulting executable
- `ui-fulldeps` — `ui` tests which require a linkable build of `rustc` (such
as using `extern crate rustc_span;` or used as a plugin)
- [`pretty`](#pretty-printer-tests) — tests for pretty printing
- [`incremental`](#incremental-tests) — tests incremental compilation behavior
- [`debuginfo`](#debuginfo-tests) — tests for debuginfo generation running debuggers
- [`codegen`](#codegen-tests) — tests for code generation
- [`codegen-units`](#codegen-units-tests) — tests for codegen unit partitioning
- [`assembly`](#assembly-tests) — verifies assembly output
- [`mir-opt`](#mir-opt-tests) — tests for MIR generation
- [`run-make`](#run-make-tests) — general purpose tests using a Makefile
- `run-make-fulldeps` — `run-make` tests which require a linkable build of `rustc`,
or the rust demangler
- [`run-pass-valgrind`](#valgrind-tests) — tests run with Valgrind
- Rustdoc tests:
- `rustdoc` — tests for rustdoc, making sure that the generated files
contain the expected documentation.
- `rustdoc-gui` — TODO
- `rustdoc-js` — TODO
- `rustdoc-js-std` — TODO
- `rustdoc-json` — TODO
- `rustdoc-ui` — TODO
[`src/test`]: https://github.com/rust-lang/rust/blob/master/src/test
[`src/tools/compiletest/src/common.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/common.rs
### Pretty-printer tests
The tests in [`src/test/pretty`] exercise the "pretty-printing" functionality of `rustc`.
The `-Z unpretty` CLI option for `rustc` causes it to translate the input source
into various different formats, such as the Rust source after macro expansion.
The pretty-printer tests have several [header commands](headers.md) described below.
These commands can significantly change the behavior of the test, but the
default behavior without any commands is to:
1. Run `rustc -Zunpretty=normal` on the source file
2. Run `rustc -Zunpretty=normal` on the output of the previous step
3. The output of the previous two steps should be the same.
4. Run `rustc -Zno-codegen` on the output to make sure that it can type check
(this is similar to running `cargo check`)
If any of the commands above fail, then the test fails.
The header commands for pretty-printing tests are:
* `pretty-mode` specifies the mode pretty-print tests should run in
(that is, the argument to `-Zunpretty`).
The default is `normal` if not specified.
* `pretty-compare-only` causes a pretty test to only compare the pretty-printed output
(stopping after step 3 from above).
It will not try to compile the expanded output to type check it.
This is needed for a pretty-mode that does not expand to valid
Rust, or for other situations where the expanded output cannot be compiled.
* `pretty-expanded` allows a pretty test to also check that the expanded
output can be type checked.
That is, after the steps above, it does two more steps:
> 5. Run `rustc -Zunpretty=expanded` on the original source
> 6. Run `rustc -Zno-codegen` on the expanded output to make sure that it can type check
This is needed because not all code can be compiled after being expanded.
Pretty tests should specify this if they can.
An example where this cannot be used is if the test includes `println!`.
That macro expands to reference private internal functions of the standard
library that cannot be called directly without the `fmt_internals` feature
gate.
More history about this may be found in
[#23616](https://github.com/rust-lang/rust/issues/23616#issuecomment-484999901).
* `pp-exact` is used to ensure a pretty-print test results in specific output.
If specified without a value, then it means the pretty-print output should
match the original source.
If specified with a value, as in `// pp-exact:foo.pp`,
it will ensure that the pretty-printed output matches the contents of the given file.
Otherwise, if `pp-exact` is not specified, then the pretty-printed output
will be pretty-printed one more time, and the output of the two
pretty-printing rounds will be compared to ensure that the pretty-printed
output converges to a steady state.
[`src/test/pretty`]: https://github.com/rust-lang/rust/tree/master/src/test/pretty
### Incremental tests
The tests in [`src/test/incremental`] exercise incremental compilation.
They use [revision headers](#revisions) to tell compiletest to run the
compiler in a series of steps.
Compiletest starts with an empty directory with the `-C incremental` flag, and
then runs the compiler for each revision, reusing the incremental results from
previous steps.
The revisions should start with:
* `rpass` — the test should compile and run successfully
* `rfail` — the test should compile successfully, but the executable should fail to run
* `cfail` — the test should fail to compile
To make the revisions unique, you should add a suffix like `rpass1` and `rpass2`.
To simulate changing the source, compiletest also passes a `--cfg` flag with
the current revision name.
For example, this will run twice, simulating changing a function:
```rust,ignore
// revisions: rpass1 rpass2
#[cfg(rpass1)]
fn foo() {
println!("one");
}
#[cfg(rpass2)]
fn foo() {
println!("two");
}
fn main() { foo(); }
```
`cfail` tests support the `forbid-output` header to specify that a certain
substring must not appear anywhere in the compiler output.
This can be useful to ensure certain errors do not appear, but this can be
fragile as error messages change over time, and a test may no longer be
checking the right thing but will still pass.
`cfail` tests support the `should-ice` header to specify that a test should
cause an Internal Compiler Error (ICE).
This is a highly specialized header to check that the incremental cache
continues to work after an ICE.
[`src/test/incremental`]: https://github.com/rust-lang/rust/tree/master/src/test/incremental
### Debuginfo tests
The tests in [`src/test/debuginfo`] test debuginfo generation.
They build a program, launch a debugger, and issue commands to the debugger.
A single test can work with cdb, gdb, and lldb.
Most tests should have the `// compile-flags: -g` header or something similar
to generate the appropriate debuginfo.
To set a breakpoint on a line, add a `// #break` comment on the line.
The debuginfo tests consist of a series of debugger commands along with
"check" lines which specify output that is expected from the debugger.
The commands are comments of the form `// $DEBUGGER-command:$COMMAND` where
`$DEBUGGER` is the debugger being used and `$COMMAND` is the debugger command
to execute.
The debugger values can be:
* `cdb`
* `gdb`
* `gdbg` — GDB without Rust support (versions older than 7.11)
* `gdbr` — GDB with Rust support
* `lldb`
* `lldbg` — LLDB without Rust support
* `lldbr` — LLDB with Rust support (this no longer exists)
The command to check the output are of the form `// $DEBUGGER-check:$OUTPUT`
where `$OUTPUT` is the output to expect.
For example, the following will build the test, start the debugger, set a
breakpoint, launch the program, inspect a value, and check what the debugger
prints:
```rust,ignore
// compile-flags: -g
// lldb-command: run
// lldb-command: print foo
// lldb-check: $0 = 123
fn main() {
let foo = 123;
b(); // #break
}
fn b() {}
```
The following [header commands](headers.md) are available to disable a
test based on the debugger currently being used:
* `min-cdb-version: 10.0.18317.1001` — ignores the test if the version of cdb
is below the given version
* `min-gdb-version: 8.2` — ignores the test if the version of gdb is below the
given version
* `ignore-gdb-version: 9.2` — ignores the test if the version of gdb is equal
to the given version
* `ignore-gdb-version: 7.11.90 - 8.0.9` — ignores the test if the version of
gdb is in a range (inclusive)
* `min-lldb-version: 310` — ignores the test if the version of lldb is below
the given version
* `rust-lldb` — ignores the test if lldb is not contain the Rust plugin.
NOTE: The "Rust" version of LLDB doesn't exist anymore, so this will always be ignored.
This should probably be removed.
[`src/test/debuginfo`]: https://github.com/rust-lang/rust/tree/master/src/test/debuginfo
### Codegen tests
The tests in [`src/test/codegen`] test LLVM code generation.
They compile the test with the `--emit=llvm-ir` flag to emit LLVM IR.
They then run the LLVM [FileCheck] tool.
The test is annotated with various `// CHECK` comments to check the generated code.
See the FileCheck documentation for a tutorial and more information.
See also the [assembly tests](#assembly-tests) for a similar set of tests.
[`src/test/codegen`]: https://github.com/rust-lang/rust/tree/master/src/test/codegen
[FileCheck]: https://llvm.org/docs/CommandGuide/FileCheck.html
### Assembly tests
The tests in [`src/test/assembly`] test LLVM assembly output.
They compile the test with the `--emit=asm` flag to emit a `.s` file with the
assembly output.
They then run the LLVM [FileCheck] tool.
Each test should be annotated with the `// assembly-output:` header
with a value of either `emit-asm` or `ptx-linker` to indicate
the type of assembly output.
Then, they should be annotated with various `// CHECK` comments to check the
assembly output.
See the FileCheck documentation for a tutorial and more information.
See also the [codegen tests](#codegen-tests) for a similar set of tests.
[`src/test/assembly`]: https://github.com/rust-lang/rust/tree/master/src/test/assembly
### Codegen-units tests
The tests in [`src/test/codegen-units`] test the
[monomorphization](../backend/monomorph.md) collector and CGU partitioning.
These tests work by running `rustc` with a flag to print the result of the
monomorphization collection pass, and then special annotations in the file are
used to compare against that.
Each test should be annotated with the `// compile-flags:-Zprint-mono-items=VAL`
header with the appropriate VAL to instruct `rustc` to print the
monomorphization information.
Then, the test should be annotated with comments of the form `//~ MONO_ITEM name`
where `name` is the monomorphized string printed by rustc like `fn <u32 as Trait>::foo`.
To check for CGU partitioning, a comment of the form `//~ MONO_ITEM name @@ cgu`
where `cgu` is a space separated list of the CGU names and the linkage
information in brackets.
For example: `//~ MONO_ITEM static function::FOO @@ statics[Internal]`
[`src/test/codegen-units`]: https://github.com/rust-lang/rust/tree/master/src/test/codegen-units
### Mir-opt tests
The tests in [`src/test/mir-opt`] check parts of the generated MIR to make
sure it is generated correctly and is doing the expected optimizations.
Check out the [MIR Optimizations](../mir/optimizations.md) chapter for more.
Compiletest will build the test with several flags to dump the MIR output and
set a baseline for optimizations:
* `-Copt-level=1`
* `-Zdump-mir=all`
* `-Zmir-opt-level=4`
* `-Zvalidate-mir`
* `-Zdump-mir-exclude-pass-number`
The test should be annotated with `// EMIT_MIR` comments that specify files that
will contain the expected MIR output.
You can use `x.py test --bless` to create the initial expected files.
There are several forms the `EMIT_MIR` comment can take:
* `// EMIT_MIR $MIR_PATH.mir` — This will check that the given filename
matches the exact output from the MIR dump.
For example, `my_test.main.SimplifyCfg-elaborate-drops.after.mir` will load
that file from the test directory, and compare it against the dump from
rustc.
Checking the "after" file (which is after optimization) is useful if you are
interested in the final state after an optimization.
Some rare cases may want to use the "before" file for completeness.
* `// EMIT_MIR $MIR_PATH.diff` — where `$MIR_PATH` is the filename of the MIR
dump, such as `my_test_name.my_function.EarlyOtherwiseBranch`.
Compiletest will diff the `.before.mir` and `.after.mir` files, and compare
the diff output to the expected `.diff` file from the `EMIT_MIR` comment.
This is useful if you want to see how an optimization changes the MIR.
* `// EMIT_MIR $MIR_PATH.dot` or `$MIR_PATH.html` — These are special cases
for other MIR outputs (via `-Z dump-mir-graphviz` and `-Z dump-mir-spanview`)
that will check that the output matches the given file.
By default 32 bit and 64 bit targets use the same dump files, which can be
problematic in the presence of pointers in constants or other bit width
dependent things. In that case you can add `// EMIT_MIR_FOR_EACH_BIT_WIDTH` to
your test, causing separate files to be generated for 32bit and 64bit systems.
[`src/test/mir-opt`]: https://github.com/rust-lang/rust/tree/master/src/test/mir-opt
### Run-make tests
The tests in [`src/test/run-make`] are general-purpose tests using Makefiles
which provide the ultimate in flexibility.
These should be used as a last resort.
If possible, you should use one of the other test suites.
If there is some minor feature missing which you need for your test,
consider extending compiletest to add a header command for what you need.
However, sometimes just running a bunch of commands is really what you
need, `run-make` is here to the rescue!
Each test should be in a separate directory with a `Makefile` indicating the
commands to run.
There is a [`tools.mk`] Makefile which you can include which provides a bunch of
utilities to make it easier to run commands and compare outputs.
Take a look at some of the other tests for some examples on how to get started.
[`tools.mk`]: https://github.com/rust-lang/rust/blob/master/src/test/run-make-fulldeps/tools.mk
[`src/test/run-make`]: https://github.com/rust-lang/rust/tree/master/src/test/run-make
### Valgrind tests
The tests in [`src/test/run-pass-valgrind`] are for use with [Valgrind].
These are currently vestigial, as Valgrind is no longer used in CI.
These may be removed in the future.
[Valgrind]: https://valgrind.org/
[`src/test/run-pass-valgrind`]: https://github.com/rust-lang/rust/tree/master/src/test/run-pass-valgrind
## Building auxiliary crates
It is common that some tests require additional auxiliary crates to be compiled.
There are two [headers](headers.md) to assist with that:
* `aux-build`
* `aux-crate`
`aux-build` will build a separate crate from the named source file.
The source file should be in a directory called `auxiliary` beside the test file.
```rust,ignore
// aux-build: my-helper.rs
extern crate my_helper;
// ... You can use my_helper.
```
The aux crate will be built as a dylib if possible (unless on a platform that
does not support them, or the `no-prefer-dynamic` header is specified in the
aux file).
The `-L` flag is used to find the extern crates.
`aux-crate` is very similar to `aux-build`; however, it uses the `--extern`
flag to link to the extern crate.
That allows you to specify the additional syntax of the `--extern` flag, such
as renaming a dependency.
For example, `// aux-crate:foo=bar.rs` will compile `auxiliary/bar.rs` and
make it available under then name `foo` within the test.
This is similar to how Cargo does dependency renaming.
### Auxiliary proc-macro
If you want a proc-macro dependency, then there currently is some ceremony
needed.
Place the proc-macro itself in a file like `auxiliary/my-proc-macro.rs`
with the following structure:
```rust,ignore
// force-host
// no-prefer-dynamic
#![crate_type = "proc-macro"]
extern crate proc_macro;
use proc_macro::TokenStream;
#[proc_macro]
pub fn foo(input: TokenStream) -> TokenStream {
"".parse().unwrap()
}
```
The `force-host` is needed because proc-macros are loaded in the host
compiler, and `no-prefer-dynamic` is needed to tell compiletest to not use
`prefer-dynamic` which is not compatible with proc-macros.
The `#![crate_type]` attribute is needed to specify the correct crate-type.
Then in your test, you can build with with `aux-build`:
```rust,ignore
// aux-build: my-proc-macro.rs
extern crate my_proc_macro;
fn main() {
my_proc_macro::foo!();
}
```
## Revisions
Certain classes of tests support "revisions" (as of <!-- date: 2021-02 --> February 2021,
this includes compile-fail, run-fail, and incremental, though
incremental tests are somewhat different).
Revisions allow a single test file to be used for multiple tests.
This is done by adding a special header at the top of the file:
```rust,ignore
// revisions: foo bar baz
```
This will result in the test being compiled (and tested) three times,
once with `--cfg foo`, once with `--cfg bar`, and once with `--cfg
baz`.
You can therefore use `#[cfg(foo)]` etc within the test to tweak
each of these results.
You can also customize headers and expected error messages to a particular
revision. To do this, add `[foo]` (or `bar`, `baz`, etc) after the `//`
comment, like so:
```rust,ignore
// A flag to pass in only for cfg `foo`:
//[foo]compile-flags: -Z verbose
#[cfg(foo)]
fn test_foo() {
let x: usize = 32_u32; //[foo]~ ERROR mismatched types
}
```
Note that not all headers have meaning when customized to a revision.
For example, the `ignore-test` header (and all "ignore" headers)
currently only apply to the test as a whole, not to particular
revisions. The only headers that are intended to really work when
customized to a revision are error patterns and compiler flags.
## Compare modes
Compiletest can be run in different modes, called *compare modes*, which can
be used to compare the behavior of all tests with different compiler flags
enabled.
This can help highlight what differences might appear with certain flags, and
check for any problems that might arise.
To run the tests in a different mode, you need to pass the `--compare-mode`
CLI flag:
```bash
./x.py test src/test/ui --compare-mode=nll
```
The possible compare modes are:
* `nll` — Runs in the "true" NLL mode with `-Zborrowck=mir`.
See [UI compare modes](ui.md#compare-modes) for more.
* `polonius` — Runs with Polonius with `-Zpolonius -Zborrowck=mir`, and reuses
the `nll` stderr files.
* `chalk` — Runs with Chalk with `-Zchalk`.
* `split-dwarf` — Runs with unpacked split-DWARF with `-Csplit-debuginfo=unpacked`.
* `split-dwarf-single` — Runs with packed split-DWARF with `-Csplit-debuginfo=packed`.
In CI, compare modes are only used in one Linux builder, and only with the
following settings:
* `src/test/ui`: Uses `nll` mode.
* `src/test/debuginfo`: Uses `split-dwarf` mode.
This helps ensure that none of the debuginfo tests are affected when
enabling split-DWARF.
Note that compare modes are separate to [revisions](#revisions).
All revisions are tested when running `./x.py test src/test/ui`, however
compare-modes must be manually run individually via the `--compare-mode` flag.

45
src/tests/crater.md Normal file
View File

@ -0,0 +1,45 @@
# Crater
[Crater](https://github.com/rust-lang/crater) is a tool for compiling
and running tests for _every_ crate on [crates.io](https://crates.io) (and a
few on GitHub). It is mainly used for checking the extent of breakage when
implementing potentially breaking changes and ensuring lack of breakage by
running beta vs stable compiler versions.
## When to run Crater
You should request a crater run if your PR makes large changes to the compiler
or could cause breakage. If you are unsure, feel free to ask your PR's reviewer.
## Requesting Crater Runs
The rust team maintains a few machines that can be used for running crater runs
on the changes introduced by a PR. If your PR needs a crater run, leave a
comment for the triage team in the PR thread. Please inform the team whether
you require a "check-only" crater run, a "build only" crater run, or a
"build-and-test" crater run. The difference is primarily in time; the
conservative (if you're not sure) option is to go for the build-and-test run.
If making changes that will only have an effect at compile-time (e.g.,
implementing a new trait) then you only need a check run.
Your PR will be enqueued by the triage team and the results will be posted when
they are ready. Check runs will take around ~3-4 days, with the other two
taking 5-6 days on average.
While crater is really useful, it is also important to be aware of a few
caveats:
- Not all code is on crates.io! There is a lot of code in repos on GitHub and
elsewhere. Also, companies may not wish to publish their code. Thus, a
successful crater run is not a magically green light that there will be no
breakage; you still need to be careful.
- Crater only runs Linux builds on x86_64. Thus, other architectures and
platforms are not tested. Critically, this includes Windows.
- Many crates are not tested. This could be for a lot of reasons, including
that the crate doesn't compile any more (e.g. used old nightly features),
has broken or flaky tests, requires network access, or other reasons.
- Before crater can be run, `@bors try` needs to succeed in building artifacts.
This means that if your code doesn't compile, you cannot run crater.

52
src/tests/docker.md Normal file
View File

@ -0,0 +1,52 @@
# Testing with Docker
The Rust tree includes [Docker] image definitions for the platforms used on
GitHub Actions in [`src/ci/docker`].
The script [`src/ci/docker/run.sh`] is used to build the Docker image, run it,
build Rust within the image, and run the tests.
You can run these images on your local development machine. This can be
helpful to test environments different from your local system. First you will
need to install Docker on a Linux, Windows, or macOS system (typically Linux
will be much faster than Windows or macOS because the later use virtual
machines to emulate a Linux environment). To enter interactive mode which will
start a bash shell in the container, run `src/ci/docker/run.sh --dev <IMAGE>`
where `<IMAGE>` is one of the directory names in `src/ci/docker` (for example
`x86_64-gnu` is a fairly standard Ubuntu environment).
The docker script will mount your local Rust source tree in read-only mode,
and an `obj` directory in read-write mode. All of the compiler artifacts will
be stored in the `obj` directory. The shell will start out in the `obj`
directory. From there, you can run `../src/ci/run.sh` which will run the build
as defined by the image.
Alternatively, you can run individual commands to do specific tasks. For
example, you can run `python3 ../x.py test src/test/ui` to just run UI tests.
Note that there is some configuration in the [`src/ci/run.sh`] script that you
may need to recreate. Particularly, set `submodules = false` in your
`config.toml` so that it doesn't attempt to modify the read-only directory.
Some additional notes about using the Docker images:
- Some of the std tests require IPv6 support. Docker on Linux seems to have it
disabled by default. Run the commands in [`enable-docker-ipv6.sh`] to enable
IPv6 before creating the container. This only needs to be done once.
- The container will be deleted automatically when you exit the shell, however
the build artifacts persist in the `obj` directory. If you are switching
between different Docker images, the artifacts from previous environments
stored in the `obj` directory may confuse the build system. Sometimes you
will need to delete parts or all of the `obj` directory before building
inside the container.
- The container is bare-bones, with only a minimal set of packages. You may
want to install some things like `apt install less vim`.
- You can open multiple shells in the container. First you need the container
name (a short hash), which is displayed in the shell prompt, or you can run
`docker container ls` outside of the container to list the available
containers. With the container name, run `docker exec -it <CONTAINER>
/bin/bash` where `<CONTAINER>` is the container name like `4ba195e95cef`.
[Docker]: https://www.docker.com/
[`src/ci/docker`]: https://github.com/rust-lang/rust/tree/master/src/ci/docker
[`src/ci/docker/run.sh`]: https://github.com/rust-lang/rust/blob/master/src/ci/docker/run.sh
[`src/ci/run.sh`]: https://github.com/rust-lang/rust/blob/master/src/ci/run.sh
[`enable-docker-ipv6.sh`]: https://github.com/rust-lang/rust/blob/master/src/ci/scripts/enable-docker-ipv6.sh

403
src/tests/headers.md Normal file
View File

@ -0,0 +1,403 @@
# Test headers
<!-- toc -->
Header commands are special comments that tell compiletest how to build and
interpret a test.
They must appear before the Rust source in the test.
They may also appear in Makefiles for [run-make tests](compiletest.md#run-make-tests).
They are normally put after the short comment that explains the point of this test.
For example, this test uses the `// compile-flags` command to specify a custom
flag to give to rustc when the test is compiled:
```rust,ignore
// Test the behavior of `0 - 1` when overflow checks are disabled.
// compile-flags: -C overflow-checks=off
fn main() {
let x = 0 - 1;
...
}
```
Header commands can be standalone (like `// run-pass`) or take a value (like
`// compile-flags: -C overflow-checks=off`).
## Header commands
The following is a list of header commands.
Commands are linked to sections the describe the command in more detail if available.
This list may not be exhaustive.
Header commands can generally be found by browsing the `TestProps` structure
found in [`header.rs`] from the compiletest source.
[`header.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/header.rs
* [Controlling pass/fail expectations](ui.md#controlling-passfail-expectations)
* `check-pass` — building (no codegen) should pass
* `build-pass` — building should pass
* `run-pass` — running the test should pass
* `check-fail` — building (no codegen) should fail (the default if no header)
* `build-fail` — building should fail
* `run-fail` — running should fail
* `ignore-pass` — ignores the `--pass` flag
* `check-run-results` — checks run-pass/fail-pass output
* [UI](ui.md) headers
* [`normalize-X`](ui.md#normalization) — normalize compiler output
* [`run-rustfix`](ui.md#rustfix-tests) — checks diagnostic suggestions
* [`rustfix-only-machine-applicable`](ui.md#rustfix-tests) — checks only
machine applicable suggestions
* [`stderr-per-bitwidth`](ui.md#output-comparison) — separate output per bit width
* [`dont-check-compiler-stderr`](ui.md#output-comparison) — don't validate stderr
* [`dont-check-compiler-stdout`](ui.md#output-comparison) — don't validate stdout
* [Building auxiliary crates](compiletest.md#building-auxiliary-crates)
* `aux-build`
* `aux-crate`
* [Pretty-printer](compiletest.md#pretty-printer-tests) headers
* `pretty-compare-only`
* `pretty-expanded`
* `pretty-mode`
* `pp-exact`
* [Ignoring tests](#ignoring-tests)
* `ignore-X`
* `only-X`
* `needs-X`
* `no-system-llvm`
* `min-llvm-versionX`
* `min-system-llvm-version`
* `ignore-llvm-version`
* `ignore-llvm-version`
* [Environment variable headers](#environment-variable-headers)
* `rustc-env`
* `exec-env`
* `unset-rustc-env`
* [Miscellaneous headers](#miscellaneous-headers)
* `compile-flags` — adds compiler flags
* `run-flags` — adds flags to executable tests
* `edition` — sets the edition
* `failure-status` — expected exit code
* `should-fail` — testing compiletest itself
* `gate-test-X` — feature gate testing
* [`error-pattern`](ui.md#error-pattern) — errors not on a line
* `incremental` — incremental tests not in the incremental test-suite
* `no-prefer-dynamic` — don't use `-C prefer-dynamic`, don't build as a dylib
* `force-host` — build only for the host target
* [`revisions`](compiletest.md#revisions) — compile multiple times
* [`forbid-output`](compiletest.md#incremental-tests) — incremental cfail rejects output pattern
* [`should-ice`](compiletest.md#incremental-tests) — incremental cfail should ICE
* [Assembly](compiletest.md#assembly-tests) headers
* `assembly-output` — the type of assembly output to check
### Ignoring tests
These header commands are used to ignore the test in some situations,
which means the test won't be compiled or run.
* `ignore-X` where `X` is a target detail or stage will ignore the
test accordingly (see below)
* `only-X` is like `ignore-X`, but will *only* run the test on that
target or stage
* `ignore-test` always ignores the test.
This can be used to temporarily disable a test if it is currently not working,
but you want to keep it in tree to re-enable it later.
Some examples of `X` in `ignore-X` or `only-X`:
* A full target triple: `aarch64-apple-ios`
* Architecture: `aarch64`, `arm`, `asmjs`, `mips`, `wasm32`, `x86_64`,
`x86`, ...
* OS: `android`, `emscripten`, `freebsd`, `ios`, `linux`, `macos`,
`windows`, ...
* Environment (fourth word of the target triple): `gnu`, `msvc`,
`musl`
* WASM: `wasm32-bare` matches `wasm32-unknown-unknown`.
`emscripten` also matches that target as well as the emscripten targets.
* Pointer width: `32bit`, `64bit`
* Endianness: `endian-big`
* Stage: `stage0`, `stage1`, `stage2`
* Channel: `stable`, `beta`
* When cross compiling: `cross-compile`
* When [remote testing] is used: `remote`
* When debug-assertions are enabled: `debug`
* When particular debuggers are being tested: `cdb`, `gdb`, `lldb`
* Specific [compare modes]: `compare-mode-nll`, `compare-mode-polonius`,
`compare-mode-chalk`, `compare-mode-split-dwarf`,
`compare-mode-split-dwarf-single`
The following header commands will check rustc build settings and target settings:
* `needs-asm-support` — ignores if it is running on a target that doesn't have
stable support for `asm!`
* `needs-profiler-support` — ignores if profiler support was not enabled for
the target (`profiler = true` in rustc's `config.toml`)
* `needs-sanitizer-support` — ignores if the sanitizer support was not enabled
for the target (`sanitizers = true` in rustc's `config.toml`)
* `needs-sanitizer-{address,hwaddress,leak,memory,thread}` — ignores
if the corresponding sanitizer is not enabled for the target
(AddressSanitizer, hardware-assisted AddressSanitizer, LeakSanitizer,
MemorySanitizer or ThreadSanitizer respectively)
* `needs-run-enabled` — ignores if it is a test that gets executed, and
running has been disabled. Running tests can be disabled with the `x.py test
--run=never` flag, or running on fuchsia.
* `needs-unwind` — ignores if the target does not support unwinding
* `needs-rust-lld` — ignores if the rust lld support is not enabled
(`rust.lld = true` in `config.toml`)
The following header commands will check LLVM support:
* `no-system-llvm` — ignores if the system llvm is used
* `min-llvm-version: 13.0` — ignored if the LLVM version is less than the given value
* `min-system-llvm-version: 12.0` — ignored if using a system LLVM and its
version is less than the given value
* `ignore-llvm-version: 9.0` — ignores a specific LLVM version
* `ignore-llvm-version: 7.0 - 9.9.9` — ignores LLVM versions in a range (inclusive)
* `needs-llvm-components: powerpc` — ignores if the specific LLVM component was not built.
Note: The test will fail on CI if the component does not exist.
* `needs-matching-clang` — ignores if the version of clang does not match the
LLVM version of rustc.
These tests are always ignored unless a special environment variable is set
(which is only done in one CI job).
See also [Debuginfo tests](compiletest.md#debuginfo-tests) for headers for
ignoring debuggers.
[remote testing]: running.md#running-tests-on-a-remote-machine
[compare modes]: ui.md#compare-modes
### Environment variable headers
The following headers affect environment variables.
* `rustc-env` is an environment variable to set when running `rustc` of the
form `KEY=VALUE`.
* `exec-env` is an environment variable to set when executing a test of the
form `KEY=VALUE`.
* `unset-rustc-env` specifies an environment variable to unset when running
`rustc`.
### Miscellaneous headers
The following headers are generally available, and not specific to particular
test suites.
* `compile-flags` passes extra command-line args to the compiler,
e.g. `compile-flags -g` which forces debuginfo to be enabled.
* `run-flags` passes extra args to the test if the test is to be executed.
* `edition` controls the edition the test should be compiled with
(defaults to 2015). Example usage: `// edition:2018`.
* `failure-status` specifies the numeric exit code that should be expected for
tests that expect an error.
If this is not set, the default is 1.
* `should-fail` indicates that the test should fail; used for "meta
testing", where we test the compiletest program itself to check that
it will generate errors in appropriate scenarios. This header is
ignored for pretty-printer tests.
* `gate-test-X` where `X` is a feature marks the test as "gate test"
for feature X.
Such tests are supposed to ensure that the compiler errors when usage of a
gated feature is attempted without the proper `#![feature(X)]` tag.
Each unstable lang feature is required to have a gate test.
This header is actually checked by [tidy](intro.md#tidy), it is not checked
by compiletest.
* `error-pattern` checks the diagnostics just like the `ERROR` annotation
without specifying error line. This is useful when the error doesn't give
any span. See [`error-pattern`](ui.md#error-pattern).
* `incremental` runs the test with the `-C incremental` flag and an empty
incremental directory. This should be avoided when possible; you should use
an *incremental mode* test instead. Incremental mode tests support running
the compiler multiple times and verifying that it can load the generated
incremental cache. This flag is for specialized circumstances, like checking
the interaction of codegen unit partitioning with generating an incremental
cache.
* `no-prefer-dynamic` will force an auxiliary crate to be built as an rlib
instead of a dylib. When specified in a test, it will remove the use of `-C
prefer-dynamic`. This can be useful in a variety of circumstances. For
example, it can prevent a proc-macro from being built with the wrong crate
type. Or if your test is specifically targeting behavior of other crate
types, it can be used to prevent building with the wrong crate type.
* `force-host` will force the test to build for the host platform instead of
the target. This is useful primarily for auxiliary proc-macros, which need
to be loaded by the host compiler.
## Substitutions
Headers values support substituting a few variables which will be replaced
with their corresponding value.
For example, if you need to pass a compiler flag with a path to a specific
file, something like the following could work:
```rust,ignore
// compile-flags: --remap-path-prefix={{src-base}}=/the/src
```
Where the sentinel `{{src-base}}` will be replaced with the appropriate path
described below:
- `{{cwd}}`: The directory where compiletest is run from. This may not be the
root of the checkout, so you should avoid using it where possible.
- Examples: `/path/to/rust`, `/path/to/build/root`
- `{{src-base}}`: The directory where the test is defined. This is equivalent to
`$DIR` for [output normalization].
- Example: `/path/to/rust/src/test/ui/error-codes`
- `{{build-base}}`: The base directory where the test's output goes. This is
equivalent to `$TEST_BUILD_DIR` for [output normalization].
- Example: `/path/to/rust/build/x86_64-unknown-linux-gnu/test/ui`
See [`src/test/ui/commandline-argfile.rs`](https://github.com/rust-lang/rust/blob/a5029ac0ab372aec515db2e718da6d7787f3d122/src/test/ui/commandline-argfile.rs)
for an example of a test that uses this substitution.
[output normalization]: ui.md#normalization
## Adding a new header command
One would add a new header command if there is a need to define some test
property or behavior on an individual, test-by-test basis.
A header command property serves as the header command's backing store (holds
the command's current value) at runtime.
To add a new header command property:
1. Look for the `pub struct TestProps` declaration in
[`src/tools/compiletest/src/header.rs`] and add the new public property to
the end of the declaration.
2. Look for the `impl TestProps` implementation block immediately following
the struct declaration and initialize the new property to its default
value.
### Adding a new header command parser
When `compiletest` encounters a test file, it parses the file a line at a time
by calling every parser defined in the `Config` struct's implementation block,
also in [`src/tools/compiletest/src/header.rs`] (note that the `Config`
struct's declaration block is found in [`src/tools/compiletest/src/common.rs`]).
`TestProps`'s `load_from()` method will try passing the current line of text to
each parser, which, in turn typically checks to see if the line begins with a
particular commented (`//`) header command such as `// must-compile-successfully`
or `// failure-status`. Whitespace after the comment marker is optional.
Parsers will override a given header command property's default value merely by
being specified in the test file as a header command or by having a parameter
value specified in the test file, depending on the header command.
Parsers defined in `impl Config` are typically named `parse_<header_command>`
(note kebab-case `<header-command>` transformed to snake-case
`<header_command>`). `impl Config` also defines several 'low-level' parsers
which make it simple to parse common patterns like simple presence or not
(`parse_name_directive()`), header-command:parameter(s)
(`parse_name_value_directive()`), optional parsing only if a particular `cfg`
attribute is defined (`has_cfg_prefix()`) and many more. The low-level parsers
are found near the end of the `impl Config` block; be sure to look through them
and their associated parsers immediately above to see how they are used to
avoid writing additional parsing code unnecessarily.
As a concrete example, here is the implementation for the
`parse_failure_status()` parser, in [`src/tools/compiletest/src/header.rs`]:
```diff
@@ -232,6 +232,7 @@ pub struct TestProps {
// customized normalization rules
pub normalize_stdout: Vec<(String, String)>,
pub normalize_stderr: Vec<(String, String)>,
+ pub failure_status: i32,
}
impl TestProps {
@@ -260,6 +261,7 @@ impl TestProps {
run_pass: false,
normalize_stdout: vec![],
normalize_stderr: vec![],
+ failure_status: 101,
}
}
@@ -383,6 +385,10 @@ impl TestProps {
if let Some(rule) = config.parse_custom_normalization(ln, "normalize-stderr") {
self.normalize_stderr.push(rule);
}
+
+ if let Some(code) = config.parse_failure_status(ln) {
+ self.failure_status = code;
+ }
});
for key in &["RUST_TEST_NOCAPTURE", "RUST_TEST_THREADS"] {
@@ -488,6 +494,13 @@ impl Config {
self.parse_name_directive(line, "pretty-compare-only")
}
+ fn parse_failure_status(&self, line: &str) -> Option<i32> {
+ match self.parse_name_value_directive(line, "failure-status") {
+ Some(code) => code.trim().parse::<i32>().ok(),
+ _ => None,
+ }
+ }
```
### Implementing the behavior change
When a test invokes a particular header command, it is expected that some
behavior will change as a result. What behavior, obviously, will depend on the
purpose of the header command. In the case of `failure-status`, the behavior
that changes is that `compiletest` expects the failure code defined by the
header command invoked in the test, rather than the default value.
Although specific to `failure-status` (as every header command will have a
different implementation in order to invoke behavior change) perhaps it is
helpful to see the behavior change implementation of one case, simply as an
example. To implement `failure-status`, the `check_correct_failure_status()`
function found in the `TestCx` implementation block, located in
[`src/tools/compiletest/src/runtest.rs`], was modified as per below:
```diff
@@ -295,11 +295,14 @@ impl<'test> TestCx<'test> {
}
fn check_correct_failure_status(&self, proc_res: &ProcRes) {
- // The value the Rust runtime returns on failure
- const RUST_ERR: i32 = 101;
- if proc_res.status.code() != Some(RUST_ERR) {
+ let expected_status = Some(self.props.failure_status);
+ let received_status = proc_res.status.code();
+
+ if expected_status != received_status {
self.fatal_proc_rec(
- &format!("failure produced the wrong error: {}", proc_res.status),
+ &format!("Error: expected failure status ({:?}) but received status {:?}.",
+ expected_status,
+ received_status),
proc_res,
);
}
@@ -320,7 +323,6 @@ impl<'test> TestCx<'test> {
);
let proc_res = self.exec_compiled_test();
-
if !proc_res.status.success() {
self.fatal_proc_rec("test run failed!", &proc_res);
}
@@ -499,7 +501,6 @@ impl<'test> TestCx<'test> {
expected,
actual
);
- panic!();
}
}
```
Note the use of `self.props.failure_status` to access the header command
property. In tests which do not specify the failure status header command,
`self.props.failure_status` will evaluate to the default value of 101 at the
time of this writing. But for a test which specifies a header command of, for
example, `// failure-status: 1`, `self.props.failure_status` will evaluate to
1, as `parse_failure_status()` will have overridden the `TestProps` default
value, for that test specifically.
[`src/tools/compiletest/src/header.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/header.rs
[`src/tools/compiletest/src/common.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/common.rs
[`src/tools/compiletest/src/runtest.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/runtest.rs

408
src/tests/intro.md
View File

@ -1,350 +1,154 @@
# The compiler testing framework
# Testing the compiler
<!-- toc -->
The Rust project runs a wide variety of different tests, orchestrated by
the build system (`./x.py test`). The main test harness for testing the
compiler itself is a tool called compiletest (located in the
[`src/tools/compiletest`] directory). This section gives a brief
overview of how the testing framework is setup, and then gets into some
of the details on [how to run tests](./running.html) as well as [how to
add new tests](./adding.html).
the build system (`./x.py test`).
This section gives a brief overview of the different testing tools.
Subsequent chapters dive into [running tests](running.md) and [adding new tests](adding.md).
[`src/tools/compiletest`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest
## Kinds of tests
## Compiletest test suites
There are several kinds of tests to exercise things in the Rust distribution.
Almost all of them are driven by `./x.py test`, with some exceptions noted below.
The compiletest tests are located in the tree in the [`src/test`]
directory. Immediately within you will see a series of subdirectories
(e.g. `ui`, `run-make`, and so forth). Each of those directories is
called a **test suite** – they house a group of tests that are run in
a distinct mode.
### Compiletest
The main test harness for testing the compiler itself is a tool called [compiletest].
It supports running different styles of tests, called *test suites*.
The tests are all located in the [`src/test`] directory.
The [Compiletest chapter][compiletest] goes into detail on how to use this tool.
> Example: `./x.py test src/test/ui`
[compiletest]: compiletest.md
[`src/test`]: https://github.com/rust-lang/rust/tree/master/src/test
Here is a brief summary of the test suites and what they mean. In some
cases, the test suites are linked to parts of the manual that give more
details.
### Package tests
- [`ui`](./adding.html#ui) – tests that check the exact
stdout/stderr from compilation and/or running the test
- `run-pass-valgrind` – tests that ought to run with valgrind
- `pretty` – tests targeting the Rust "pretty printer", which
generates valid Rust code from the AST
- `debuginfo` – tests that run in gdb or lldb and query the debug info
- `codegen` – tests that compile and then test the generated LLVM
code to make sure that the optimizations we want are taking effect.
See [LLVM docs](https://llvm.org/docs/CommandGuide/FileCheck.html) for how to
write such tests.
- `codegen-units` – tests for the [monomorphization](../backend/monomorph.md)
collector and CGU partitioning
- `assembly` – similar to `codegen` tests, but verifies assembly output
to make sure LLVM target backend can handle provided code.
- `mir-opt` – tests that check parts of the generated MIR to make
sure we are building things correctly or doing the optimizations we
expect.
- `incremental` – tests for incremental compilation, checking that
when certain modifications are performed, we are able to reuse the
results from previous compilations.
- `run-make` – tests that basically just execute a `Makefile`; the
ultimate in flexibility but quite annoying to write.
- `rustdoc` – tests for rustdoc, making sure that the generated files
contain the expected documentation.
- `*-fulldeps` – same as above, but indicates that the test depends
on things other than `std` (and hence those things must be built)
The standard library and many of the compiler packages include typical Rust `#[test]`
unit tests, integration tests, and documentation tests.
You can pass a path to `x.py` to almost any package in the `library` or `compiler` directory,
and `x.py` will essentially run `cargo test` on that package.
## Other Tests
Examples:
The Rust build system handles running tests for various other things,
including:
| Command | Description |
|---------|-------------|
| `./x.py test library/std` | Runs tests on `std` |
| `./x.py test library/core` | Runs tests on `core` |
| `./x.py test compiler/rustc_data_structures` | Runs tests on `rustc_data_structures` |
- **Tidy** – This is a custom tool used for validating source code
style and formatting conventions, such as rejecting long lines.
There is more information in the
[section on coding conventions](../conventions.html#formatting).
The standard library relies very heavily on documentation tests to cover its functionality.
However, unit tests and integration tests can also be used as needed.
Almost all of the compiler packages have doctests disabled.
Example: `./x.py test tidy`
The standard library and compiler always place all unit tests in a separate `tests` file
(this is enforced in [tidy][tidy-unit-tests]).
This approach ensures that when the test file is changed, the crate does not need to be recompiled.
For example:
- **Formatting** – Rustfmt is integrated with the build system to enforce
uniform style across the compiler. In the CI, we check that the formatting
is correct. The formatting check is also automatically run by the Tidy tool
mentioned above.
Example: `./x.py fmt --check` checks formatting and exits with an error if
formatting is needed.
Example: `./x.py fmt` runs rustfmt on the codebase.
Example: `./x.py test tidy --bless` does formatting before doing
other tidy checks.
- **Unit tests** – The Rust standard library and many of the Rust packages
include typical Rust `#[test]` unittests. Under the hood, `x.py` will run
`cargo test` on each package to run all the tests.
Example: `./x.py test library/std`
- **Doc tests** – Example code embedded within Rust documentation is executed
via `rustdoc --test`. Examples:
`./x.py test src/doc` – Runs `rustdoc --test` for all documentation in
`src/doc`.
`./x.py test --doc library/std` – Runs `rustdoc --test` on the standard
library.
- **Link checker** – A small tool for verifying `href` links within
documentation.
Example: `./x.py test src/tools/linkchecker`
- **Dist check** – This verifies that the source distribution tarball created
by the build system will unpack, build, and run all tests.
Example: `./x.py test distcheck`
- **Tool tests** – Packages that are included with Rust have all of their
tests run as well (typically by running `cargo test` within their
directory). This includes things such as cargo, clippy, rustfmt, rls, miri,
bootstrap (testing the Rust build system itself), etc.
- **Cargo test** – This is a small tool which runs `cargo test` on a few
significant projects (such as `servo`, `ripgrep`, `tokei`, etc.) just to
ensure there aren't any significant regressions.
Example: `./x.py test src/tools/cargotest`
## Testing infrastructure
When a Pull Request is opened on Github, [GitHub Actions] will automatically
launch a build that will run all tests on some configurations
(x86_64-gnu-llvm-8 linux. x86_64-gnu-tools linux, mingw-check linux). In
essence, it runs `./x.py test` after building for each of them.
The integration bot [bors] is used for coordinating merges to the master branch.
When a PR is approved, it goes into a [queue] where merges are tested one at a
time on a wide set of platforms using GitHub Actions. Due to the limit on the
number of parallel jobs, we run CI under the [rust-lang-ci] organization except
for PRs. Most platforms only run the build steps, some run a restricted set of
tests, only a subset run the full suite of tests (see Rust's [platform tiers]).
[GitHub Actions]: https://github.com/rust-lang/rust/actions
[rust-lang-ci]: https://github.com/rust-lang-ci/rust/actions
[bors]: https://github.com/servo/homu
[queue]: https://bors.rust-lang.org/queue/rust
[platform tiers]: https://forge.rust-lang.org/release/platform-support.html#rust-platform-support
## Testing with Docker images
The Rust tree includes [Docker] image definitions for the platforms used on
GitHub Actions in [`src/ci/docker`]. The script [`src/ci/docker/run.sh`] is used to build
the Docker image, run it, build Rust within the image, and run the tests.
You can run these images on your local development machine. This can be
helpful to test environments different from your local system. First you will
need to install Docker on a Linux, Windows, or macOS system (typically Linux
will be much faster than Windows or macOS because the later use virtual
machines to emulate a Linux environment). To enter interactive mode which will
start a bash shell in the container, run `src/ci/docker/run.sh --dev <IMAGE>`
where `<IMAGE>` is one of the directory names in `src/ci/docker` (for example
`x86_64-gnu` is a fairly standard Ubuntu environment).
The docker script will mount your local Rust source tree in read-only mode,
and an `obj` directory in read-write mode. All of the compiler artifacts will
be stored in the `obj` directory. The shell will start out in the `obj`
directory. From there, you can run `../src/ci/run.sh` which will run the build
as defined by the image.
Alternatively, you can run individual commands to do specific tasks. For
example, you can run `python3 ../x.py test src/test/ui` to just run UI tests.
Note that there is some configuration in the [`src/ci/run.sh`] script that you
may need to recreate. Particularly, set `submodules = false` in your
`config.toml` so that it doesn't attempt to modify the read-only directory.
Some additional notes about using the Docker images:
- Some of the std tests require IPv6 support. Docker on Linux seems to have it
disabled by default. Run the commands in [`enable-docker-ipv6.sh`] to enable
IPv6 before creating the container. This only needs to be done once.
- The container will be deleted automatically when you exit the shell, however
the build artifacts persist in the `obj` directory. If you are switching
between different Docker images, the artifacts from previous environments
stored in the `obj` directory may confuse the build system. Sometimes you
will need to delete parts or all of the `obj` directory before building
inside the container.
- The container is bare-bones, with only a minimal set of packages. You may
want to install some things like `apt install less vim`.
- You can open multiple shells in the container. First you need the container
name (a short hash), which is displayed in the shell prompt, or you can run
`docker container ls` outside of the container to list the available
containers. With the container name, run `docker exec -it <CONTAINER>
/bin/bash` where `<CONTAINER>` is the container name like `4ba195e95cef`.
[Docker]: https://www.docker.com/
[`src/ci/docker`]: https://github.com/rust-lang/rust/tree/master/src/ci/docker
[`src/ci/docker/run.sh`]: https://github.com/rust-lang/rust/blob/master/src/ci/docker/run.sh
[`src/ci/run.sh`]: https://github.com/rust-lang/rust/blob/master/src/ci/run.sh
[`enable-docker-ipv6.sh`]: https://github.com/rust-lang/rust/blob/master/src/ci/scripts/enable-docker-ipv6.sh
## Running tests on a remote machine
Tests may be run on a remote machine (e.g. to test builds for a different
architecture). This is done using `remote-test-client` on the build machine
to send test programs to `remote-test-server` running on the remote machine.
`remote-test-server` executes the test programs and sends the results back to
the build machine. `remote-test-server` provides *unauthenticated remote code
execution* so be careful where it is used.
To do this, first build `remote-test-server` for the remote
machine, e.g. for RISC-V
```sh
./x.py build src/tools/remote-test-server --target riscv64gc-unknown-linux-gnu
```rust,ignore
#[cfg(test)]
mod tests;
```
The binary will be created at
`./build/$HOST_ARCH/stage2-tools/$TARGET_ARCH/release/remote-test-server`. Copy
this over to the remote machine.
If it wasn't done this way, and the tests were placed in the same file as the source,
then changing or adding a test would cause the crate you are working on to be recompiled.
If you were working on something like `core`,
then that would require recompiling the entire standard library, and the entirety of `rustc`.
On the remote machine, run the `remote-test-server` with the `remote` argument
(and optionally `-v` for verbose output). Output should look like this:
```sh
$ ./remote-test-server -v remote
starting test server
listening on 0.0.0.0:12345!
```
`./x.py test` includes some CLI options for controlling the behavior with these tests:
You can test if the `remote-test-server` is working by connecting to it and
sending `ping\n`. It should reply `pong`:
```sh
$ nc $REMOTE_IP 12345
ping
pong
```
* `--doc` — Only runs documentation tests in the package.
* `--no-doc` — Run all tests *except* documentation tests.
To run tests using the remote runner, set the `TEST_DEVICE_ADDR` environment
variable then use `x.py` as usual. For example, to run `ui` tests for a RISC-V
machine with the IP address `1.2.3.4` use
```sh
export TEST_DEVICE_ADDR="1.2.3.4:12345"
./x.py test src/test/ui --target riscv64gc-unknown-linux-gnu
```
[tidy-unit-tests]: https://github.com/rust-lang/rust/blob/master/src/tools/tidy/src/unit_tests.rs
If `remote-test-server` was run with the verbose flag, output on the test machine
may look something like
```
[...]
run "/tmp/work/test1007/a"
run "/tmp/work/test1008/a"
run "/tmp/work/test1009/a"
run "/tmp/work/test1010/a"
run "/tmp/work/test1011/a"
run "/tmp/work/test1012/a"
run "/tmp/work/test1013/a"
run "/tmp/work/test1014/a"
run "/tmp/work/test1015/a"
run "/tmp/work/test1016/a"
run "/tmp/work/test1017/a"
run "/tmp/work/test1018/a"
[...]
```
### Tidy
Tests are built on the machine running `x.py` not on the remote machine. Tests
which fail to build unexpectedly (or `ui` tests producing incorrect build
output) may fail without ever running on the remote machine.
Tidy is a custom tool used for validating source code style and formatting conventions,
such as rejecting long lines.
There is more information in the [section on coding conventions](../conventions.md#formatting).
## Testing on emulators
> Example: `./x.py test tidy`
Some platforms are tested via an emulator for architectures that aren't
readily available. For architectures where the standard library is well
supported and the host operating system supports TCP/IP networking, see the
above instructions for testing on a remote machine (in this case the
remote machine is emulated).
### Formatting
There is also a set of tools for orchestrating running the
tests within the emulator. Platforms such as `arm-android` and
`arm-unknown-linux-gnueabihf` are set up to automatically run the tests under
emulation on GitHub Actions. The following will take a look at how a target's tests
are run under emulation.
Rustfmt is integrated with the build system to enforce uniform style across the compiler.
The formatting check is automatically run by the Tidy tool mentioned above.
The Docker image for [armhf-gnu] includes [QEMU] to emulate the ARM CPU
architecture. Included in the Rust tree are the tools [remote-test-client]
and [remote-test-server] which are programs for sending test programs and
libraries to the emulator, and running the tests within the emulator, and
reading the results. The Docker image is set up to launch
`remote-test-server` and the build tools use `remote-test-client` to
communicate with the server to coordinate running tests (see
[src/bootstrap/test.rs]).
Examples:
> TODO:
> Is there any support for using an iOS emulator?
>
> It's also unclear to me how the wasm or asm.js tests are run.
| Command | Description |
|---------|-------------|
| `./x.py fmt --check` | Checks formatting and exits with an error if formatting is needed. |
| `./x.py fmt` | Runs rustfmt across the entire codebase. |
| `./x.py test tidy --bless` | First runs rustfmt to format the codebase, then runs tidy checks. |
[armhf-gnu]: https://github.com/rust-lang/rust/tree/master/src/ci/docker/host-x86_64/armhf-gnu/Dockerfile
[QEMU]: https://www.qemu.org/
[remote-test-client]: https://github.com/rust-lang/rust/tree/master/src/tools/remote-test-client
[remote-test-server]: https://github.com/rust-lang/rust/tree/master/src/tools/remote-test-server
[src/bootstrap/test.rs]: https://github.com/rust-lang/rust/tree/master/src/bootstrap/test.rs
### Book documentation tests
## Crater
All of the books that are published have their own tests,
primarily for validating that the Rust code examples pass.
Under the hood, these are essentially using `rustdoc --test` on the markdown files.
The tests can be run by passing a path to a book to `./x.py test`.
[Crater](https://github.com/rust-lang/crater) is a tool for compiling
and running tests for _every_ crate on [crates.io](https://crates.io) (and a
few on GitHub). It is mainly used for checking for extent of breakage when
implementing potentially breaking changes and ensuring lack of breakage by
running beta vs stable compiler versions.
> Example: `./x.py test src/doc/book`
### When to run Crater
### Documentation link checker
You should request a crater run if your PR makes large changes to the compiler
or could cause breakage. If you are unsure, feel free to ask your PR's reviewer.
Links across all documentation is validated with a link checker tool.
### Requesting Crater Runs
> Example: `./x.py test src/tools/linkchecker`
The Rust team maintains a few machines that can be used for running crater runs
on the changes introduced by a PR. If your PR needs a crater run, leave a
comment for the triage team in the PR thread. Please inform the team whether
you require a "check-only" crater run, a "build only" crater run, or a
"build-and-test" crater run. The difference is primarily in time; the
conservative (if you're not sure) option is to go for the build-and-test run.
If making changes that will only have an effect at compile-time (e.g.,
implementing a new trait) then you only need a check run.
This requires building all of the documentation, which might take a while.
Your PR will be enqueued by the triage team and the results will be posted when
they are ready. Check runs will take around ~3-4 days, with the other two
taking 5-6 days on average.
### Dist check
While crater is really useful, it is also important to be aware of a few
caveats:
`distcheck` verifies that the source distribution tarball created by the build system
will unpack, build, and run all tests.
- Not all code is on crates.io! There is a lot of code in repos on GitHub and
elsewhere. Also, companies may not wish to publish their code. Thus, a
successful crater run is not a magically green light that there will be no
breakage; you still need to be careful.
> Example: `./x.py test distcheck`
- Crater only runs Linux builds on x86_64. Thus, other architectures and
platforms are not tested. Critically, this includes Windows.
### Tool tests
- Many crates are not tested. This could be for a lot of reasons, including
that the crate doesn't compile any more (e.g. used old nightly features),
has broken or flaky tests, requires network access, or other reasons.
Packages that are included with Rust have all of their tests run as well.
This includes things such as cargo, clippy, rustfmt, rls, miri, bootstrap
(testing the Rust build system itself), etc.
- Before crater can be run, `@bors try` needs to succeed in building artifacts.
This means that if your code doesn't compile, you cannot run crater.
Most of the tools are located in the [`src/tools`] directory.
To run the tool's tests, just pass its path to `./x.py test`.
## Perf runs
> Example: `./x.py test src/tools/cargo`
A lot of work is put into improving the performance of the compiler and
preventing performance regressions. A "perf run" is used to compare the
performance of the compiler in different configurations for a large collection
of popular crates. Different configurations include "fresh builds", builds
with incremental compilation, etc.
Usually these tools involve running `cargo test` within the tool's directory.
The result of a perf run is a comparison between two versions of the
compiler (by their commit hashes).
In CI, some tools are allowed to fail.
Failures send notifications to the corresponding teams, and is tracked on the [toolstate website].
More information can be found in the [toolstate documentation].
You should request a perf run if your PR may affect performance, especially
if it can affect performance adversely.
[`src/tools`]: https://github.com/rust-lang/rust/tree/master/src/tools/
[toolstate documentation]: https://forge.rust-lang.org/infra/toolstate.html
[toolstate website]: https://rust-lang-nursery.github.io/rust-toolstate/
### Cargo test
`cargotest` is a small tool which runs `cargo test` on a few sample projects
(such as `servo`, `ripgrep`, `tokei`, etc.).
This ensures there aren't any significant regressions.
> Example: `./x.py test src/tools/cargotest`
### Crater
Crater is a tool which runs tests on many thousands of public projects.
This tool has its own separate infrastructure for running.
See the [Crater chapter](crater.md) for more details.
### Performance testing
A separate infrastructure is used for testing and tracking performance of the compiler.
See the [Performance testing chapter](perf.md) for more details.
## Further reading

50
src/tests/perf.md Normal file
View File

@ -0,0 +1,50 @@
# Performance testing
## rustc-perf
A lot of work is put into improving the performance of the compiler and
preventing performance regressions.
The [rustc-perf](https://github.com/rust-lang/rustc-perf) project provides
several services for testing and tracking performance.
It provides hosted infrastructure for running benchmarks as a service.
At this time, only `x86_64-unknown-linux-gnu` builds are tracked.
A "perf run" is used to compare the performance of the compiler in different
configurations for a large collection of popular crates.
Different configurations include "fresh builds", builds with incremental compilation, etc.
The result of a perf run is a comparison between two versions of the compiler
(by their commit hashes).
### Automatic perf runs
After every PR is merged, a suite of benchmarks are run against the compiler.
The results are tracked over time on the <https://perf.rust-lang.org/> website.
Any changes are noted in a comment on the PR.
### Manual perf runs
Additionally, performance tests can be ran before a PR is merged on an as-needed basis.
You should request a perf run if your PR may affect performance, especially if
it can affect performance adversely.
To evaluate the performance impact of a PR, write this comment on the PR:
`@bors try @rust-timer queue`
> **Note**: Only users authorized to do perf runs are allowed to post this comment.
> Teams that are allowed to use it are tracked in the [Teams
> repository](https://github.com/rust-lang/team) with the `perf = true` value
> in the `[permissions]` section (and bors permissions are also required).
> If you are not on one of those teams, feel free to ask for someone to post
> it for you (either on Zulip or ask the assigned reviewer).
This will first tell bors to do a "try" build which do a full release build
for `x86_64-unknown-linux-gnu`.
After the build finishes, it will place it in the queue to run the performance
suite against it.
After the performance tests finish, the bot will post a comment on the PR with
a summary and a link to a full report.
More details are available in the [perf collector
documentation](https://github.com/rust-lang/rustc-perf/blob/master/collector/README.md).

136
src/tests/running.md
View File

@ -18,8 +18,9 @@ I think is done, but rarely otherwise. -nmatsakis)
The test results are cached and previously successful tests are
`ignored` during testing. The stdout/stderr contents as well as a
timestamp file for every test can be found under `build/ARCH/test/`.
To force-rerun a test (e.g. in case the test runner fails to notice
a change) you can simply remove the timestamp file.
To force-rerun a test (e.g. in case the test runner fails to notice a change)
you can simply remove the timestamp file, or use the `--force-rerun` CLI
option.
Note that some tests require a Python-enabled gdb. You can test if
your gdb install supports Python by using the `python` command from
@ -143,6 +144,18 @@ to automatically adjust the `.stderr`, `.stdout` or `.fixed` files of
all tests. Of course you can also target just specific tests with the
`--test-args your_test_name` flag, just like when running the tests.
## Configuring test running
There are a few options for running tests:
* `config.toml` has the `rust.verbose-tests` option.
If `false`, each test will print a single dot (the default).
If `true`, the name of every test will be printed.
This is equivalent to the `--quiet` option in the [Rust test
harness](https://doc.rust-lang.org/rustc/tests/)
* The environment variable `RUST_TEST_THREADS` can be set to the number of
concurrent threads to use for testing.
## Passing `--pass $mode`
Pass UI tests now have three modes, `check-pass`, `build-pass` and
@ -156,9 +169,8 @@ exists in the test file. For example, you can run all the tests in
```
By passing `--pass $mode`, you can reduce the testing time. For each
mode, please see [here][mode].
[mode]: ./adding.md#tests-that-do-not-result-in-compile-errors
mode, please see [Controlling pass/fail
expectations](ui.md#controlling-passfail-expectations).
## Using incremental compilation
@ -193,17 +205,7 @@ To run the UI test suite in NLL mode, one would use the following:
./x.py test src/test/ui --compare-mode=nll
```
The possible compare modes are:
* nll - currently nll is implemented in migrate mode, this option runs with true nll.
* polonius
* chalk
* split-dwarf
* split-dwarf-single
Note that compare modes are separate to [revisions](./adding.html#revisions).
All revisions are tested when running `./x.py test src/test/ui`,
however compare-modes must be manually run individually via the `--compare-mode` flag.
See [Compare modes](compiletest.md#compare-modes) for more details.
## Running tests manually
@ -219,3 +221,105 @@ rustc +stage1 src/test/ui/issue-1234.rs
This is much faster, but doesn't always work. For example, some tests
include directives that specify specific compiler flags, or which rely
on other crates, and they may not run the same without those options.
## Running tests on a remote machine
Tests may be run on a remote machine (e.g. to test builds for a different
architecture). This is done using `remote-test-client` on the build machine
to send test programs to `remote-test-server` running on the remote machine.
`remote-test-server` executes the test programs and sends the results back to
the build machine. `remote-test-server` provides *unauthenticated remote code
execution* so be careful where it is used.
To do this, first build `remote-test-server` for the remote
machine, e.g. for RISC-V
```sh
./x.py build src/tools/remote-test-server --target riscv64gc-unknown-linux-gnu
```
The binary will be created at
`./build/$HOST_ARCH/stage2-tools/$TARGET_ARCH/release/remote-test-server`. Copy
this over to the remote machine.
On the remote machine, run the `remote-test-server` with the `remote` argument
(and optionally `-v` for verbose output). Output should look like this:
```sh
$ ./remote-test-server -v remote
starting test server
listening on 0.0.0.0:12345!
```
You can test if the `remote-test-server` is working by connecting to it and
sending `ping\n`. It should reply `pong`:
```sh
$ nc $REMOTE_IP 12345
ping
pong
```
To run tests using the remote runner, set the `TEST_DEVICE_ADDR` environment
variable then use `x.py` as usual. For example, to run `ui` tests for a RISC-V
machine with the IP address `1.2.3.4` use
```sh
export TEST_DEVICE_ADDR="1.2.3.4:12345"
./x.py test src/test/ui --target riscv64gc-unknown-linux-gnu
```
If `remote-test-server` was run with the verbose flag, output on the test machine
may look something like
```
[...]
run "/tmp/work/test1007/a"
run "/tmp/work/test1008/a"
run "/tmp/work/test1009/a"
run "/tmp/work/test1010/a"
run "/tmp/work/test1011/a"
run "/tmp/work/test1012/a"
run "/tmp/work/test1013/a"
run "/tmp/work/test1014/a"
run "/tmp/work/test1015/a"
run "/tmp/work/test1016/a"
run "/tmp/work/test1017/a"
run "/tmp/work/test1018/a"
[...]
```
Tests are built on the machine running `x.py` not on the remote machine. Tests
which fail to build unexpectedly (or `ui` tests producing incorrect build
output) may fail without ever running on the remote machine.
## Testing on emulators
Some platforms are tested via an emulator for architectures that aren't
readily available. For architectures where the standard library is well
supported and the host operating system supports TCP/IP networking, see the
above instructions for testing on a remote machine (in this case the
remote machine is emulated).
There is also a set of tools for orchestrating running the
tests within the emulator. Platforms such as `arm-android` and
`arm-unknown-linux-gnueabihf` are set up to automatically run the tests under
emulation on GitHub Actions. The following will take a look at how a target's tests
are run under emulation.
The Docker image for [armhf-gnu] includes [QEMU] to emulate the ARM CPU
architecture. Included in the Rust tree are the tools [remote-test-client]
and [remote-test-server] which are programs for sending test programs and
libraries to the emulator, and running the tests within the emulator, and
reading the results. The Docker image is set up to launch
`remote-test-server` and the build tools use `remote-test-client` to
communicate with the server to coordinate running tests (see
[src/bootstrap/test.rs]).
> TODO:
> Is there any support for using an iOS emulator?
>
> It's also unclear to me how the wasm or asm.js tests are run.
[armhf-gnu]: https://github.com/rust-lang/rust/tree/master/src/ci/docker/host-x86_64/armhf-gnu/Dockerfile
[QEMU]: https://www.qemu.org/
[remote-test-client]: https://github.com/rust-lang/rust/tree/master/src/tools/remote-test-client
[remote-test-server]: https://github.com/rust-lang/rust/tree/master/src/tools/remote-test-server
[src/bootstrap/test.rs]: https://github.com/rust-lang/rust/tree/master/src/bootstrap/test.rs

500
src/tests/ui.md Normal file
View File

@ -0,0 +1,500 @@
# UI tests
<!-- toc -->
UI tests are a particular [test suite](compiletest.md#test-suites) of compiletest.
## Introduction
The tests in [`src/test/ui`] are a collection of general-purpose tests which
primarily focus on validating the console output of the compiler, but can be
used for many other purposes.
For example, tests can also be configured to [run the resulting
program](#controlling-passfail-expectations) to verify its behavior.
[`src/test/ui`]: https://github.com/rust-lang/rust/blob/master/src/test/ui
## General structure of a test
A test consists of a Rust source file located anywhere in the `src/test/ui` directory.
For example, [`src/test/ui/hello.rs`] is a basic hello-world test.
Compiletest will use `rustc` to compile the test, and compare the output
against the expected output which is stored in a `.stdout` or `.stderr` file
located next to the test.
See [Output comparison](#output-comparison) for more.
Additionally, errors and warnings should be annotated with comments within
the source file.
See [Error annotations](#error-annotations) for more.
[Headers](headers.md) in the form of comments at the top of the file control
how the test is compiled and what the expected behavior is.
Tests are expected to fail to compile, since most tests are testing compiler
errors.
You can change that behavior with a header, see [Controlling pass/fail
expectations](#controlling-passfail-expectations).
By default, a test is built as an executable binary.
If you need a different crate type, you can use the `#![crate_type]` attribute
to set it as needed.
[`src/test/ui/hello.rs`]: https://github.com/rust-lang/rust/blob/master/src/test/ui/hello.rs
## Output comparison
UI tests store the expected output from the compiler in `.stderr` and
`.stdout` files next to the test.
You normally generate these files with the `--bless` CLI option, and then
inspect them manually to verify they contain what you expect.
The output is normalized to ignore unwanted differences, see the
[Normalization](#normalization) section.
If the file is missing, then compiletest expects the corresponding output to
be empty.
There can be multiple stdout/stderr files.
The general form is:
*test-name*`.`*revision*`.`*compare_mode*`.`*extension*
* *revision* is the [revision](#cfg-revisions) name.
This is not included when not using revisions.
* *compare_mode* is the [compare mode](#compare-modes).
This will only be checked when the given compare mode is active.
If the file does not exist, then compiletest will check for a file without
the compare mode.
* *extension* is the kind of output being checked:
* `stderr` — compiler stderr
* `stdout` — compiler stdout
* `run.stderr` — stderr when running the test
* `run.stdout` — stdout when running the test
* `64bit.stderr` — compiler stderr with `stderr-per-bitwidth` header on a 64-bit target
* `32bit.stderr` — compiler stderr with `stderr-per-bitwidth` header on a 32-bit target
A simple example would be `foo.stderr` next to a `foo.rs` test.
A more complex example would be `foo.my-revision.nll.stderr`.
There are several [headers](headers.md) which will change how compiletest will
check for output files:
* `stderr-per-bitwidth` — checks separate output files based on the target
pointer width. Consider using the `normalize-stderr` header instead (see
[Normalization](#normalization)).
* `dont-check-compiler-stderr` — Ignores stderr from the compiler.
* `dont-check-compiler-stdout` — Ignores stdout from the compiler.
UI tests run with with `-Zdeduplicate-diagnostics=no` flag which disables
rustc's built-in diagnostic deduplication mechanism.
This means you may see some duplicate messages in the output.
This helps illuminate situations where duplicate diagnostics are being
generated.
### Normalization
The compiler output is normalized to eliminate output difference between
platforms, mainly about filenames.
Compiletest makes the following replacements on the compiler output:
- The directory where the test is defined is replaced with `$DIR`.
Example: `/path/to/rust/src/test/ui/error-codes`
- The directory to the standard library source is replaced with `$SRC_DIR`.
Example: `/path/to/rust/library`
- Line and column numbers for paths in `$SRC_DIR` are replaced with `LL:COL`.
This helps ensure that changes to the layout of the standard library do not
cause widespread changes to the `.stderr` files.
Example: `$SRC_DIR/alloc/src/sync.rs:53:46`
- The base directory where the test's output goes is replaced with `$TEST_BUILD_DIR`.
This only comes up in a few rare circumstances.
Example: `/path/to/rust/build/x86_64-unknown-linux-gnu/test/ui`
- Tabs are replaced with `\t`.
- Backslashes (`\`) are converted to forward slashes (`/`) within paths (using
a heuristic). This helps normalize differences with Windows-style paths.
- CRLF newlines are converted to LF.
- Error line annotations like `//~ ERROR some message` are removed.
- Various v0 and legacy symbol hashes are replaced with placeholders like
`[HASH]` or `<SYMBOL_HASH>`.
Additionally, the compiler is run with the `-Z ui-testing` flag which causes
the compiler itself to apply some changes to the diagnostic output to make it
more suitable for UI testing.
For example, it will anonymize line numbers in the output (line numbers
prefixing each source line are replaced with `LL`).
In extremely rare situations, this mode can be disabled with the header
command `// compile-flags: -Z ui-testing=no`.
Note: The line and column numbers for `-->` lines pointing to the test are
*not* normalized, and left as-is. This ensures that the compiler continues
to point to the correct location, and keeps the stderr files readable.
Ideally all line/column information would be retained, but small changes to
the source causes large diffs, and more frequent merge conflicts and test
errors.
Sometimes these built-in normalizations are not enough. In such cases, you
may provide custom normalization rules using the header commands, e.g.
```rust,ignore
// normalize-stdout-test: "foo" -> "bar"
// normalize-stderr-32bit: "fn\(\) \(32 bits\)" -> "fn\(\) \($$PTR bits\)"
// normalize-stderr-64bit: "fn\(\) \(64 bits\)" -> "fn\(\) \($$PTR bits\)"
```
This tells the test, on 32-bit platforms, whenever the compiler writes
`fn() (32 bits)` to stderr, it should be normalized to read `fn() ($PTR bits)`
instead. Similar for 64-bit. The replacement is performed by regexes using
default regex flavor provided by `regex` crate.
The corresponding reference file will use the normalized output to test both
32-bit and 64-bit platforms:
```text
...
|
= note: source type: fn() ($PTR bits)
= note: target type: u16 (16 bits)
...
```
Please see [`ui/transmute/main.rs`][mrs] and [`main.stderr`] for a
concrete usage example.
[mrs]: https://github.com/rust-lang/rust/blob/master/src/test/ui/transmute/main.rs
[`main.stderr`]: https://github.com/rust-lang/rust/blob/master/src/test/ui/transmute/main.stderr
Besides `normalize-stderr-32bit` and `-64bit`, one may use any target
information or stage supported by [`ignore-X`](headers.md#ignoring-tests)
here as well (e.g. `normalize-stderr-windows` or simply
`normalize-stderr-test` for unconditional replacement).
## Error annotations
Error annotations specify the errors that the compiler is expected to emit.
They are "attached" to the line in source where the error is located.
```rust,ignore
fn main() {
boom //~ ERROR cannot find value `boom` in this scope [E0425]
}
```
Although UI tests have a `.stderr` file which contains the entire compiler output,
UI tests require that errors are also annotated within the source.
This redundancy helps avoid mistakes since the `.stderr` files are usually
auto-generated.
It also helps to directly see where the error spans are expected to point to
by looking at one file instead of having to compare the `.stderr` file with
the source.
Finally, they ensure that no additional unexpected errors are generated.
They have several forms, but generally are a comment with the diagnostic
level (such as `ERROR`) and a substring of the expected error output.
You don't have to write out the entire message, just make sure to include the
important part of the message to make it self-documenting.
The error annotation needs to match with the line of the diagnostic.
There are several ways to match the message with the line (see the examples below):
* `~`: Associates the error level and message with the current line
* `~^`: Associates the error level and message with the previous error
annotation line.
Each caret (`^`) that you add adds a line to this, so `~^^^` is three lines
above the error annotation line.
* `~|`: Associates the error level and message with the same line as the
previous comment.
This is more convenient than using multiple carets when there are multiple
messages associated with the same line.
### Error annotation examples
Here are examples of error annotations on different lines of UI test
source.
#### Positioned on error line
Use the `//~ ERROR` idiom:
```rust,ignore
fn main() {
let x = (1, 2, 3);
match x {
(_a, _x @ ..) => {} //~ ERROR `_x @` is not allowed in a tuple
_ => {}
}
}
```
#### Positioned below error line
Use the `//~^` idiom with number of carets in the string to indicate the
number of lines above.
In the example below, the error line is four lines above the error annotation
line so four carets are included in the annotation.
```rust,ignore
fn main() {
let x = (1, 2, 3);
match x {
(_a, _x @ ..) => {} // <- the error is on this line
_ => {}
}
}
//~^^^^ ERROR `_x @` is not allowed in a tuple
```
#### Use same error line as defined on error annotation line above
Use the `//~|` idiom to define the same error line as the error annotation
line above:
```rust,ignore
struct Binder(i32, i32, i32);
fn main() {
let x = Binder(1, 2, 3);
match x {
Binder(_a, _x @ ..) => {} // <- the error is on this line
_ => {}
}
}
//~^^^^ ERROR `_x @` is not allowed in a tuple struct
//~| ERROR this pattern has 1 field, but the corresponding tuple struct has 3 fields [E0023]
```
### `error-pattern`
The `error-pattern` [header](headers.md) can be used for
messages that don't have a specific span.
Let's think about this test:
```rust,ignore
fn main() {
let a: *const [_] = &[1, 2, 3];
unsafe {
let _b = (*a)[3];
}
}
```
We want to ensure this shows "index out of bounds" but we cannot use the
`ERROR` annotation since the error doesn't have any span.
Then it's time to use the `error-pattern` header:
```rust,ignore
// error-pattern: index out of bounds
fn main() {
let a: *const [_] = &[1, 2, 3];
unsafe {
let _b = (*a)[3];
}
}
```
But for strict testing, try to use the `ERROR` annotation as much as possible.
### Error levels
The error levels that you can have are:
1. `ERROR`
2. `WARN` or `WARNING`
3. `NOTE`
4. `HELP` and `SUGGESTION`
You are allowed to not include a level, but you should include it at least for
the primary message.
The `SUGGESTION` level is used for specifying what the expected replacement
text should be for a diagnostic suggestion.
UI tests use the `-A unused` flag by default to ignore all unused warnings, as
unused warnings are usually not the focus of a test.
However, simple code samples often have unused warnings.
If the test is specifically testing an unused warning, just add the
appropriate `#![warn(unused)]` attribute as needed.
### cfg revisions
When using [revisions](compiletest.md#revisions), different messages can be
conditionally checked based on the current revision.
This is done by placing the revision cfg name in brackets like this:
```rust,ignore
// edition:2018
// revisions: mir thir
// [thir]compile-flags: -Z thir-unsafeck
async unsafe fn f() {}
async fn g() {
f(); //~ ERROR call to unsafe function is unsafe
}
fn main() {
f(); //[mir]~ ERROR call to unsafe function is unsafe
}
```
In this example, the second error message is only emitted in the `mir` revision.
The `thir` revision only emits the first error.
If the cfg causes the compiler to emit different output, then a test can have
multiple `.stderr` files for the different outputs.
In the example above, there would be a `.mir.stderr` and `.thir.stderr` file
with the different outputs of the different revisions.
## Controlling pass/fail expectations
By default, a UI test is expected to **generate a compile error** because most
of the tests are checking for invalid input and error diagnostics.
However, you can also make UI tests where compilation is expected to succeed,
and you can even run the resulting program.
Just add one of the following [header commands](headers.md):
* Pass headers:
* `// check-pass` — compilation should succeed but skip codegen
(which is expensive and isn't supposed to fail in most cases).
* `// build-pass` — compilation and linking should succeed but do
not run the resulting binary.
* `// run-pass` — compilation should succeed and running the resulting
binary should also succeed.
* Fail headers:
* `// check-fail` — compilation should fail (the codegen phase is skipped).
This is the default for UI tests.
* `// build-fail` — compilation should fail during the codegen phase.
This will run `rustc` twice, once to verify that it compiles successfully
without the codegen phase, then a second time the full compile should
fail.
* `// run-fail` — compilation should succeed, but running the resulting
binary should fail.
For `run-pass` and `run-fail` tests, by default the output of the program
itself is not checked.
If you want to check the output of running the program, include the
`check-run-results` header.
This will check for a `.run.stderr` and `.run.stdout` files to compare
against the actual output of the program.
Tests with the `*-pass` headers can be overridden with the `--pass`
command-line option:
```sh
./x.py test src/test/ui --pass check
```
The `--pass` option only affects UI tests.
Using `--pass check` can run the UI test suite much faster (roughly twice as
fast on my system), though obviously not exercising as much.
The `ignore-pass` header can be used to ignore the `--pass` CLI flag if the
test won't work properly with that override.
## Test organization
When deciding where to place a test file, please try to find a subdirectory
that best matches what you are trying to exercise.
Do your best to keep things organized.
Admittedly it can be difficult as some tests can overlap different categories,
and the existing layout may not fit well.
For regression tests – basically, some random snippet of code that came in
from the internet – we often name the test after the issue plus a short
description.
Ideally, the test should be added to a directory that helps identify what
piece of code is being tested here (e.g.,
`src/test/ui/borrowck/issue-54597-reject-move-out-of-borrow-via-pat.rs`)
When writing a new feature, **create a subdirectory to store your tests**.
For example, if you are implementing RFC 1234 ("Widgets"), then it might make
sense to put the tests in a directory like `src/test/ui/rfc1234-widgets/`.
In other cases, there may already be a suitable directory. (The proper
directory structure to use is actually an area of active debate.)
Over time, the [`src/test/ui`] directory has grown very fast.
There is a check in [tidy](intro.md#tidy) that will ensure none of the
subdirectories has more than 1000 entries.
Having too many files causes problems because it isn't editor/IDE friendly and
the GitHub UI won't show more than 1000 entries.
However, since `src/test/ui` (UI test root directory) and `src/test/ui/issues`
directories have more than 1000 entries, we set a different limit for those
directories.
So, please avoid putting a new test there and try to find a more relevant
place.
For example, if your test is related to closures, you should put it in
`src/test/ui/closures`.
If you're not sure where is the best place, it's still okay to add to
`src/test/ui/issues/`.
When you reach the limit, you could increase it by tweaking [here][ui test
tidy].
[ui test tidy]: https://github.com/rust-lang/rust/blob/master/src/tools/tidy/src/ui_tests.rs
## Rustfix tests
UI tests can validate that diagnostic suggestions apply correctly
and that the resulting changes compile correctly.
This can be done with the `run-rustfix` header:
```rust,ignore
// run-rustfix
// check-pass
#![crate_type = "lib"]
pub struct not_camel_case {}
//~^ WARN `not_camel_case` should have an upper camel case name
//~| HELP convert the identifier to upper camel case
//~| SUGGESTION NotCamelCase
```
Rustfix tests should have a file with the `.fixed` extension which contains
the source file after the suggestion has been applied.
When the test is run, compiletest first checks that the correct
lint/warning is generated.
Then, it applies the suggestion and compares against `.fixed` (they must match).
Finally, the fixed source is compiled, and this compilation is required to succeed.
Usually when creating a rustfix test you will generate the `.fixed` file
automatically with the `x.py test --bless` option.
The `run-rustfix` header will cause *all* suggestions to be applied, even
if they are not [`MachineApplicable`](../diagnostics.md#suggestions).
If this is a problem, then you can instead use the `rustfix-only-machine-applicable`
header.
This should be used if there is a mixture of different suggestion levels, and
some of the non-machine-applicable ones do not apply cleanly.
## Compare modes
[Compare modes](compiletest.md#compare-modes) can be used to run all tests
with different flags from what they are normally compiled with.
In some cases, this might result in different output from the compiler.
To support this, different output files can be saved which contain the
output based on the compare mode.
For example, when in "non-lexical lifetimes" (NLL) mode a test `foo.rs` will
first look for expected output in `foo.nll.stderr`, falling back to the usual
`foo.stderr` if not found.
This is useful as "true" NLL mode can sometimes result in different
diagnostics and behavior compared to the "migrate mode" NLL (which is the
current default).
This can help track which tests have differences between the modes, and to
visually inspect those diagnostic differences.
If in the rare case you encounter a test that has different behavior, you can
run something like the following to generate the alternate stderr file:
```sh
./x.py test src/test/ui --compare-mode=nll --bless
```
Currently, only `nll` mode is checked in CI for UI tests.